The CBC is A/B Testing the News

Visiting the front page of CBC Prince Edward Island’s website this morning, I noticed that the headline on the main story was Let’s have some babies’: Aylward calls for support for families trying to conceive, but that this was, after a brief pause, replaced with the headline Opposition calls for support for families trying to conceive.

The original headline looked like this:

The quickly-replaced-by headline looked like this:

This had all the appearances of “A/B testing,” something websites do to test out different versions of headlines, photos, buttons and other elements to see which has a better “conversion rate” (conversion, in this case, meaning “clicking on the headline to read the story”).

Generally A/B testing is opaque to we the reader: we don’t know it’s happening to us. But in this case, something about the technology the CBC is using to conduct the A/B test is introducing a delay long enough for us to see behind the curtain.

Looking under the hood of the CBC’s site with Firefox’s Developer Tools, I see a call to a URL at Chartbeat, and loading this URL into a browser shows all the headline tests currently running on the CBC PEI site:

{
  "status": "success",
  "data": {
    "domain": "cbc.ca",
    "probability_is_control": 0,
    "probability_is_lift": 0,
    "host": "cbc.ca",
    "experiments": {
      "[e8f119d5]": {
        "metadata": "{\"version\":3,\"sub_location\":\"\",\"initial_content\":\"How a planning firm thinks Borden-Carleton could attract more tourists\"}",
        "specific_location": "*[@id='blogroll-story-1.4434879'][@href='http:\/\/www.cbc.ca\/news\/canada\/prince-edward-island\/pei-borden-carleton-draft-spatial-tourism-plan-1.4434879']",
        "variants": {
          "A": {
            "content": "How a planning firm thinks Borden-Carleton could attract more tourists",
            "probability": 0.6954,
            "content_type": "text",
            "metadata": "{}"
          },
          "B": {
            "content": "Borden-Carleton aims to attract more tourists with spatial plan",
            "probability": 0.3046,
            "content_type": "text",
            "metadata": "{}"
          }
        },
        "location": "*[@id='blogroll-story-1.4434879'][@href='http:\/\/www.cbc.ca\/news\/canada\/prince-edward-island\/pei-borden-carleton-draft-spatial-tourism-plan-1.4434879']",
        "strategy_last_updated": 1512572157
      },
      "[c5d38929]": {
        "metadata": "{\"version\":3,\"sub_location\":\"\",\"initial_content\":\"'Let's have some babies': Aylward looks to support for families trying to conceive\"}",
        "specific_location": "*[@id='topstory-headline-1.4434994'][@href='http:\/\/www.cbc.ca\/news\/canada\/prince-edward-island\/pei-fertility-treatment-access-support-1.4434994']",
        "variants": {
          "A": {
            "content": "'Let's have some babies': Aylward looks to support for families trying to conceive",
            "probability": 0.408,
            "content_type": "text",
            "metadata": "{}"
          },
          "B": {
            "content": "Opposition calls for support for families trying to conceive",
            "probability": 0.592,
            "content_type": "text",
            "metadata": "{}"
          }
        },
        "location": "*[@id='topstory-headline-1.4434994'][@href='http:\/\/www.cbc.ca\/news\/canada\/prince-edward-island\/pei-fertility-treatment-access-support-1.4434994']",
        "strategy_last_updated": 1512572157
      },
      "[7c9d7236]": {
        "metadata": "{\"version\":3,\"sub_location\":\"\",\"initial_content\":\"'The whole waiting room calmed down:' Therapy dogs approved for Charlottetown ER\"}",
        "specific_location": "*[@id='blogroll-story-1.4433307'][@href='http:\/\/www.cbc.ca\/news\/canada\/prince-edward-island\/dogs-therapy-queen-elizabeth-hospital-p-e-i-therapy-paws-of-canada-1.4433307']",
        "variants": {
          "A": {
            "content": "'The whole waiting room calmed down:' Therapy dogs approved for Charlottetown ER",
            "probability": 0.7157,
            "content_type": "text",
            "metadata": "{}"
          },
          "B": {
            "content": "How therapy dogs are helping patients at QEH",
            "probability": 0.2843,
            "content_type": "text",
            "metadata": "{}"
          }
        },
        "location": "*[@id='blogroll-story-1.4433307'][@href='http:\/\/www.cbc.ca\/news\/canada\/prince-edward-island\/dogs-therapy-queen-elizabeth-hospital-p-e-i-therapy-paws-of-canada-1.4433307']",
        "strategy_last_updated": 1512572157
      }
    },
    "path": "\/news\/canada\/prince-edward-island"
  }
}

For the story about Borden-Carleton’s development plans we have:

1. How a planning firm thinks Borden-Carleton could attract more tourists
2. Borden-Carleton aims to attract more tourists with spatial plan

For the story about therapy dogs at the Queen Elizabeth Hospital there are:

1. ‘The whole waiting room calmed down:’ Therapy dogs approved for Charlottetown ER
2. How therapy dogs are helping patients at QEH

And for the story about infertility services, as above, we have:

1. ‘Let’s have some babies’: Aylward looks to support for families trying to conceive
2. Opposition calls for support for families trying to conceive

The options are injected into the CBC site by a third-party service called Chartbeat, which bills itself as offering “Content Intelligence for Publishers.”

Chartbeat has a service called Headline Testing that it promotes like this:

Run headline tests that optimize for reader engagement after the click, so you can maximize each story’s readership by ensuring it has the best headline.

The reason that I was able to see both the “before” and “after” headlines was because the CBC hasn’t implemented what Chartbeat calls Asynchronous Implementation With Flicker Control, an option that would hide the page until the switch was in.

What’s most interesting about the data that Chartbeat returns as part of its headline-switching magic is the “probability” data. While this doesn’t appear to be publicly documented anywhere, I’ve noticed that it changes over time, and I’m assuming that this is a reflection of which of the variants is “winning” the race to conversion, something that Chartbeat goes into more detail about:

The test ends when the winning headline has been determined with 95% confidence and Chartbeat begins to automatically serve that headline 100% of the time.

To make sure that tests come to a conclusion as efficiently as possible, we have an alternate way to determine a winner, called ‘soft convergence’. If 20 minutes have passed and we’re 95% confident that no headline is better by a margin of 25%, the leading headline will win.

So I presume that, for any of the variants above, once the “probability” level reaches 0.95–95%–that headline will be deemed to have “won” and we’ll all see it all the time.

I’m a big fan of A/B testing, and I practice it all the time to test designs, wording, button text and other aspects of my client’s websites: it’s proved a useful tool.

But I don’t think that journalism is something that should be A/B tested. Headlines are not “marketing,” they are part of the journalism, and I want my journalism to be crafted by journalists based on what they see as the most effective way of telling a story, not based on “what converts more.”

If you want to learn more about A/B testing, listen to Jon Ronson’s recent audio series The Butterfly Effect, which is available as a podcast.

The series is described like this:

Join bestselling author Jon Ronson as he traces a very strange butterfly effect. A teen in Brussels had an idea – to make porn free and easy to stream online. The consequences of that idea are mysterious, delightful, surprising, and sad. This 7-part series takes you on a journey to places you’ll never expect.

One of those “places you’ll never expect” is into the tech behind the hub of the free porn industry, based in Montreal, which uses A/B testing extensively. Ronson interviews Pornhub founder Fabian Thylmann, and one of his technical people, Brandon:

Fabian Thylmann: If you would walk into their offices today, unless you stumbled on the one floor, you would not notice what they do. Because you have a huge amount of support people, you have a huge amount of SEO people–search engine people. You would not notice; it’s quite impressive.

Jon Ronson: Brandon was put in charge of building Mansef’s fledgling mobile division. If you’ve ever watched Pornhub on a smart phone, you have Brandon to thank. And you also have him to thank for enticing you with immaculately data-analyzed category thumbs. On Pornhub, the porn is divided into categories: Asian and blow job and teen and so on. Each category has its own front page photograph. Brandon and his team would put three or four alternate photos through “continuous A/B testing” to find out which of them converted the best. They might be almost identical, with just the hint of a change in the facial expression.

Ronson’s tale is fascinating and well-told.

And after listening to it, I’m more convinced than ever that the A/B testing used to find the best porn thumbnails isn’t a tool that should be found in the journalist’s toolbox.