What “impact” means in an AI newsroom

Ask a newsroom to define impact, and you'll usually get a page view number. Ask what actually changed because of the story, and the room goes quiet.

Impact is one of those words every newsroom says it cares about and almost nobody measures well. It shows up in mission statements, in grant applications, in the opening slide of an editorial strategy deck. And then, when a story actually publishes, it gets reduced to whatever's easiest to screenshot: traffic, shares, maybe a mention on a bigger outlet's roundup. Not because anyone decided that's what impact means. Just because it's the thing the dashboard already shows.

That gap, between what newsrooms say impact is and what they actually measure, isn't new. But it's getting more consequential, not less, as AI reshapes what's cheap to produce and what's expensive to prove.

In an AI newsroom, impact means two specific things, and most newsrooms aren't measuring either of them well.

First: a traceable, attributable change. A decision made, a policy reversed, a wrong corrected, a reader who understood something well enough to act differently; not simply that a story existed and was read. “We covered it” is not impact. “Something happened because we covered it” is.

Second, and this is the part that's new: that change has to be provable even though AI now sits between your journalism and the person it reached. A reader who acted on your reporting after encountering it secondhand, summarised inside a chatbot with no byline attached, is still impact but it's impact your newsroom can no longer see, prove, or claim credit for using the methods built for a world where readers landed on your site directly. An AI newsroom's definition of impact has to account for that invisible middle step, or it's measuring a world that no longer fully exists.

Why “extra admin” is the wrong frame

One thing that comes up constantly when newsrooms discuss impact work: a real, understandable resistance to treating it as anything more than extra admin. Tracking outcomes, following up on whether a policy actually changed, documenting what happened after publication. It feels like paperwork bolted onto the actual job, which is reporting the story and moving on to the next one.

That resistance makes sense in a newsroom built around volume. It makes a lot less sense in a newsroom where AI has quietly changed the economics of volume altogether. If a tool can help draft, summarise or localise content faster than ever, “we published a lot” stops being a meaningful claim on its own; everyone can now publish a lot. What becomes scarce, and therefore valuable, is exactly the definition above: proof that something published changed something, traceable all the way through. Impact tracking isn't admin on top of the job anymore. In an AI-saturated information environment, it's close to becoming the job.

Pageviews were always a proxy. 

Pageviews were never really a measure of impact. They were a measure of distribution, standing in for impact because it was the easiest thing to count and, for a long time, strongly correlated with advertising revenue. That correlation is breaking down. Traffic increasingly arrives filtered through a search summary, an AI answer, a social platform's algorithm, several steps removed from anyone actually sitting with the journalism.

We've written before about what it means when your audience reads your story through ChatGPT rather than your own site and the same shift applies here. If the traffic number is increasingly disconnected from genuine engagement, using it as your impact metric means measuring something that's quietly stopped meaning what it used to.

Redefining success before AI redefines it for you

If newsrooms don't deliberately redefine what counts as success, AI and the platforms built on it will do it by default and the definition that wins will be whatever's easiest to optimise for algorithmically, which is rarely the thing a newsroom actually exists to do.

Which is exactly why the definition has to be decided deliberately, not inherited from whatever the dashboard happens to measure. The alternative isn't complicated to describe, even if it's harder to execute: get specific, before a story ever publishes, about what change it's actually trying to produce. Not “we hope this raises awareness”; awareness of what, leading to what decision, made by whom? That's a very different reporting and follow-up process to “publish and watch the numbers,” and it has to start at the pitch stage, not get bolted on afterward once someone asks how the story performed.

This is an editorial ROI question, not just an editorial values question

We made a related case in our piece on measuring the ROI of AI in your newsroom: real ROI shows up in efficiency, in quality and risk, and in trust and retention and almost nobody measures the third one because it's harder than counting hours saved. Impact is the editorial sibling of that same problem. It's the dimension of newsroom value that's genuinely hardest to quantify, which is exactly why it's the one most likely to get quietly dropped when AI makes the easy numbers even easier to generate.

Put differently: a newsroom that's rigorous about AI governance but vague about what its journalism is actually meant to achieve has only solved half the problem. Knowing how a story got made is a governance question. Knowing whether it mattered and being able to prove it, even when AI is standing between the story and the reader is the impact question. Newsrooms need both answers, not just the one that's easier to produce a policy document for.

Where this actually starts

Not with a new measurement framework bolted onto existing workflows, that's exactly the “extra admin” trap again. It starts earlier: building the question “what change is this story actually trying to produce, and how would we know even if a reader never hits our site” into the pitch meeting itself, so impact becomes something a story is designed for from the outset, rather than something measured, apologetically, after the fact.

This connects to the conversation in our pieces on measuring the ROI of AI in your newsroom and what it means when your audience reads you through ChatGPT, and to the conversations happening on The Signal. If you're trying to work out what impact should actually mean for your newsroom, get in touch.