We’re Measuring Activity, Not Outcomes with AI

The bottleneck in knowledge work isn’t the number of ideas - Most people are already swimming in notes, tabs, messages, documents, half-drafts of products/projects, and a long list of things that might be worth doing. AI mostly adds more surface area: more options, more variants, more angles, more language, more drafts, and allows anyone to make them quickly. But how much value is it adding?

The real constraint is choosing the right thing to work on, and then turning it into something that actually has an impact in your world: a decision, a deliverable, a shipped artifact, a change someone can use.

AI will happily help you explore the wrong thing for as long as you want it to. I don't think I've seen a scenario where Codex/Claude/your tool of choice has said: "I've nothing more to say on this topic, these are all the things I know, you have enough to make a decision". It will always give one more answer.

So the question aren’t "how much code did you generate?" or "how many prototypes did you build?" or "how many tokens did you burn?".

The questions to ask are: Did you produce an output that someone would recognise as useful? Did it help the bottom line? Is a feature better/faster as a result of what you did? Did you learn something?

For example, a successful outcome might be:

• a decision with a rationale and a next step.
• a draft that is ready to send or publish (not just “a draft”).
• a plan that suggests one clear next step and a route to a successful outcome.
• a feature that is widely used or solves a particular problem for your company/product.

It's why I don't like the 'tokenmaxxing' narrative we're seeing lately.  It measures activity, not outcomes. Did someone really create 500m worth of value with this spend?

The better measure is simpler: how often does an AI session end with something you can ship, use, or hand off?

It has been said that it's hard to measure the value of AI - some companies are actively putting limits on tool usage, which also gives an answer as to how they believe it impacts their bottom line.

I still think AI has enormous potential - I used it to help me write this article! But if we want it to become more than another layer of noise, we need to get sharper about how we measure its value. Adding it everywhere is not the same as making work better. The useful question is not whether AI was involved, or how much it produced, but whether it helped turn work into something that mattered: a better decision, a faster delivery, a clearer plan, a useful artifact, or a measurable change in how work gets done. Until we measure that, we’re mostly measuring motion.