The Problem With Measuring AI By Tokens

For years, organisations have searched for ways to measure technology adoption. First it was software licences. Then it was active users. Then it became logins, dashboards and usage statistics. Now, increasingly, it’s tokens.

As organisations invest more heavily in AI, many platforms make token consumption highly visible. Leaders can see millions of tokens being used across departments, teams and projects, often in real time. At first glance, those numbers appear to answer an important question: Are people actually using AI? But while token consumption tells us how much AI is being used, it doesn’t necessarily tell us whether anything valuable is happening as a result.

Tokens Are Easy To Measure

A token is simply a unit used by AI models to process language. Every prompt you type, every response you receive and every document you upload consumes tokens. That makes them incredibly useful for understanding how AI is being used and, increasingly, how much it is costing.

From a leadership perspective, dashboards showing millions of tokens consumed each month can feel reassuring. They suggest people are experimenting with AI, engaging with new tools and making use of the investment the organisation has made. Finance teams can also use token consumption to understand where costs are coming from, particularly as many AI platforms charge based on usage rather than a fixed licence fee.

The difficulty comes when those numbers start being treated as a measure of success rather than simply a measure of activity.

When The Metric Becomes The Goal

One observation from a recent conversation with Dr Nicola Millard perfectly captured the risk.

// “I was talking to somebody who’s a Chief Technology Officer… they’ve got token targets. People have got to use a certain number of tokens per week to prove they’re using AI.”

The unintended consequence was immediate.

//”I’m doing stuff just to use tokens.”

It’s easy to understand why organisations introduce targets like this. If they’ve invested heavily in AI, they naturally want employees to experiment with the tools rather than ignore them. But the moment token usage becomes a target, behaviour starts to change. Instead of asking whether AI is helping people work more effectively, people begin asking how they can increase their token count.

We’ve Seen This Before

This isn’t a new problem. Organisations have been here before. Contact centres became obsessed with average handling time. Marketing teams focused on page views. Social media platforms rewarded likes and impressions. None of those metrics were meaningless. The problem came when they became targets rather than indicators — something we’ve explored before through Goodhart’s Law and the problem with measuring AI adoption.

People naturally optimise for whatever they’re measured against. Employees can always generate more token usage by:

• asking unnecessary questions
• uploading documents that don’t need analysing
• having longer conversations with AI

Token consumption goes up. That doesn’t necessarily mean better work is being done.

Adoption Is Only The Beginning

Later in our conversation, Nicola reflected on another challenge she encounters regularly. Many people still ask whether they’re “missing out” by not using AI yet, but her response was simple: “It depends what you want to do.”

That observation gets to the heart of the issue. AI doesn’t create value simply because it’s available, and it certainly doesn’t create value because people use more of it. The real benefit comes when it helps someone solve a problem more effectively, improve the quality of their work, make a better decision or free up time for something more valuable.

Those outcomes are far harder to measure than token consumption, but they’re ultimately the reason organisations invested in AI in the first place.

The Better Question

Perhaps organisations are asking the wrong question. Instead of asking: How many tokens did we consume this month?

Perhaps they should be asking:

• Did work become easier?
• Did quality improve?
• Did customers benefit?
• Did employees learn something?
• Did people make better decisions?

Those are much more difficult questions to answer, but they’re also the ones that matter.

The Metric Is Not The Outcome

Many organisations have already moved from asking “Do we have AI?” to “Are people using AI?” Measuring token consumption feels like the next logical step, because it’s visible, measurable and easy to report. But tokens measure how much AI was used. They don’t measure whether anything valuable happened.

Organisations are becoming very good at measuring AI activity. The bigger challenge over the next few years may be learning how to measure AI value instead.