The Problem With Measuring AI Adoption

As organisations look for ways to encourage AI adoption, many are searching for metrics that demonstrate progress.

They track things like:

• How many people are using AI tools?
• How many prompts are being submitted?
• How many tokens are being consumed?

On the surface, these measurements seem sensible. If usage is increasing, adoption must be increasing too.

But the relationship is not always that simple.

During a recent conversation, futurist and researcher Dr Nicola Millard shared an example of an organisation that had introduced token targets, encouraging employees to use a certain amount of AI each week. The intention was understandable: encourage experimentation and help people become familiar with new tools. Yet the result was that some employees simply started using tokens for the sake of using tokens.

The organisation wanted meaningful adoption, but what it measured was usage.

This reflects a well-known principle known as Goodhart’s Law, often summarised as: “When a measure becomes a target, it ceases to be a good measure.”

Once people are given a target, they naturally start optimising for the thing being measured. A person can generate hundreds of prompts without improving a single customer interaction. An employee can use AI every day without making better decisions. A team can hit its usage targets without changing how it works.

The real question is not whether people are using AI. It is whether AI is helping them achieve better outcomes. Questions such as:

• Are customers getting better experiences?
• Are employees becoming more productive?
• Are organisations making better decisions?

These are much harder things to measure than prompts, tokens, or logins. They are also the things that matter most.

As AI moves from experimentation into everyday work, organisations will need to be careful about the signals they choose to track. Measuring activity is relatively easy. Measuring impact is much harder.

Sometimes the easiest thing to measure is not the most important thing to improve.