Researchers found that 10% of AI tokens use 80% of the compute

A new paper from arXiv measures how much computation each token in a large language model actually needs.

The researchers used a method called Mixture-of-Agents to test this.

They set up a panel of 15 different language models with varying sizes.

Each model tried to predict the next word in a sequence.

The smallest model that got the answer right defined the token's cost.

This approach revealed a surprising imbalance in how AI works.

Most tokens are easy for the smallest models to handle.

A 0.5B parameter agent reproduced 92% to 95% of reference tokens.

However, the most difficult 10% of tokens required the most compute.

These hard tokens accounted for 64% to 80% of the total estimated FLOPs.

Imagine paying the same price for a cup of water and a diamond.

That is what current AI systems do for every single word.

The team tested this on benchmarks like MATH-500.

Using this data, they improved model routing efficiency.

Projected latency dropped from 7.59 seconds to 5.12 seconds.

Drafting models also used 32.6% fewer tokens with this method.

This suggests current AI systems waste significant energy on easy words.

The study shows that smarter allocation could make AI faster and cheaper.

The paper does not claim this works for every possible task.

It focuses on specific benchmarks and model families like Qwen and OLMo.

Further testing is needed to see if this holds up in real-world use.