Four top AI models fell to 13% or lower on the hardest long-list tracking tests in StateBench

Even the best AI models lose the thread when a long text keeps changing the facts, according to a new benchmark called StateBench. It measures how well models follow the current state of things as a document unfolds.

The headline result

Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna were asked to give the final value of every variable. All four did well on the fixed and replaced levels. On running totals and on copying and swapping, accuracy dropped sharply as inputs grew. On the longest copy-and-swap problems it fell to 13% or lower.

What state tracking means

Think of a detective story in which John hands Mary a knife, then later hands her whatever is in his backpack. To answer a question, you must follow each change in order. Code does the same with variables that get set and reset.

Four levels of difficulty

• Fixed values: a value is set once and never changes, like a birth date.
• Replaced values: only the latest setting counts, like a thermostat.
• Running totals: each update builds on the current value, like a game score.
• Copying and swapping: one variable takes another's current value. Think of a magician swapping cups with a ball under one of them.

The team built all four levels from the same sequence, with the same number of variables and the same update points. Only the type of update differs, so the comparison is fair.

Open models as they come

The team also tried open pretrained models, among them the transformer Olmo 3 and Olmo Hybrid, a version with added Gated DeltaNet layers. They mostly did well on fixed values, lost accuracy on replaced values as inputs grew, and stayed at chance on copy-and-swap. Format also mattered: two 275-million-parameter models scored near the ceiling on one format of fixed-value problem but near chance on the same problem framed as spreadsheet edits.

Training directly on the task

The team then trained 275-million-parameter models from scratch on sequences of up to 256 assignments, and tested them on up to 64,000. Gated DeltaNet and hybrid models held up almost perfectly on the first three levels, even across tens of thousands of assignments. Transformers learned the easier levels but faded on longer inputs, and never learned the hardest one. The more flexible Gated DeltaNet variant stayed near perfect on the hardest level out to about 10 times its training length.

So for Gated DeltaNet and hybrids, the weak pretrained results point to training, not design. For transformers, even direct training did not help on the hardest level.

If a model can lose track after a few thousand swaps, how much should you trust it with a very long log?

Limits

StateBench assumes a fixed number of variables, while real conversations add new people and objects over time. The team says that case is not covered yet. The exact place of a more limited Gated DeltaNet variant in the hierarchy is also still open, including whether it can solve all running-total tasks.