StateBench found four top AI models fall to 13% or lower on the hardest long-document tracking

Ai2 has released StateBench, a test of how well language models keep track of things that change across a long document. The results come from the lab's own blog post, which describes tests on open models and four frontier models.

What state tracking means

Imagine a story where John hands Mary a knife. Later, someone asks who holds it. The model must remember every handoff, not just the last sentence. Code works the same way, with variables that get set and reset.

Four levels of difficulty

StateBench has four levels. Each allows a different kind of update.

• R-trivial: a value is set once and never changes, like a birth date.
• Aperiodic: a value can be replaced, like a thermostat setting. Only the latest counts.
• Soluble: updates build on the current value, like a game score.
• General finite: one variable can copy another, as in swapping cups with a ball under one of them.

All four problems share the same number of variables and the same update points. Only the type of update changes, so the comparison is fair.

What the frontier models did

Ai2 tested Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna. Each read a list of assignments and gave the final value of every variable. All four did well on r-trivial and aperiodic problems. On soluble and general finite problems, accuracy fell sharply as inputs grew, down to 13% or lower on the longest general finite ones.

Open models and direct training

Among open models pretrained on the same data, almost all did well on r-trivial problems, slipped on aperiodic ones as inputs grew, and stayed at chance on general finite ones. Format mattered: two small 275-million-parameter models scored near the ceiling on one format but near chance on the same problem framed as spreadsheet edits.

Ai2 then trained 275-million-parameter models from scratch on StateBench, using sequences of up to 256 assignments and testing on up to 64,000. Gated DeltaNet and hybrid models, which carry a running memory forward, generalized almost perfectly on the first three levels. Transformers degraded on longer sequences and never learned the general finite level. The most flexible Gated DeltaNet variant stayed near perfect on the hardest level out to roughly 10 times its training length.

So for those recurrent and hybrid designs, the weakness seems to come from pretraining, not the architecture. Transformers did not get that rescue. Would you trust a model with your long spreadsheet?

Limits

StateBench assumes a fixed number of variables, while real conversations add new people and objects. Ai2 says the exact place of the more limited Gated DeltaNet in the hierarchy is still open, including how precision and attention layers affect it. The frontier models were tested only as described, and the source gives no results beyond those.