StateBench: Language model state tracking benchmark

What changed

Allen Institute for AI introduced StateBench on November 3, 2026, a benchmark that measures how accurately language models track state changes as they process long documents. The benchmark includes four difficulty levels representing different types of state-tracking complexity, grounded in Krohn-Rhodes theory, a branch of algebra that describes how state-tracking problems can be decomposed into atomic primitives.

The four levels progress from simple to complex. R-trivial problems require finding immutable values that never change. Aperiodic problems require finding the most recent value among reassignments. Soluble problems add self-referential updates where each change builds on the current value. General finite problems, the hardest level, introduce assignment by reference, where one variable takes another variable’s current value, creating dependencies across updates.

The researchers tested the benchmark against multiple model architectures, ranging from open pretrained models with 275 million to 7 billion parameters, to frontier models like Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol, and Luna. They also conducted targeted direct training experiments with models built from scratch on StateBench tasks. Code and data from the benchmark are publicly available.

Why it matters

State tracking underlies critical tasks requiring understanding how information changes over time. Reading code involves tracking variable assignments and reassignments. Following narratives requires knowing how characters move and objects change hands. For teams building AI agents or systems handling complex workflows, a model’s ability to maintain accurate state is essential for reliability.

The benchmark reveals significant architectural differences with implications for production use. Frontier models, despite their size and training, show rapid accuracy drops on harder state-tracking problems as inputs grow longer. Accuracy fell to 13% or lower on the longest general finite problems tested. This suggests that scaling model size alone does not solve harder state-tracking tasks, affecting how teams choose models for workflows requiring long-document reasoning.

For long-context applications and agents, the results indicate that architecture choice matters more than previously understood. Hybrid models and Gated DeltaNet variants showed substantially better generalization than standard transformers when trained directly on state-tracking tasks. When trained on sequences up to 256 assignments, Gated DeltaNet models generalized nearly perfectly on harder problems out to sequences 250 times longer than their training data, while transformers degraded even with direct training. Transformers never learned to handle the hardest general finite problems, regardless of training approach.

What to test

Before adopting a model for state-tracking workloads, evaluate these points:

Test on your actual task complexity. The benchmark shows that problem format matters significantly. Models that scored near ceiling on r-trivial problems presented in one format performed near chance on the same problems presented differently. Verify that benchmark performance on StateBench translates to your specific use case.

Check architecture claims against your context length requirements. Frontier model performance drops sharply as input length increases, particularly on harder state-tracking problems. If your workflow requires handling documents longer than the model was trained on, expect degraded accuracy unless you test directly.

Understand training versus pretraining performance gaps. The researchers found that pretrained models perform worse than models trained directly on state-tracking tasks, suggesting that general pretraining may not teach state tracking effectively. If state tracking is critical, consider whether targeted fine-tuning could improve results beyond off-the-shelf performance.

Assess sensitivity to missed updates. On harder problems, many steps affect the final answer, leaving less room for error. Test how tolerant your application is to single missed updates in long dependency chains.

Verify open versus closed model behavior. The benchmark tested both open pretrained models and frontier closed models. Frontier model accuracy drops on general finite problems are documented; verify whether this aligns with what you observe in production on your specific workloads.

Common questions

How does state tracking differ from long-context understanding?

State tracking specifically measures whether a model accurately follows how information changes across a document, not just whether it can access information from long contexts. A model might retrieve distant facts correctly but still fail to track how those facts evolve through a sequence of updates. StateBench tests this distinction across four levels of complexity.

Which model architecture handles state tracking best?

When trained directly on state-tracking tasks, Gated DeltaNet models with flexible memory updates generalized farthest on the hardest problems, staying near perfect accuracy out to roughly 10 times their maximum training sequence length. Hybrid models combining transformers with Gated DeltaNet layers showed similar generalization. Standard transformers, even with direct training, never learned the hardest general finite problems.

Do frontier models solve state tracking well?

Frontier models including Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol, and Luna performed well on easier r-trivial and aperiodic problems but accuracy fell sharply on soluble and general finite problems as inputs grew longer, dropping to 13% or lower on the longest general finite test cases. This indicates that model scale alone does not solve harder state-tracking challenges.

The conclusion

StateBench provides the first rigorous, multi-level benchmark for a capability that matters in production systems but has not been systematically measured before. The results show that frontier models have real, predictable limitations on harder state-tracking problems, and that architecture matters more than pretraining when direct training is possible.

Teams evaluating models for agents or workflows that require accurate state tracking should test against StateBench before committing. The benchmark’s four-level structure makes it practical to identify exactly where a model breaks down. For now, hybrid architectures and Gated DeltaNet variants offer better theoretical and empirical guarantees on harder problems, though they may not yet be available in all commercial offerings.

Watch for how vendors respond to these findings. As recurrent and hybrid architectures become more common, we should expect to see more focus on state-tracking capabilities in model announcements. The open availability of StateBench code and data means the community can now test new architectures and training methods against this standard, which should accelerate progress on a capability that has received less attention than raw scaling.

Related reading

• Anthropic Message Threads reduces latency 35% for long conversations
• Mirendil Fern 0.1: 10M-token context coding model