AutoIndexer let a language model edit its own output after the fact

AutoIndexer gives a causal language model a text-editor trick: it can go back, move a cursor, and change earlier output after it has already started generating.

The work comes from an arXiv cs.CL paper published on 2026-10-12.

The core idea is a Chain of Edits. Instead of only predicting the next token in a fixed left-to-right line, the model learns marker tokens that open and close edits while it points to positions inside the existing sequence.

That lets it insert, replace, or delete tokens in hindsight. In plain terms, it does not just keep typing forward. It can also fix what it already wrote.

Training uses ground-truth labels that are first perturbed with inverse edit operations. The model then learns to undo those inverses, which makes editing part of the learning process rather than a separate post-processing step.

The paper says this setup is meant for data that does not fit neat causal order, such as cross-references or future dependencies. That is a common headache for standard autoregressive decoding, which cannot revise earlier text once later tokens arrive.

AutoIndexer is also described as more flexible than diffusion LLMs for this job, because it can insert or remove an arbitrary number of tokens from a pre-generated context.

The authors say the same perturbation training makes the model more robust to noisy context and useful for editing tasks such as code correction. It also keeps performance on standard LLM benchmarks.

So the pitch is simple: keep the language-model brain, but give it an editor’s hands. Imagine asking a model to draft a paragraph and then nudge one sentence in place instead of starting over.

The excerpt does not give benchmark numbers, and it does not show what happened on a full user-facing product. It also does not say how the method compares across every editing task, only that it preserves benchmark performance and adds post-generation editing ability.