Researchers developed a new way to stop language models from forgetting old facts

When you teach an AI a new skill, it often ends up overwriting what it learned before. This is a common hurdle in the world of machine learning known as catastrophic forgetting. A new research paper introduces a method called condition-anchored generative distillation, or CAGD, to solve this problem.

How it works

Instead of forcing the model to remember everything perfectly, the researchers use a clever workaround. They keep a small library of previous prompts and use an older, frozen version of the model to guide the learning process. The system looks at how the old model handled those specific prompts and tries to match that behavior while absorbing new information.

Imagine you are studying for a new subject while trying not to lose your grasp of the previous one. Instead of re-reading every single textbook from last year, you keep a cheat sheet of key concepts that help you recall the most important patterns.

Better performance

The team tested this on a 219M masked diffusion language model. Using their new method, they significantly reduced the error rate during continual training across four different tasks. They also found that this approach works well across different types of language models, including Qwen3.

Limits to the technique

While the results are promising, the method has its boundaries. The researchers noted that while their approach helps keep the format of answers consistent in tasks like GSM8K, the ability of the model to retain exact matches for facts can degrade as the model size increases. The team has not yet tested this on massive commercial models with hundreds of billions of parameters, so it remains to be seen if the same principle holds at that scale. Can we eventually build a model that learns forever without needing to be retrained from scratch?