The breakthrough needs to make LLMs better at learning from context at test time. Some examples of results that could demonstrate this breakthrough would be a model that shows less degradation than contemporaries on long-context or one that can more effectively learn from mistakes during agentic tasks.
The breakthrough needs to be major and conceptual, it can't be something as simple as models getting better at in-context learning through more scaling.
In order to count this breakthrough must apply to general LLMs, not just using continual learning to solve a narrow problem or class of problems
A paper claiming to have a really good result isn't enough (like the Titans paper). The breakthrough must be widely accepted as legitimate and there should be publicly accessible models that use it
A lightweight control mechanism of plastic weights that sit on top of the static weights. The LLM can be trained using RL to produce sequences of "continual learning tokens", upon which it optimizes some objective, backpropagating through the plastic weights. The control mechanism can be implemented as lightweight residual layers initialised at zero (no modification at initialisation).
This adds another scaling dimension on top of pretraining and reasoning. The key challenges include the formulation of the RL objective and the implementation of the control mechanism. Solving these problems enables agents to operate on much longer task horizons (7-30 human equivalent days) without getting stuck in typical LLM amnesia loops. The caveat is that the gradient steps to update the fast weights requires substantial GPU compute in order to perform the test-time training.