During 'grokking,' AI models learn the math perfectly thousands of steps before they actually start giving the right answers.
April 16, 2026
Original Paper
The Long Delay to Arithmetic Generalization: When Learned Representations Outrun Behavior
arXiv · 2604.13082
The Takeaway
This paper reveals a bizarre lag in neural learning: the underlying mathematical representation is often fully formed inside the transformer while the behavioral output is still failing. 'Understanding' and 'performance' are decoupled, with the internal logic ready long before it manifests in the results. This suggests that models we think are 'failing' a task might already possess the latent capability to solve it. For practitioners, this means current evaluation metrics might be missing the 'near-miss' models that are just a few steps away from a breakthrough. It changes how we think about training schedules and early-stopping criteria. You might be killing a model right before it's about to 'click.'
From the abstract
Grokking in transformers trained on algorithmic tasks is characterized by a long delay between training-set fit and abrupt generalization, but the source of that delay remains poorly understood. In encoder-decoder arithmetic models, we argue that this delay reflects limited access to already learned structure rather than failure to acquire that structure in the first place. We study one-step Collatz prediction and find that the encoder organizes parity and residue structure within the first few