We've finally reverse-engineered the Transformer: it's literally running a simple mathematical recursion to learn from your prompt.
April 15, 2026
Original Paper
Layerwise Dynamics for In-Context Classification in Transformers
arXiv · 2604.11613
The Takeaway
Researchers extracted an explicit, depth-indexed recursion—a 'hidden update rule'—from inside a softmax Transformer. This explains exactly how the model performs in-context classification by 'learning' from examples on the fly. Before this, the Transformer's inner workings were a 'black box' of weights. Now, we have a readable algorithm that explains the process. This allows researchers to theoretically predict how a model will respond to a prompt and could lead to 'designing' prompts that perfectly trigger this internal recursion for maximum accuracy. The mystery of the 'black box' is starting to crack.
From the abstract
Transformers can perform in-context classification from a few labeled examples, yet the inference-time algorithm remains opaque. We study multi-class linear classification in the hard no-margin regime and make the computation identifiable by enforcing feature- and label-permutation equivariance at every layer. This enables interpretability while maintaining functional equivalence and yields highly structured weights. From these models we extract an explicit depth-indexed recursion: an end-to-end