Bridges the gap between generative (MAE) and predictive (I-JEPA) self-supervised learning, achieving a 10% performance boost.
March 17, 2026
Original Paper
Self-Distillation of Hidden Layers for Self-Supervised Representation Learning
arXiv · 2603.15553
The Takeaway
By tasking models with predicting hidden layer representations rather than just the final output, it stabilizes training and improves feature abstraction. This is a significant jump in efficiency and accuracy for vision-based self-supervised representation learning.
From the abstract
The landscape of self-supervised learning (SSL) is currently dominated by generative approaches (e.g., MAE) that reconstruct raw low-level data, and predictive approaches (e.g., I-JEPA) that predict high-level abstract embeddings. While generative methods provide strong grounding, they are computationally inefficient for high-redundancy modalities like imagery, and their training objective does not prioritize learning high-level, conceptual features. Conversely, predictive methods often suffer f