Diffusion models have a mathematical tipping point where they stop memorizing and start creating.
Determining when a model is plagiarizing its training data versus generalizing has been a major challenge. This research uses conditional entropy to detect the exact moment a diffusion model transitions into associative memory. This transition is directly tied to the size of the dataset and the complexity of the patterns. It provides a way to verify that a model is actually learning to synthesize new ideas rather than just repeating what it saw. This lens helps us understand the fundamental boundary between AI memory and AI creativity. We can now measure exactly how much of an AI output is truly original.
Language Diffusion Models are Associative Memories Capable of Retrieving Unseen Data
arXiv · 2604.26841
When do language diffusion models memorize their training data, and how to quantitatively assess their true generative regime? We address these questions by showing that Uniform-based Discrete Diffusion Models (UDDMs) fundamentally behave as Associative Memories (AMs) $\textit{with emergent creative capabilities}$. The core idea of an AM is to reliably recover stored data points as $\textit{memories}$ by establishing distinct basins of attraction around them. Historically, models like Hopfield n