Current methods for proving an AI was trained on copyrighted data are no better than a coin flip.
April 24, 2026
Original Paper
Detecting Data Contamination in Large Language Models
arXiv · 2604.19561
The Takeaway
Membership Inference Attacks are the primary tool used to detect if a specific document was used in a training set. Modern evaluations show these tests fail almost every time, with accuracy scores hovering around 0.5. This means there is currently no reliable way to prove data leakage or copyright infringement in court using black-box testing. The sheer size of modern datasets makes it nearly impossible to find the fingerprints of a single document. This creates a massive legal and ethical challenge for the entire AI industry.
From the abstract
Large Language Models (LLMs) utilize large amounts of data for their training, some of which may come from copyrighted sources. Membership Inference Attacks (MIA) aim to detect those documents and whether they have been included in the training corpora of the LLMs. The black-box MIAs require a significant amount of data manipulation; therefore, their comparison is often challenging. We study state-of-the-art (SOTA) MIAs under the black-box assumptions and compare them to each other using a unifi