An AI agent named LM-Tree can out-calculate human editors by 40% when deciding exactly how much a news article is worth to a search engine.
April 23, 2026
Original Paper
Pay-Per-Crawl Pricing for AI: The LM-Tree Agent
SSRN · 6507138
The Takeaway
Publishers struggle to figure out how to charge AI companies for the data used to train their models. The LM-Tree agent automatically discovers the value of different content types based on real-time buyer feedback. This automated mechanism outperforms traditional human-made taxonomies by a significant margin. Most people assume that data pricing is a static or arbitrary negotiation between giant corporations. This technology offers a concrete way for smaller creators to get paid based on the actual utility their work provides to an AI.
From the abstract
As AI systems shift from directing users to content toward consuming it directly, publishers need a new revenue model: charging AI crawlers for content access. This model, called pay-per-crawl, must solve a problem of mechanism selection at scale: content is too heterogeneous for a fixed pricing framework. Different sub-types warrant not only different price levels but different pricing rules based on different unstructured features, and there are too many to enumerate or design by hand. We prop