DDPO addresses the 'overthinking' and 'overconfidence' issues in Large Reasoning Models (LRMs) by optimizing answer length based on task difficulty.
March 20, 2026
Original Paper
Balancing the Reasoning Load: Difficulty-Differentiated Policy Optimization with Length Redistribution for Efficient and Robust Reinforcement Learning
arXiv · 2603.18533
The Takeaway
It introduces a policy optimization that shortens simple answers while expanding reasoning for complex ones, reducing average output length by 12% while improving accuracy. This is a practical solution for deploying expensive 'o1-style' reasoning models more efficiently.
From the abstract
Large Reasoning Models (LRMs) have shown exceptional reasoning capabilities, but they also suffer from the issue of overthinking, often generating excessively long and redundant answers.For problems that exceed the model's capabilities, LRMs tend to exhibit the overconfidence phenomenon, generating overly short but incorrect answers, which may contribute to suboptimal performance.To address these issues, we propose Difficulty-Differentiated Policy Optimization (DDPO), an efficient reinforcement