You're wasting money generating 'fresh' data for RLHF when recycling old samples works just as well.
April 15, 2026
Original Paper
Efficient RL Training for LLMs with Experience Replay
arXiv · 2604.08706
The Takeaway
The prevailing wisdom in LLM post-training is that Reinforcement Learning requires fresh, on-policy data to avoid distribution shift. This paper challenges that paradigm, proving that 'experience replay'—reusing old data samples—maintains performance while significantly reducing inference compute during training. By breaking the requirement for constant new data generation, we can slash the cost of high-quality RLHF. It’s a major efficiency win for any team fine-tuning models on a budget, making high-performance RL accessible without massive GPU clusters.
From the abstract
While Experience Replay - the practice of storing rollouts and reusing them multiple times during training - is a foundational technique in general RL, it remains largely unexplored in LLM post-training due to the prevailing belief that fresh, on-policy data is essential for high performance. In this work, we challenge this assumption. We present a systematic study of replay buffers for LLM post-training, formalizing the optimal design as a trade-off between staleness-induced variance, sample di