Careful AI agents will betray their partners even faster when things get unpredictable.
April 20, 2026
Original Paper
The Price of Paranoia: Robust Risk-Sensitive Cooperation in Non-Stationary Multi-Agent Reinforcement Learning
arXiv · 2604.15695
The Takeaway
Applying standard risk-sensitive updates to AI agents in cooperative games actually accelerates the collapse of the partnership. This paradox shows that being cautious against a partner potential mistakes creates a feedback loop of mutual distrust. The instability region of the game expands when agents try to hedge their bets against uncertainty. Conventional wisdom suggests that being careful protects the system, but it actually destroys the foundation of cooperation. Designers must build agents that are willing to take risks on their partners to maintain long-term stability.
From the abstract
Cooperative equilibria are fragile. When agents learn alongside each other rather than in a fixed environment, the process of learning destabilizes the cooperation they are trying to sustain: every gradient step an agent takes shifts the distribution of actions its partner will play, turning a cooperative partner into a source of stochastic noise precisely where the cooperation decision is most sensitive. We study how this co-learning noise propagates through the structure of coordination games,