Overoptimisation dynamics and stability of iterated RLHF beyond controlled small-scale settings

Reinforcement learning from human feedback (RLHF) aligns large language models with human preferences but is prone to reward-model overoptimisation: the policy exploits idiosyncrasies of the learned proxy reward, so proxy scores rise while true quality degrades. Repeatedly retraining the reward model on freshly collected feedback and re-optimising the policy (iterated RLHF) is the standard mitigation, yet its dynamics were first studied systematically by Wolf, Kirk and Musolesi (2025), who found on the simulated AlpacaFarm benchmark that overoptimisation decreases over iterations while gains plateau after roughly three, that concatenating all preference data across iterations is the most robust transfer but costs training compute that grows with iteration count, that re-initialising the policy from the base model is robust but limits optimisation flexibility, and that some random seeds become strongly over-optimised under warm-started initialisation and are not recovered by the respective methods — all at small scale (Pythia-410M policies, 70M and 160M reward models, a static simulated gold reward model, one dataset, four iterations). The open problem is to establish whether these regularities persist under realistic conditions and whether iterated RLHF can be made provably or reliably stable: at substantially larger policy and reward-model scales, on diverse task distributions, with real noisy and non-stationary human annotators, at iteration depths well beyond four, at bounded computational cost, and under theoretical conditions that hold for nonconvex neural reward models and PPO-style optimisers rather than the strong-convexity and Lipschitz assumptions of the existing contraction argument.

Background

Reinforcement learning from human feedback (RLHF) is the dominant method for aligning large language models with human preferences: humans rank pairs of model outputs, a reward model is trained on these preferences (classically via the Bradley–Terry pairwise-comparison model, which converts pairwise preference labels into a scalar score for each output), and the language-model policy is fine-tuned with reinforcement learning — typically PPO, proximal policy optimisation, a clipped-gradient policy-gradient algorithm — to maximise the reward score under a penalty for diverging from the initial policy, measured by the Kullback–Leibler (KL) divergence. Because the learned reward is only a proxy for true human utility, aggressive optimisation makes the policy exploit the proxy's idiosyncrasies (verbosity, stylistic shortcuts): proxy reward keeps rising while true quality degrades. This failure, called reward-model overoptimisation (or reward hacking), is an instance of Goodhart's law — once a proxy measure becomes the target of optimisation, it stops tracking the quantity it was meant to approximate — and was quantified for RLHF by Gao, Schulman and Hilton (2023), whose scaling-law experiments also indicate that policy size is not the main driver of its severity. Proposed mitigations include reward-model ensembles (Coste, Anwar, Kirk and Krueger, 2024) and weight-averaged reward models trained at production scale (Ramé et al., 2024, WARM).

A common practical mitigation is iterated RLHF: after each round of policy optimisation, fresh preference data is collected on the current policy's outputs, the reward model is retrained, and the policy is re-optimised. Wolf, Kirk and Musolesi (2025) presented the first controlled study of overoptimisation in this iterated setting, using the AlpacaFarm simulation framework (Dubois et al., 2023), in which a fixed simulated 'gold' reward model stands in for true human preference and a proxy reward model is retrained each iteration on simulated preferences about the current policy's outputs. They systematically varied how preference data is transferred across iterations (take last, concatenate all, or combine reward models in function or parameter space), which reward function is optimised, and how the policy is initialised (from the base supervised model or warm-started from the previous iterate, with interpolation variants). Key findings: reward models become increasingly robust across iterations, so overoptimisation diminishes, but performance gains plateau after roughly three iterations; concatenating all preference data is the most robust transfer but requires retraining on a dataset that grows linearly with the number of iterations; re-initialising from the base policy is the most robust initialisation but limits per-iteration optimisation, and warm-started policies often cannot recover from early overoptimisation; some random seeds become strongly over-optimised under warm-started initialisation (Take last, linear interpolation) and are not recovered by the respective methods, while concatenating all preference data with re-initialisation from the base policy is the most robust configuration studied. These experiments used Pythia-410M policies (Pythia is an open suite of language models for scaling analysis) with 70M and 160M reward models, a single instruction-following dataset, a static gold model, and four iterations.

The study also gives a theoretical frame by casting iterated RLHF as performative prediction (Perdomo et al., 2020) — the framework in which a model's training distribution depends on the model itself. Writing $\phi$ for reward-model parameters and $D(\phi)$ for the data distribution induced by optimising a policy against reward model $\phi$, their Theorem A.2 recovers the standard contraction result: if the per-example loss is $\alpha$-strongly convex and $\beta$-smooth in $\phi$ and the map $\phi\mapsto D(\phi)$ is $L$-Lipschitz, then $L\beta/\alpha<1$ makes the iterated retraining map a contraction, giving linear convergence to a unique performatively stable reward model (a reward model that is optimal for the data distribution that its own deployment induces). The paper's own discussion notes both hypotheses fail in practice: neural reward losses are not globally strongly convex, and PPO-type optimisers can induce highly non-Lipschitz changes in $D(\phi)$. Idealised iterative preference-learning theory with finite-sample guarantees exists only under strong simplifications (KL-regularised contextual-bandit formulations — simplified reinforcement-learning settings in which the agent picks an action given the current context and observes feedback only for that action — with exact or well-approximated optimisation oracles; Xiong et al., 2023), not for nonconvex neural reward models combined with approximate policy optimisers. This leaves a gap between the empirical design recommendations and any guarantee that iterated RLHF remains stable — especially since later mechanism-design analysis (Gauthier, Bach and Jordan, 2026) shows that standard iterative RLHF, viewed as a Stackelberg game (a two-stage leader–follower game in which one player commits first and the other best-responds) between policy and reward model, omits a 'parameter-steering' term and can provably suffer alignment collapse, in which the policy exploits the reward model's blind spots and the retraining feedback reinforces those very errors.

Problem Statement

The problem, as posed by the unresolved questions of Wolf, Kirk and Musolesi (2025), is to determine when iterated RLHF is stable against reward-model overoptimisation under realistic deployment conditions. Its facets share one objective — converting the small-scale empirical design recommendations into reliable, scalable iterated-RLHF pipelines — and are the source's separate open directions toward it:

(1) Persistence at scale and depth. Establish whether the study's empirical regularities persist under conditions substantially beyond the experimental envelope: (i) overoptimisation decreases across iterations, with true-preference (gold-score) gains plateauing after roughly three iterations; (ii) concatenating preference data from all iterations is the most robust data-transfer strategy and outperforms cheaper reward-model combination methods; (iii) re-initialising the policy from the base supervised model is the most robust policy initialisation, while warm-started policies often fail to recover from early overoptimisation; (iv) some random seeds become strongly over-optimised under warm-started initialisation strategies (Take last, linear interpolation) and are not recovered by the respective methods. Do these hold for substantially larger policy and reward-model scales, diverse task distributions beyond a single instruction-following benchmark, and real human annotators whose preferences are noisy and non-stationary (replacing the static simulated gold reward model)? And what happens at iteration depths well beyond four — further convergence, plateau, new failure modes, or cyclic/oscillatory behaviour of the induced data distribution $D(\phi)$?

(2) Stability theory under realistic assumptions. The only available guarantee is a contraction argument (Theorem A.2 of the source): if the reward-model loss is $\alpha$-strongly convex and $\beta$-smooth in the reward-model parameters $\phi$ (curvature bounded from below and above, respectively) and the induced data distribution map $\phi\mapsto D(\phi)$ is $L$-Lipschitz (a bounded parameter change moves the induced distribution by at most a proportional amount), then $L\beta/\alpha<1$ guarantees linear convergence of iterated retraining to a unique performatively stable reward model — hypotheses violated by nonconvex neural reward models and PPO-style policy optimisers. Provide a stability or convergence guarantee (or a rigorous demonstration of instability) for iterated RLHF under assumptions that hold in, or can be practically enforced on, realistic pipelines — and identify which practical mechanisms (data aggregation, reward-model ensembles or weight averaging, policy resets, steering-aware regularisation) provably realise the guarantee.

(3) Robustness at bounded cost. Design and validate schemes — improved parameter-space reward-model combination, distilled or compressed ensemble representations (a single compact model trained to imitate a reward-model ensemble), or better data-transfer strategies — that achieve concatenation-level robustness against overoptimisation without training compute growing linearly with the number of iterations.

The overarching question is which of the source study's design recommendations survive (1), what the theory of (2) establishes about them, and which of the cost-bounded mechanisms of (3) realise a stable pipeline, for iterated RLHF as used in practice. The three facets are the source's separate open directions, not a conjunctive all-or-nothing completion rule.

References

1. Lorenz Wolf, Robert Kirk, and Mirco Musolesi. Reward Model Overoptimisation in Iterated RLHF. arXiv:2505.18126 (2025). DOI: 10.48550/arXiv.2505.18126. https://arxiv.org/abs/2505.18126
2. Leo Gao, John Schulman, and Jacob Hilton. Scaling Laws for Reward Model Overoptimization. arXiv:2210.10760 (2022; ICML 2023). DOI: 10.48550/arXiv.2210.10760. https://arxiv.org/abs/2210.10760
3. Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward Model Ensembles Help Mitigate Overoptimization. arXiv:2310.02743 (2023; ICLR 2024). DOI: 10.48550/arXiv.2310.02743. https://arxiv.org/abs/2310.02743
4. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback. arXiv:2305.14387 (2023; NeurIPS 2023). DOI: 10.48550/arXiv.2305.14387. https://arxiv.org/abs/2305.14387
5. Juan C. Perdomo, Tijana Zrnic, Celestine Mendler-Dünner, and Moritz Hardt. Performative Prediction. arXiv:2002.06673 (2020; ICML 2020). DOI: 10.48550/arXiv.2002.06673. https://arxiv.org/abs/2002.06673
6. Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. WARM: On the Benefits of Weight Averaged Reward Models. arXiv:2401.12187 (2024; ICML 2024). DOI: 10.48550/arXiv.2401.12187. https://arxiv.org/abs/2401.12187
7. Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint. arXiv:2312.11456 (2023; ICML 2024). DOI: 10.48550/arXiv.2312.11456. https://arxiv.org/abs/2312.11456
8. Etienne Gauthier, Francis Bach, and Michael I. Jordan. Explaining and Preventing Alignment Collapse in Iterative RLHF. arXiv:2605.04266 (2026). DOI: 10.48550/arXiv.2605.04266. https://arxiv.org/abs/2605.04266
9. Xiaohua Wang, Muzhao Tian, Yuqi Zeng, Zisu Huang, Jiakang Yuan, Bowen Chen, Jingwen Xu, Mingbo Zhou, Wenhao Liu, Muling Wu, Zhengkang Guo, Qi Qian, Yifei Wang, Feiran Zhang, Ruicheng Yin, Shihan Dou, Changze Lv, Tao Chen, Kaitao Song, Xu Tan, Tao Gui, Xiaoqing Zheng, and Xuanjing Huang. Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges. arXiv:2604.13602 (2026). DOI: 10.48550/arXiv.2604.13602. https://arxiv.org/abs/2604.13602