In reinforcement learning from human feedback (RLHF) with composite reward models, the constrained-RLHF approach of Moskovitz et al. (ICLR 2024) prevents reward-model overoptimization by constraining each component reward to stay below its proxy point — the reward value beyond which further optimization degrades true quality — but locating these proxy points requires periodic queries to a ground-truth evaluation metric (human ratings or held-out proxies). It remains an open problem to identify per-component overoptimization thresholds of composite reward models purely from the proxy reward models themselves or other unsupervised signals, with no access to the true evaluation metric at any stage of threshold identification, and to use them within constrained RLHF.

Background

Reinforcement learning from human feedback (RLHF) is the dominant procedure for aligning large language models (LLMs) with human preferences: human ratings of model outputs are collected, a reward model (RM) — a neural network trained to score outputs — is fitted to them, and the LLM is then fine-tuned with reinforcement learning (RL) against that RM. Because any RM is an imperfect proxy for human judgment, optimizing it too hard eventually degrades true output quality: Gao, Schulman, and Hilton (2022) documented this reward model overoptimization, showing that after a certain point higher proxy reward is accompanied by lower scores under a gold-standard evaluation. Modern practice increasingly derives reward from a composition of several simpler RMs (for example, one scoring helpfulness and another harmlessness), which adds the question of how to weight the components. Moskovitz et al. (2023; published at the International Conference on Learning Representations, ICLR 2024) gave the first systematic study of overoptimization for composite RMs. They define the proxy point of a component RM as the value of that reward beyond which further optimization decreases true quality; they show that correlation between component RMs significantly shifts where these points lie; and they propose constrained RLHF, in which each component reward is treated as a constraint and a constrained Markov decision process (CMDP — an MDP augmented with auxiliary cost constraints that the policy must satisfy) is solved by primal-dual optimization, where Lagrange multipliers (dual variables that adaptively price each constraint) learn the component weights dynamically. One variant identifies proxy points online during a single training run using the derivative-free Nelder–Mead simplex algorithm (a classical hill-climbing method that iteratively reshapes a simplex of candidate points), saving the cost of multiple training runs. Their experiments use two correlated component rewards — the lexical-overlap metric METEOR and a dialogue-intent classifier score — on the DailyDialog corpus of human–human dialogues, with an average of held-out metrics serving as the ground-truth evaluation, in the style of Gao et al. The shared weakness of all their identification methods is that they periodically query this ground-truth metric to locate the proxy points, and the single-run threshold search is untested beyond two component RMs with a relatively small feasible region. Prior work has also developed unsupervised signals that correlate with overoptimization — reward-model ensemble disagreement (Coste et al., 2023), reward-model internal representations (Miao et al., 2024), and training-dynamics statistics (Miao et al., 2025) — but these detect generic hacking events rather than localizing per-component thresholds of composite reward models.

Problem Statement

Develop and validate a procedure that, in RLHF with composite reward models, identifies each component reward model's proxy point — the value of that component reward beyond which further optimization decreases true output quality — using only signals derived from the proxy reward models themselves or other unsupervised sources, with no access to human ratings, gold reward models, or held-out evaluation metrics at any stage of threshold identification. The identified thresholds must then be usable as constraints in constrained RLHF. The ground-truth-free restriction applies to threshold identification; validation may consult the held-out evaluation metrics, but only to check the identified thresholds against each component's true proxy point.

References

  1. Ted Moskovitz, Aaditya K. Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca D. Dragan, Stephen McAleer (2023), 'Confronting Reward Model Overoptimization with Constrained RLHF', arXiv:2310.04373, DOI: 10.48550/arXiv.2310.04373, https://arxiv.org/abs/2310.04373 (published at ICLR 2024, https://openreview.net/forum?id=gkfUvn0fLU)
  2. Leo Gao, John Schulman, Jacob Hilton (2022), 'Scaling Laws for Reward Model Overoptimization', arXiv:2210.10760, DOI: 10.48550/arXiv.2210.10760, https://arxiv.org/abs/2210.10760
  3. Ted Moskovitz, Brendan O'Donoghue, Vivek Veeriah, Sebastian Flennerhag, Satinder Singh, Tom Zahavy (2023), 'ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPs', arXiv:2302.01275, DOI: 10.48550/arXiv.2302.01275, https://arxiv.org/abs/2302.01275
  4. Thomas Coste, Usman Anwar, Robert Kirk, David Krueger (2023), 'Reward Model Ensembles Help Mitigate Overoptimization', arXiv:2310.02743, DOI: 10.48550/arXiv.2310.02743, https://arxiv.org/abs/2310.02743
  5. Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, Zhaoran Wang (2024), 'Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer', arXiv:2405.16436, DOI: 10.48550/arXiv.2405.16436, https://arxiv.org/abs/2405.16436
  6. Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, Dacheng Tao (2024), 'InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling', arXiv:2402.09345, DOI: 10.48550/arXiv.2402.09345, https://arxiv.org/abs/2402.09345
  7. Yuchun Miao, Sen Zhang, Liang Ding, Yuqi Zhang, Lefei Zhang, Dacheng Tao (2025), 'The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking', arXiv:2501.19358, DOI: 10.48550/arXiv.2501.19358, https://arxiv.org/abs/2501.19358
  8. Yuchun Miao, Liang Ding, Sen Zhang, Rong Bao, Lefei Zhang, Dacheng Tao (2025), 'Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hacking', arXiv:2510.13694, DOI: 10.48550/arXiv.2510.13694, https://arxiv.org/abs/2510.13694
  9. Songtao Wang, Quang Hieu Pham, Fangcong Yin, Xinpeng Wang, Jocelyn Qiaochu Chen, Greg Durrett, Xi Ye (2026), 'Detecting and Suppressing Reward Hacking with Gradient Fingerprints', arXiv:2604.16242, DOI: 10.48550/arXiv.2604.16242, https://arxiv.org/abs/2604.16242
  10. Patrick Wilhelm, Thorsten Wittkopp, Odej Kao (2026), 'Monitoring Emergent Reward Hacking During Generation via Internal Activations', arXiv:2603.04069, DOI: 10.48550/arXiv.2603.04069, https://arxiv.org/abs/2603.04069
  11. Guilin Zhang, Chuanyi Sun, Kai Zhao, Xu Chu, Shahryar Sarkani, John M. Fossaceca (2026), 'EvalStop: Using World Feedback to Detect and Correct Reward Overoptimization in Multi-Tenant RLHF Platforms', arXiv:2606.04145, DOI: 10.48550/arXiv.2606.04145, https://arxiv.org/abs/2606.04145
  12. Xuekang Wang, Zhuoyuan Hao, Shuo Hou, Hao Peng, Juanzi Li, Xiaozhi Wang (2026), 'Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learning', arXiv:2606.04923, DOI: 10.48550/arXiv.2606.04923, https://arxiv.org/abs/2606.04923
  13. Xiaohua Wang, Muzhao Tian, Yuqi Zeng, Zisu Huang, Jiakang Yuan, Bowen Chen, Jingwen Xu, Mingbo Zhou, Wenhao Liu, Muling Wu, Zhengkang Guo, Qi Qian, Yifei Wang, Feiran Zhang, Ruicheng Yin, Shihan Dou, Changze Lv, Tao Chen, Kaitao Song, Xu Tan, Tao Gui, Xiaoqing Zheng, Xuanjing Huang (2026), 'Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges', arXiv:2604.13602, DOI: 10.48550/arXiv.2604.13602, https://arxiv.org/abs/2604.13602
  14. Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, Scott Niekum (2024), 'Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms', arXiv:2406.02900, DOI: 10.48550/arXiv.2406.02900, https://arxiv.org/abs/2406.02900
  15. Lorenz Wolf, Robert Kirk, Mirco Musolesi (2025), 'Reward Model Overoptimisation in Iterated RLHF', arXiv:2505.18126, DOI: 10.48550/arXiv.2505.18126, https://arxiv.org/abs/2505.18126
  16. Jiayi Fu, Xuandong Zhao, Chengyuan Yao, Heng Wang, Qi Han, Yanghua Xiao (2025), 'Reward Shaping to Mitigate Reward Hacking in RLHF', arXiv:2502.18770, DOI: 10.48550/arXiv.2502.18770, https://arxiv.org/abs/2502.18770
  17. Zhuohao Yu, Zhiwei Steven Wu, Adam Block (2026), 'From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimism', arXiv:2604.04648, DOI: 10.48550/arXiv.2604.04648, https://arxiv.org/abs/2604.04648
  18. Zihao Wang, Chirag Nagpal, Jonathan Berant, Jacob Eisenstein, Alex D'Amour, Sanmi Koyejo, Victor Veitch (2024), 'Transforming and Combining Rewards for Aligning Large Language Models', arXiv:2402.00742, DOI: 10.48550/arXiv.2402.00742, https://arxiv.org/abs/2402.00742
  19. Yuhang Lai, Siyuan Wang, Shujun Liu, Xuanjing Huang, Zhongyu Wei (2024), 'ALaRM: Align Language Models via Hierarchical Rewards Modeling', arXiv:2403.06754, DOI: 10.48550/arXiv.2403.06754, https://arxiv.org/abs/2403.06754
  20. Tengyu Xu, Eryk Helenowski, Karthik Abinav Sankararaman, Di Jin, Kaiyan Peng, Eric Han, Shaoliang Nie, Chen Zhu, Hejia Zhang, Wenxuan Zhou, Zhouhao Zeng, Yun He, Karishma Mandyam, Arya Talabzadeh, Madian Khabsa, Gabriel Cohen, Yuandong Tian, Hao Ma, Sinong Wang, Han Fang (2024), 'The Perfect Blend: Redefining RLHF with Mixture of Judges', arXiv:2409.20370, DOI: 10.48550/arXiv.2409.20370, https://arxiv.org/abs/2409.20370
  21. Botong Zhang, Shuo Li, Ignacio Hounie, Osbert Bastani, Dongsheng Ding, Alejandro Ribeiro (2025), 'Alignment of large language models with constrained learning', arXiv:2505.19387, DOI: 10.48550/arXiv.2505.19387, https://arxiv.org/abs/2505.19387
  22. Daniel Yang, Samuel Stante, Florian Redhardt, Lena Libon, Parnian Kassraie, Ido Hakimi, Barna Pásztor, Andreas Krause (2026), 'RewardUQ: A Unified Framework for Uncertainty-Aware Reward Models', arXiv:2602.24040, DOI: 10.48550/arXiv.2602.24040, https://arxiv.org/abs/2602.24040
  23. Sunghwan Kim, Dongjin Kang, Taeyoon Kwon, Hyungjoo Chae, Dongha Lee, Jinyoung Yeo (2025), 'Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimization', DOI: 10.18653/v1/2025.acl-long.649, https://aclanthology.org/2025.acl-long.649/ (preprint arXiv:2505.12763, https://arxiv.org/abs/2505.12763)
  24. Sungdong Kim, Minjoon Seo (2024), 'Rethinking the Role of Proxy Rewards in Language Model Alignment', arXiv:2402.03469, DOI: 10.48550/arXiv.2402.03469, https://arxiv.org/abs/2402.03469