Reward models (RMs) score language-model outputs and drive reinforcement learning from human feedback (RLHF), yet static RM benchmarks correlate only weakly with the performance of policies optimized against the RM. Kim et al. (ACL 2025) proposed the degree of overoptimization γ\gamma — the normalized area between the gold-reward and proxy-reward curves plotted against policy KL divergence — as an RM-evaluation criterion that strongly predicts downstream policy performance, but measuring γ\gamma for a single RM cost over 500,000 inferences in their experiments, and γ\gamma itself depends on the RL algorithm and hyperparameters, so engineering benchmarks toward maximal correlation with γ\gamma invites Goodhart-style overfitting. The open problem is to make overoptimization-anchored RM evaluation practical: develop a cost-efficient benchmark score, cheap proxy, or analytic approximation anchored on γ\gamma that provides reliable RM evaluation at a small fraction of the cost of direct measurement, heeding the paper's conclusion that the degree of overoptimization should be used as a tool to validate benchmark reliability rather than treated as an end goal.

Background

Reinforcement learning from human feedback (RLHF) aligns large language models by training a reward model (RM) on human preference comparisons and then optimizing the language model (the policy) to maximize the RM's score, for example with PPO (proximal policy optimization, a policy-gradient RL algorithm) or best-of-nn sampling (BoN, generating nn responses per prompt and selecting the one the RM scores highest). Because the RM is only a proxy for true human preference, sustained optimization against it can raise the proxy score while true quality falls — reward overoptimization, an instance of Goodhart's law ('when a measure becomes a target, it ceases to be a good measure'). In the controlled setup introduced by Gao, Schulman, and Hilton (ICML 2023), a 'gold' reward model stands in for true preference while the policy optimizes a separate 'proxy' RM; plotting gold reward against the KL divergence DKL(π∥πinit)D_{\mathrm{KL}}(\pi\|\pi_{\mathrm{init}}) (Kullback–Leibler divergence, a standard measure of how far the optimized policy's output distribution has moved from the initial policy's) shows the gold reward rising and then declining as optimization proceeds.

Existing RM benchmarks such as RewardBench score RMs by accuracy on static chosen/rejected preference pairs, and these scores correlate only weakly with the downstream performance of policies optimized against the RM. Kim et al. (ACL 2025) therefore proposed evaluating RMs through the degree of overoptimization γ\gamma: fit the BoN reward curve R(x)=x(α−βx)R(x)=x(\alpha-\beta x), with x=DKL(π∥πinit)x=\sqrt{D_{\mathrm{KL}}(\pi\|\pi_{\mathrm{init}})}, separately for a reference reward (curve ff) and the proxy RM under evaluation (curve gg), and define

γ=∫0k∣f(x)−g(x)∣ dx∫0kf(x) dx\gamma=\frac{\int_{0}^{k}|f(x)-g(x)|\,dx}{\int_{0}^{k}f(x)\,dx}

where kk represents the largest DKL(π∥πinit)D_{\mathrm{KL}}(\pi\|\pi_{\mathrm{init}}) observed; larger γ\gamma means the RM's reward signal degrades faster under optimization. In the source, γ\gamma is measured against a reference reward — either a designated gold RM (giving γgold\gamma_{\text{gold}}; Skywork-o1-Open-PRM-Qwen-2.5-7B, whose own γgold\gamma_{\text{gold}} is 0 by construction) or oracle accuracy (giving γoracle\gamma_{\text{oracle}}) — and its value depends on the policy model being optimized: Skywork-Reward-Llama-3.1-8B-v0.2 received γgold=0.663\gamma_{\text{gold}}=0.663 with MetaMATH-Mistral-7B and 0.4270.427 with Llama3-8B-Instruct, so γ\gamma is not a single context-free scalar attached to an RM. They found γ\gamma strongly correlated with downstream performance (r2>0.7r^{2}>0.7), supporting γ\gamma as an RM-evaluation criterion, and identified three benchmark-design factors for reliable, lower-cost evaluation: small distributional differences between chosen and rejected responses beyond correctness, response diversity (responses sourced from many models), and multiple comparisons across a wide range of chosen and rejected responses — with the accuracy metric capturing γ\gamma itself more effectively in their metric comparison, while their multi-pairwise 'matrix' metric better tracks downstream PPO performance.

Three obstacles keep this from being a usable evaluation methodology, and the paper states them as open. First, cost: directly measuring γ\gamma required over 500,000 inferences per RM in their experiments, because the BoN reward curve must be traced over the full KL range for both gold and proxy RMs; cheap, reliable proxies or analytic approximations to γ\gamma do not exist. Second, setting dependence and Goodhart effects: γ\gamma is influenced by the RL algorithm and hyperparameters, and the paper observes that the benchmark designs with the highest correlation to γ\gamma can show reduced correlation with downstream performance — one design with correlation close to 0.9 with γ\gamma reached only r2=0.74r^{2}=0.74 with downstream PPO performance, while a less γ\gamma-tuned design achieved r2>0.83r^{2}>0.83 — so a benchmark engineered toward γ\gamma may be valid only for specific RL configurations, and principled objective/regularizer selection that avoids this overfitting is undeveloped. Third, coverage and label noise: the experiments used small-scale RMs and policies, only BoN and PPO as optimization methods, only accuracy and the matrix metric, and in domains such as safety the downstream labels themselves come from noisy LLM judgments, which destabilized the correlation analysis.

Problem Statement

Make overoptimization-anchored reward-model evaluation practical. The source states the problem directly: measuring γ\gamma 'can be computationally expensive; for instance, evaluating a single RM in this experiment requires over 500K inferences', and directly leveraging γ\gamma is 'impractical for benchmarking purposes', whereas 'developing a benchmark that strongly correlates with γ\gamma would not only provide a reliable evaluation but also offer a cost-efficient approach to evaluate RMs'. The open problem is to design and validate such a cost-efficient evaluation — a benchmark score, a cheap proxy, or an analytic approximation anchored on the degree of overoptimization — that retains γ\gamma's reliability as an RM-evaluation criterion, including its predictive power for downstream policy performance, at a small fraction of the inference cost of direct measurement. Can γ\gamma itself be estimated cheaply for a given reward model, and does such an estimate agree with directly measured γ\gamma?

Two hazards documented in the source shape this goal. First, the paper concludes that γ\gamma 'should be used as a tool to validate benchmark reliability rather than treated as an end goal': a benchmark design with correlation close to 0.9 with γ\gamma showed a lower correlation with downstream PPO performance (r2=0.74r^{2}=0.74) than a less γ\gamma-tuned design (r2>0.83r^{2}>0.83), and benchmarks over-tuned to γ\gamma may be valid only for specific RL configurations. Second, in domains such as safety, downstream evaluation labels come from noisy LLM judgments, which destabilized the paper's correlation analysis (an insignificant r2r^{2} there, though a Spearman correlation exceeding 0.6 was still achieved).

The paper leaves the following open as stated limitations and future directions: experiments with larger reward and policy models; alternative optimization methods beyond the BoN sampling and PPO used to assess policy performance; and further investigation of multi-pairwise comparison metrics beyond the accuracy and matrix metrics analyzed. Its code- and safety-domain experiments (Section 6) support generalizing the mathematics-domain findings to other domains, but the scale, algorithm, and metric coverage above remain untested. These untested dimensions delimit the setting of the open problem rather than adding further deliverables.

References

  1. Sunghwan Kim, Dongjin Kang, Taeyoon Kwon, Hyungjoo Chae, Dongha Lee, Jinyoung Yeo. Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimization. ACL 2025. DOI: 10.18653/v1/2025.acl-long.649; arXiv:2505.12763. https://aclanthology.org/2025.acl-long.649/ ; https://arxiv.org/abs/2505.12763
  2. Leo Gao, John Schulman, Jacob Hilton. Scaling Laws for Reward Model Overoptimization. ICML 2023. arXiv:2210.10760. https://arxiv.org/abs/2210.10760
  3. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi. RewardBench: Evaluating Reward Models for Language Modeling. arXiv:2403.13787. https://arxiv.org/abs/2403.13787
  4. Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A. Smith, Hannaneh Hajishirzi, Nathan Lambert. RewardBench 2: Advancing Reward Model Evaluation. arXiv:2506.01937. https://arxiv.org/abs/2506.01937
  5. Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, Juanzi Li. RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style. arXiv:2410.16184. https://arxiv.org/abs/2410.16184
  6. Sunghwan Kim, Dongjin Kang, Taeyoon Kwon, Hyungjoo Chae, Jungsoo Won, Dongha Lee, Jinyoung Yeo. Evaluating Robustness of Reward Models for Mathematical Reasoning. arXiv:2410.01729. https://arxiv.org/abs/2410.01729
  7. Thomas Coste, Usman Anwar, Robert Kirk, David Krueger. Reward Model Ensembles Help Mitigate Overoptimization. arXiv:2310.02743. https://arxiv.org/abs/2310.02743
  8. Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, Scott Niekum. Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms. arXiv:2406.02900. https://arxiv.org/abs/2406.02900
  9. Sebastian Gehrmann. Reward Models are Metrics in a Trench Coat. arXiv:2510.03231. https://arxiv.org/abs/2510.03231
  10. Eddie Landesberg. When LLM Judge Scores Look Good but Best-of-N Decisions Fail. arXiv:2603.12520. https://arxiv.org/abs/2603.12520
  11. Shunyang Luo, Peibei Cao, Zhihui Zhu, Kehua Feng, Zhihua Wang, Keyan Ding. Evaluating Reward Model Generalization via Pairwise Maximum Discrepancy Competitions. arXiv:2601.16987. https://arxiv.org/abs/2601.16987
  12. Tarun Raheja, Nilay Pochhi. From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models. arXiv:2601.06108. https://arxiv.org/abs/2601.06108
  13. Guilin Zhang, Chuanyi Sun, Kai Zhao, Xu Chu, Shahryar Sarkani, John M. Fossaceca. EvalStop: Using World Feedback to Detect and Correct Reward Overoptimization in Multi-Tenant RLHF Platforms. arXiv:2606.04145. https://arxiv.org/abs/2606.04145