Safe out-of-distribution value generalization in offline reinforcement learning beyond the convex hull and its neighborhood

Offline reinforcement learning must evaluate actions the dataset never contains, and the Smooth Bellman Operator (SBO) with its algorithm SQOG showed that out-of-distribution (OOD) $Q$-values inside the convex hull of the dataset and a bounded neighborhood of it (the CHN) can be estimated safely by smoothing them with neighboring in-sample values. However, the safety guarantees hold only under three assumptions: the learned policy stays KL-close to the behavior policy, the evaluated OOD actions lie inside the CHN, and the critic is an infinitely wide two-layer network analyzable in the neural tangent kernel (NTK) regime, while the OOD samples used for smoothing are generated by heuristically tuned noise. The authors identify two directions as open: they state that extending the approach to learn $Q$-values for faraway OOD actions, which fall outside the CHN, 'remains an open challenge in the field', and they defer to future work a systematic investigation of the role of noise in OOD sampling. This problem asks for a theory of safe OOD value generalization along these directions: guarantees or impossibility results for OOD actions far outside the CHN, including how the achievable estimation error grows with distance from the dataset convex hull, and a principled foundation for the OOD sampling distribution.

Background

Offline reinforcement learning (offline RL) trains a decision-making agent from a fixed dataset of previously collected trajectories without further environment interaction. Let $\mathcal{D}$ be such a dataset, collected by an unknown behavior policy $\mu$. An action $a$ at state $s$ is called out-of-distribution (OOD) if the state-action pair $(s,a)$ does not appear in $\mathcal{D}$. Because the learned $Q$-function — the estimated long-term return of taking action $a$ in state $s$ — is supervised only on dataset pairs, bootstrapped temporal-difference updates can assign arbitrarily wrong, typically inflated, values to OOD actions. This failure is known as value overestimation or extrapolation error, and it is the central obstacle of offline RL. Classical remedies constrain the learned policy $\pi$ to stay close to $\mu$ (e.g., BCQ), penalize OOD values during learning (e.g., Conservative Q-Learning), or avoid OOD actions altogether by using only in-sample Bellman targets (e.g., Implicit Q-Learning). Each remedy injects conservatism: genuinely good but unvisited actions are pessimistically undervalued, which limits policy improvement — the over-constraint issue.

A parallel line of work therefore asks when generalization to OOD actions is safe rather than harmful. DOGE (Li et al., 2022) showed, under the neural tangent kernel (NTK) approximation — the regime of infinitely wide two-layer ReLU networks trained with infinitesimally small learning rates, in which network outputs become linear in their parameters — that the extrapolation error of a deep $Q$-function is controlled for inputs inside the convex hull of the dataset. The ICLR 2025 paper 'Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhood' (Yao et al.) extends this geometric analysis to the CHN: the convex hull of the dataset's state-action pairs together with an external neighborhood of radius $r$ (chosen no larger than the hull diameter). Under the NTK regime, the authors prove that $Q$-value differences remain controlled for points inside the CHN (their Proposition 1, generalizing DOGE's theorem), and they introduce the Smooth Bellman Operator (SBO), whose smooth generalization operator replaces the $Q$-value of an OOD state-action pair inside the CHN by the value of a neighboring in-sample pair, so that OOD values are smoothed toward in-sample values during training. Their Theorems 1–3 bound the distance between the empirical SBO backup and the true Bellman backup (the one-step target computed from the Bellman optimality equation), and their practical algorithm SQOG mitigates both over- and underestimation on the D4RL benchmark suites (a standard collection of offline-RL datasets for continuous-control and navigation tasks).

These guarantees rest on three assumptions. First, Theorem 1 assumes the learned policy is kept close to the behavior policy, $\max(KL(\pi,\mu),KL(\mu,\pi))\le\epsilon$ — here $KL$ is the Kullback–Leibler divergence between the two action distributions — which SQOG enforces by coupling to a policy-constraint method such as TD3+BC (Twin Delayed DDPG, a deep deterministic actor-critic, with an added behavior-cloning regularization term); the authors defend this assumption in Appendix E as satisfied by any policy-constraint offline RL algorithm, with conclusions generalizable beyond TD3+BC. Second, all safety statements are local to the CHN: the paper states verbatim that 'extending this approach to learn Q-values for faraway OOD actions, which fall outside the CHN, remains an open challenge in the field'. Third, every result is proved in the NTK regime for infinitely wide two-layer critics rather than the finite-width deep networks used in practice; the authors do not flag this regime as a restriction to remove. In addition, the OOD samples toward which SBO smooths values are generated by clipped Gaussian noise whose type, scale, and clipping are chosen empirically — the authors leave 'a more systematic investigation into the role of noise in OOD sampling and Q-function generalization' to future work — and the paper notes that precisely computing the CHN and identifying all OOD pairs inside it for a given dataset is 'impractical and unnecessary', leaving high-dimensional CHN-region estimation to heuristic noise-based proxies.

The open problem, in the two directions the authors themselves identify, is the theory of this estimator family beyond its local regime: characterizing when smoothing-based OOD value estimation is provably safe for actions far outside the CHN — through guarantees or impossibility results — and giving the OOD-sample generation mechanism a principled foundation. Successor work through 2026 — CSDG, DMG, and ANQ (successor offline-RL algorithms that permit value generalization only within a neighborhood of the dataset) and the multi-agent extension OMGMG — has continued to build near-dataset generalization methods whose analyses retain the same locality, so the question of what is achievable — and what is provably impossible — far outside the CHN remains open.

Problem Statement

Characterize safe out-of-distribution value generalization in offline reinforcement learning beyond the convex hull and its neighborhood, for smoothing-based OOD value estimators of the SBO/SQOG family of Yao et al. (ICLR 2025) — which update the $Q$-values of OOD state-action pairs by smoothing them with neighboring in-sample values under the convex-hull-and-neighborhood (CHN) geometry. The source guarantees (Proposition 1 and Theorems 1–3) are proved in the NTK regime, under the assumption $\max(KL(\pi,\mu),KL(\mu,\pi))\le\epsilon$, and locally to the CHN; this problem targets the two open directions the source itself identifies, asking what is achievable, or provably unachievable, beyond the CHN-locality restriction and the empirically tuned sampling mechanism:

1. Beyond the CHN. Extend safety guarantees, or establish impossibility/lower bounds, for OOD actions far outside the convex hull and its neighborhood, where the source states that learning $Q$-values 'remains an open challenge in the field'; in particular, characterize how the achievable estimation error grows with distance from the dataset convex hull.

2. Principled OOD sampling. Provide a principled theory for the OOD-sample generation mechanism — the choice of noise type, scale, and clipping — replacing the current empirical tuning, under which generated neighbor samples remain inside the CHN in the source geometry yet are maximally informative; the source defers 'a more systematic investigation into the role of noise in OOD sampling and Q-function generalization' to future work.

References

1. Qingmao Yao, Zhichao Lei, Tianyuan Chen, Ziyue Yuan, Xuefan Chen, Jianxiang Liu, Faguo Wu, Xiao Zhang. Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhood. ICLR 2025; arXiv:2506.08417. DOI: 10.48550/arXiv.2506.08417. https://arxiv.org/abs/2506.08417 (source of the open question)
2. Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, Ya-Qin Zhang. When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learning. arXiv:2205.11027, 2022. DOI: 10.48550/arXiv.2205.11027. https://arxiv.org/abs/2205.11027 (DOGE)
3. Yixiu Mao, Qi Wang, Yun Qu, Yuhang Jiang, Xiangyang Ji. Doubly Mild Generalization for Offline Reinforcement Learning. arXiv:2411.07934, 2024. DOI: 10.48550/arXiv.2411.07934. https://arxiv.org/abs/2411.07934 (DMG)
4. Yixiu Mao, Yun Qu, Qi Wang, Xiangyang Ji. Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning. arXiv:2511.02567, 2025. DOI: 10.48550/arXiv.2511.02567. https://arxiv.org/abs/2511.02567 (ANQ)
5. Yi Yang, Zhennan Chen, Mingfeng Lv, Hanlei Li, Zhengsen Ruan, Lvqing Yang. Convex-Hull-Neighborhood Smooth Dual Generalization: Controlling Local Correction Propagation in Offline RL. arXiv:2608.03108, 2026. DOI: 10.48550/arXiv.2608.03108. https://arxiv.org/abs/2608.03108 (CSDG)
6. Yuanrui Duan, Wengang Zhou, Yufeng Shi, Xiancheng Gao, Lin Liu, Houqiang Li. Offline Multi-Agent Reinforcement Learning with Global Moderate Generalization. Proceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2026. DOI: 10.65109/CVXE7640. https://doi.org/10.65109/CVXE7640
7. Scott Fujimoto, David Meger, Doina Precup. Off-Policy Deep Reinforcement Learning without Exploration. arXiv:1812.02900, 2018. DOI: 10.48550/arXiv.1812.02900. https://arxiv.org/abs/1812.02900 (BCQ)
8. Aviral Kumar, Aurick Zhou, George Tucker, Sergey Levine. Conservative Q-Learning for Offline Reinforcement Learning. arXiv:2006.04779, 2020. DOI: 10.48550/arXiv.2006.04779. https://arxiv.org/abs/2006.04779 (CQL)
9. Ilya Kostrikov, Ashvin Nair, Sergey Levine. Offline Reinforcement Learning with Implicit Q-Learning. arXiv:2110.06169, 2021. DOI: 10.48550/arXiv.2110.06169. https://arxiv.org/abs/2110.06169 (IQL)
10. Johan Obando-Ceron, Walter Mayor, Samuel Lavoie, Scott Fujimoto, Aaron Courville, Pablo Samuel Castro. Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents. arXiv:2510.13704, 2025. DOI: 10.48550/arXiv.2510.13704. https://arxiv.org/abs/2510.13704