Hansen and Wang (2026) argue that hallucination in generative world models — visually fluent rollouts that drift from ground-truth dynamics — is primarily a data-coverage problem, and demonstrate this at the 350M-parameter scale on 210 simulated control tasks: hallucination concentrates in low-coverage state-action regions, is predictable at runtime from three lightweight data-centric signals (tokenizer round-trip residual, flow instability, and inter-seed denoising variance, tracking failure with rank correlation approximately 0.80 against rollout error), and is preventable through coverage-aware sampling and predictor-driven targeted data collection. The paper's Limitations section explicitly poses the open empirical question of whether these findings translate to billion-parameter models or to real robot data with sensor noise and partial observability.

Background

Generative world models are neural networks trained to predict future observations of an environment conditioned on actions; in model-based reinforcement learning and robot learning they are used for planning (for example model-predictive control, which repeatedly optimizes a short action sequence against predicted futures), policy evaluation, and synthetic data generation. A recurring failure of these models is hallucination: rollouts that remain visually fluent and superficially plausible while drifting from the true dynamics of the environment — a term borrowed from the language-modeling literature, where it denotes fluent but factually wrong generation. Because hallucinated trajectories are fed directly to downstream planners and policies, silent hallucination during rollout translates into silently incorrect control decisions.

Hansen and Wang (2026) study this failure mode in a controlled setting. They introduce MMBench2, a 427-hour dataset for visual world modeling that extends the earlier MMBench multitask benchmark (Hansen, Su, and Wang, 2025) and spans 210 simulated control tasks (200 for pretraining, 10 held out as entirely unseen transfer tasks) with ground-truth actions, rewards, and live simulators, and they train a 350M-parameter world model of the Dreamer 4 family (a model-based reinforcement-learning architecture, due to Hafner, Yan, and Lillicrap, 2025, whose agent learns a world model and trains an actor-critic policy entirely inside it) on this corpus. The model follows the Dreamer 4 recipe: a video tokenizer — an encoder-decoder trained by masked auto-encoding — first compresses each frame into spatial latent tokens, and a Transformer dynamics model trained via flow matching (a diffusion-style generative objective in which the model learns to produce predictions by iterative denoising) predicts future latent tokens conditioned on actions. They identify three hallucination modes, each anchored to a stage of this pipeline: perceptual hallucination, in which the tokenizer reconstructs out-of-distribution scenes as similar scenes seen during training (for example, an unseen maze layout reconstructed as a familiar one); action marginalization, in which the predicted video is plausible but ignores the action actually commanded; and scene divergence, in which multi-step rollouts drift into implausible dynamics (for example an object teleporting back into play). Their central claim is that hallucination is primarily a data-coverage problem: it concentrates in low-coverage regions of the state-action space (regions underrepresented in the training data) rather than being a purely architectural deficiency. Accordingly, they show that hallucination is predictable at runtime from three lightweight, data-centric signals — the tokenizer round-trip residual (the latent-space discrepancy between the dynamics model's predicted next latent and its decode-encode round trip), flow instability (how much the denoiser's predicted clean target moves between successive denoising substeps), and inter-seed denoising variance (disagreement between generations produced from different random seeds) — which track failure locations with a strong Spearman rank correlation (approximately 0.80 in magnitude) against the drop in PSNR (peak signal-to-noise ratio, a standard pixel-level fidelity measure) between predicted and ground-truth rollouts. They further show that hallucination is preventable by coverage-targeted data interventions: coverage-aware sampling during pretraining reduces all three failure modes simultaneously, and using the predictors as curiosity rewards (intrinsic rewards that drive targeted online data collection where predicted hallucination is high) adapts the pretrained model to entirely unseen environments with as few as 50 real environment trajectories, approaching the effectiveness of human data collection.

All of this evidence comes from one regime: a single 350M-parameter model trained entirely on simulated, fully observed, noise-free tasks. The paper's Limitations section states the resulting open empirical question: whether these findings translate to billion-parameter models, or to real robot data with sensor noise and partial observability. The same discussion section also raises two adjacent threads: how world models should be evaluated given that downstream tasks are goal-directed and exercise a much narrower state-action distribution than the training corpus (so closed-loop evaluations measure accuracy only in a subspace), and how tokenizer generalization should be handled, given that off-the-shelf tokenizers trained on much larger corpora mitigate perceptual hallucination on unseen tasks while in-domain finetuning still wins when it is possible.

Problem Statement

Does the coverage-based account of world-model hallucination established by Hansen and Wang (2026) at the 350M-parameter scale on 210 simulated control tasks generalize beyond that regime?

(1) Scale. When a world model is trained at billion-parameter scale — on the order of 10910^9 parameters, comparable to the largest publicly trainable action-conditioned video world models — does hallucination still concentrate in low-coverage regions of the state-action space; do the three failure modes (perceptual, action-marginalized, and scene-diverging) remain an adequate taxonomy of how the model fails (every observed failure mode classifiable as one of the three); and do lightweight, data-centric runtime signals — the tokenizer round-trip residual, flow instability, and inter-seed denoising variance, or faithful re-derivations appropriate to the architecture at hand — still predict where rollouts will hallucinate, with a rank correlation against ground-truth rollout error comparable to the approximately 0.80 measured at 350M scale?

(2) Realism. When such a world model is trained or adapted on real robot data exhibiting sensor noise and partial observability, does hallucination remain predictable from such signals, and do coverage-targeted interventions — coverage-aware sampling of training data and predictor-driven targeted online data collection — still measurably mitigate hallucination relative to matched controls?

The question is resolved, for at least one of the two regimes — billion-parameter scale, or real robot data with sensor noise and partial observability — by settling all of the sub-questions listed under that regime, quantifying the strength of the predictors against rollout error and characterizing where and why the account holds or breaks.

References

  1. Nicklas Hansen, Xiaolong Wang. "Hallucination in World Models is Predictable and Preventable." arXiv:2606.27326 (2026). DOI: 10.48550/arXiv.2606.27326. https://arxiv.org/abs/2606.27326
  2. Peterson Co, Sicheng Hu, Chunxuan Jiao, Hongyang Cheng, Yulin Luo, Yijie Xu, Sixiang Chen, Zhongxia Zhao, Zihao Wang, DaFeng Chi, Peidong Liu, YuTong Chen, Henghua Liu, Zhihao Yuan, Huizhu Jia, Yuzheng Zhuang, Tianle Zhang, Liang Lin, Huajie Tan, Shanghang Zhang. "WorldSimProbe: Diagnosing Simulator Faithfulness in Action-Conditioned World Models for Embodied Manipulation." arXiv:2608.09298 (2026). https://arxiv.org/abs/2608.09298
  3. Quanquan Peng, Yutong Liang, Rui Yan, Nicklas Hansen, Xiaolong Wang. "FACT: Failure-Aware Causal Training for World-Action Models." arXiv:2608.10232 (2026). https://arxiv.org/abs/2608.10232
  4. Yuquan Xue, Le Xu, Zeyi Liu, Zhenyu Wu, Zhengyi Gu, Xinyang Song, Bofang Jia, Ziwei Wang. "WorldSample: Closed-loop Real-robot RL with World Modelling." arXiv:2607.02431 (2026). https://arxiv.org/abs/2607.02431
  5. Zirui Zheng, Jiaqian Yu, Xiongfeng Peng, Jun Shi, Mingyi Li, Chao Zhang, Weiming Li, Dong Wang, Huchuan Lu, Xu Jia. "Mem-World: Memory-Augmented Action-Conditioned World Models for Persistent Robot Manipulation." arXiv:2606.18960 (2026). https://arxiv.org/abs/2606.18960
  6. Fangyuan Wang, Ziyuan Wang, Guorui Pei, Mengshi Zhang, Canxi Liang, Jun Hu, Zhongxuan Li, Jinsong Wu, Ning Han, Zeqing Zhang, Jiaming Qi, Hongmin Wu, Shiyao Zhang, Pai Zheng, Jia Pan, David Navarro-Alarcon, Sichao Liu, Peng Zhou. "World Models for Robotic Manipulation: A Survey." arXiv:2606.00113 (2026). https://arxiv.org/abs/2606.00113
  7. Haiyu Wu, Randall Balestriero, Morgan Levine. "VIScore: Diagnosing Planning-Relevant Quality in Latent World Models." arXiv:2608.11174 (2026). https://arxiv.org/abs/2608.11174
  8. Danijar Hafner, Wilson Yan, Timothy Lillicrap. "Training Agents Inside of Scalable World Models." arXiv:2509.24527 (2025). https://arxiv.org/abs/2509.24527
  9. Nicklas Hansen, Hao Su, Xiaolong Wang. "Learning Massively Multitask World Models for Continuous Control." arXiv:2511.19584 (2025). https://arxiv.org/abs/2511.19584