MetaVQA (arXiv:2501.09167, 2025) evaluates and improves the embodied scene understanding of Vision-Language Models (VLMs) — their ability to read spatial relations and scene dynamics and act on them — through visual question answering (VQA) auto-generated from scene graphs of real traffic scenarios, validated by closed-loop, safety-critical driving simulation. Its observations, however, are single frames from a single fixed camera perspective. The paper's authors state as an open limitation that embodied agents may need multi-step historical observations to make optimal decisions in complex situations, and that multi-camera observations could provide more contextual information for better decision-making. The open problem is therefore to extend this embodied scene understanding paradigm — scene-graph-derived QA generation with Set-of-Mark visual annotations, and closed-loop safety-critical simulation evaluation — to observations that include multi-step temporal history and multiple camera perspectives, and to determine whether VLMs evaluated and fine-tuned on such observations achieve improved spatial-reasoning accuracy and improved closed-loop driving safety relative to single-frame, single-perspective baselines. Later open-loop benchmarks (TAD, DriveSpatial, nuReasoning, AUTOPILOT-VQA) now cover temporal and multi-view evaluation of driving VLMs, but the closed-loop training-transfer question at the heart of MetaVQA remains unaddressed.

Background

Vision-Language Models (VLMs) — large multimodal models that interpret images jointly with text — are increasingly proposed as the perception and reasoning backbone of embodied agents such as autonomous vehicles. Whether a VLM truly understands a driving scene, rather than merely describing it, is usually probed with visual question answering (VQA): questions about object positions, orientations, distances, and motion whose answers are checked against ground truth. Most driving VQA benchmarks (for example those built on nuScenes, a large multimodal autonomous-driving dataset with camera, LiDAR, and annotated object tracks) evaluate models offline on recorded clips, so a model's answers have no consequences for the environment; this leaves untested the interactive, sequential nature of driving. MetaVQA (Wang, Duan, Peng, Liu, and Zhou, 2025) was designed to close that gap. It extracts scene graphs — structured representations encoding objects, their attributes, and their spatial relations — from real traffic scenarios in the nuScenes and Waymo Open Motion Dataset (a large-scale interactive motion dataset for autonomous driving), then automatically generates large numbers of question-answer pairs from these graphs, using Set-of-Mark prompting (a technique that overlays numbered visual markers on image regions so the model can refer to specific objects) to make questions object-centric and grounded. Crucially, MetaVQA also evaluates VLMs as driving agents in closed-loop simulation: the VLM repeatedly observes the scene and chooses driving actions inside the MetaDrive simulator (a lightweight driving simulator that can replay imported real-world scenarios), so its decisions change the future of the episode, and safety can be stress-tested in reconstructed safety-critical situations. Closed-loop performance is measured with standard driving metrics such as route completion, collision rate, and off-road rate. Experiments showed that representative VLMs (GPT-4o, Qwen2, Llama3.2, InternVL2-8B, LLaVA variants) have imperfect zero-shot grounding on these embodied questions (average grounding accuracy about 0.70), that fine-tuning on the auto-generated QA pairs markedly improves both VQA accuracy and closed-loop driving metrics, and that knowledge transfers between simulated and real observations. However, every observation in the dataset is a single image from one fixed camera perspective. The authors state as limitations that embodied agents might need multi-step historical information (past frames or trajectory context) to make optimal decisions in complex situations, and that multi-camera observations could provide more contextual information for better decision-making. This is not a cosmetic data gap: single-frame, single-view observations cannot disambiguate occlusions, depth, or temporal developments (e.g., a vehicle that is about to cut in), which are precisely what sequential safety-critical decisions depend on. The question is whether the embodied scene understanding paradigm — auto-generated grounded QA plus closed-loop safety evaluation — survives and improves when extended to temporal history and multiple perspectives, an observation regime that stresses current VLMs' temporal reasoning and cross-view spatial integration abilities.

Problem Statement

Extend the MetaVQA embodied scene understanding paradigm to richer observations and determine their causal effect on embodied decision-making. Concretely: (1) build a benchmark and training corpus in which the question-answer pairs and the closed-loop agent observations are derived from multi-step temporal history (past frames or explicit trajectory context) and from multiple camera perspectives, with Set-of-Mark-style grounding annotations auto-generated from scene-graph or simulator state as in the original work; (2) evaluate representative VLMs on this corpus in closed-loop, safety-critical driving simulation using the established metrics (route completion, collision rate, off-road rate, displacement errors) alongside VQA accuracy; and (3) determine whether VLMs fine-tuned with temporal, multi-perspective embodied QA observations achieve improved spatial-reasoning accuracy and improved closed-loop driving safety relative to matched single-frame, single-perspective baselines — or demonstrate, through controlled comparison, that such observations do not help, and characterize why (e.g., temporal-integration or cross-view grounding failures). The two observation extensions (temporal history and multi-camera views) may be addressed separately or together: the temporal-only, multi-camera-only, and joint temporal-plus-multi-camera conditions each independently count as a resolving answer, each compared against its own matched single-frame, single-perspective baseline (the same corpus with that added extension ablated). In every case the closed-loop safety consequence of the learned scene understanding must be measured, since that is the embodied claim the original benchmark was built to test.

References

  1. Weizhen Wang, Chenda Duan, Zhenghao Peng, Yuxin Liu, Bolei Zhou. Embodied Scene Understanding for Vision Language Models via MetaVQA. arXiv:2501.09167 (2025). DOI: 10.48550/arXiv.2501.09167. https://arxiv.org/abs/2501.09167
  2. Kevin Cannons, Saeed Ranjbar Alvar, Mohammad Asiful Hossain, Ahmad Rezaei, Mohsen Gholami, Alireza Heidarikhazaei, Zhou Weimin, Yong Zhang, Mohammad Akbari. From Segments to Scenes: Temporal Understanding for Agentic Autonomous Driving via Vision-Language Models. arXiv:2512.05277 (2025). https://arxiv.org/abs/2512.05277
  3. Hao Vo, Khoa Vo, Phu Loc Nguyen, Sieu Tran, Duc Minh Nguyen, Ngo Xuan Cuong, Gladys Gawugah, Sreevenkata Anjani Tishita Godavarthi, Chase Rainwater, Nghi D. Q. Bui, Anh Nguyen, Duy Minh Ho Nguyen, Ngan Le. DriveSpatial: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving. arXiv:2605.23176 (2026). https://arxiv.org/abs/2605.23176
  4. Zhiyu Huang, Johnson Liu, Rui Song, Zewei Zhou, Ruining Yang, Yun Zhang, Tianhui Cai, Hanyin Zhang, Mingxuan Gao, Valeria Xu, Jiali Chen, Yishan Shen, Yiluan Guo, Tony (Xuewei) Qi, Jiaqi Ma. nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving. arXiv:2605.31572 (2026). https://arxiv.org/abs/2605.31572
  5. Siddharth Damodharan, Radhika Gupta, Ali Alshami, Ryan Rabinowitz, Jugal Kalita. AUTOPILOT VQA: Benchmarking Vision-Language Models for Incident-Centric Dashcam Understanding. arXiv:2607.08745 (2026). https://arxiv.org/abs/2607.08745
  6. Xianda Guo, Ruijun Zhang, Yiqun Duan, Yuhang He, Dujun Nie, Wenke Huang, Chenming Zhang, Shuai Liu, Hao Zhao, Long Chen. SURDS: Benchmarking Spatial Understanding and Reasoning in Driving Scenarios with Vision Language Models (originally released as DriveMLLM). arXiv:2411.13112 (2024, revised 2025). https://arxiv.org/abs/2411.13112
  7. Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, Oscar Beijbom. nuScenes: A multimodal dataset for autonomous driving. arXiv:1903.11027 (2019). https://arxiv.org/abs/1903.11027
  8. Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles Qi, Yin Zhou, Zoey Yang, Aurelien Chouard, Pei Sun, Jiquan Ngiam, Vijay Vasudevan, Alexander McCauley, Jonathon Shlens, Dragomir Anguelov. Large Scale Interactive Motion Forecasting for Autonomous Driving: The Waymo Open Motion Dataset. arXiv:2104.10133 (2021). https://arxiv.org/abs/2104.10133
  9. Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, Bolei Zhou. MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning. arXiv:2109.12674 (2021). https://arxiv.org/abs/2109.12674
  10. Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, Jianfeng Gao. Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V. arXiv:2310.11441 (2023). https://arxiv.org/abs/2310.11441