Adaptation-free heterogeneous collaborative perception under realistic deployment factors: sensor noise, communication noise, localization error, and latency

Collaborative perception lets an autonomous vehicle (the ego agent) improve its 3D object detection by exchanging information with other agents such as nearby vehicles or roadside infrastructure. The ALF framework (Bae and Ahn, 2026) recently made such collaboration possible with auxiliary agents whose configurations — LiDAR beam count, encoder architecture — were never seen during training, requiring zero adaptation at deployment, by lifting compact box-level messages (about 120 bytes per agent per frame) into ego-compatible latent features. However, ALF deliberately isolates configuration heterogeneity and excludes the conditions expected in real deployment: sensor noise, communication noise, localization (relative-pose) error, and communication latency. The open problem is to extend adaptation-free heterogeneous collaborative perception to these realistic deployment factors while preserving zero-shot generalization to unseen agent configurations and low-bandwidth communication.

Background

Collaborative perception is a paradigm in which multiple agents — connected vehicles and infrastructure units in vehicle-to-everything (V2X) communication — exchange information about a scene so that each agent can perceive beyond the range and occlusion limits of its own sensors. Frameworks are commonly categorized by what is exchanged: early fusion shares raw sensor data, intermediate fusion shares latent feature representations, and late fusion shares object-level predictions. A widely used representation for 3D object detection is the bird's-eye-view (BEV) feature map, a top-down spatial grid of learned features produced by an encoder from LiDAR point clouds, with detection quality measured by mean average precision at an intersection-over-union threshold (mAP@0.7). Intermediate fusion strikes a balance between bandwidth and representational fidelity, but it presupposes that all agents share a compatible latent feature space. In practice this fails: an agent configuration — the combination of sensor specifications (e.g., LiDAR beam count) and encoder choices (backbone architecture, voxelization, feature resolution) — determines the feature domain, and deployed agents inevitably differ. Most heterogeneity-handling methods require per-agent preparation: STAMP (Gao et al., 2025) trains lightweight adapter-reverter pairs (small trainable modules that map each agent's features into, and back out of, a shared feature space) per agent model, PolyInter (Xia et al., 2025) learns per-agent prompts (short learned conditioning vectors that steer a shared model to each agent's feature space), UniTrans (Li et al., 2026) and INTACT (Li et al., 2026) reduce but do not eliminate per-agent coupling, and BOLT (Yang et al., 2026) adapts neighboring features online via ego-as-teacher distillation (the ego's fused model supervises online adaptation of the neighboring agents' features). In parallel, a robustness literature studies realistic deployment factors for collaborative perception in settings where agent stacks are known and compatible: CoAlign (Lu et al., 2023) corrects pose errors via agent-object pose graphs, V2X-DGPE (Wang et al., 2025) handles domain gaps and latency/GPS-induced pose errors for trained vehicle-infrastructure pairs, CoAnchor (Li et al., 2026) jointly compensates communication delay and relative-pose noise via object-level spatio-temporal anchors, AFFormer (Zhou et al., 2026) addresses channel impairments in feature-level fusion, and V2VLoc (Lin et al., 2025) provides pose estimates that do not rely on GNSS (Global Navigation Satellite System) localization. ALF (Bae and Ahn, 2026) opened a different axis: an open-world setting in which auxiliary agents with configurations never observed during training ($c_i \notin \mathcal{C}_{\mathrm{seen}}$) may appear after deployment, and the ego must collaborate with them zero-shot, without any retraining or adaptation. ALF achieves this by having each auxiliary agent transmit only a compact set of quantized 3D bounding boxes (position, size, heading, confidence; at most $K_{\max}$ boxes, on the order of 100 bytes per agent per frame), which the ego converts into a pseudo-BEV map and lifts into its own latent space using ego-scene context; on the real-world V2X-Real benchmark it outperforms the strongest prior baseline by 35.91% in relative mAP@0.7 across 64 zero-shot case studies. HyDRA (Song et al., 2026), a concurrent framework from the same group, routes agents identified as heterogeneous by a domain classifier into late fusion and corrects their poses by anchor-guided pose-graph optimization, but it requires each auxiliary agent to transmit feature maps, detections, and poses, and assumes a homogeneous intermediate-fusion group exists to anchor the pose correction. The two research threads — unseen-configuration adaptability and deployment-factor robustness — have not been unified. ALF's authors explicitly state this as their limitation and future work: the paper isolates configuration heterogeneity and does not consider sensor noise, communication noise, localization error, or latency. Notably, ALF's ablation also shows a robustness-adjacent weakness inside its own pipeline: increasing the cap on transmitted boxes beyond $K_{\max} = 20$ degrades mAP, indicating that low-confidence boxes from imperfect detectors inject noise rather than information.

Problem Statement

How can adaptation-free heterogeneous collaborative perception be extended to realistic open-world deployment conditions while preserving its core guarantee — zero-shot collaboration between an ego agent and auxiliary agents whose configurations (sensor specifications such as LiDAR beam count, and encoder architectures) were never seen during training? Concretely: design and validate a collaborative 3D object detection mechanism in which the ego agent consumes messages from unseen-configuration auxiliary agents that are corrupted or degraded by (i) sensor noise on the auxiliary observations, (ii) communication noise on the transmitted messages, (iii) localization error in the relative poses $T_{i\to 1}$ used to place auxiliary detections in the ego frame, and (iv) communication latency that makes auxiliary observations stale. All four factors are mandatory: the mechanism must remain robust to each factor individually, to every pairwise and triple combination, and to all four jointly, not merely accommodate the class of realistic deployment conditions. Robustness is assessed under non-degenerate degradation severities fixed a priori at the stress ranges standard in the collaborative-perception robustness literature (channel and communication impairments per AFFormer, relative-pose and localization error per CoAlign and V2X-DGPE, communication delay per CoAnchor), never at negligible perturbation. 'Clean-condition operation' denotes the same mechanism run with zero degradation; that clean-condition run must reproduce ALF's clean collaboration benefit on V2X-Real by staying within 0.02 mAP@0.7 of ALF's reported clean mAP@0.7 and above the ego-only (no-collaboration) baseline, and under every required degradation setting detection performance (mAP@0.7 on public V2X benchmarks such as V2X-Real) must remain within 0.02 mAP@0.7 of the clean-condition run and above the ego-only baseline, so that the mechanism must genuinely exploit the auxiliary messages rather than ignore them. The low-bandwidth communication budget (on the order of 100 bytes per agent per frame in ALF) must be respected throughout. The problem is the one posed by Bae and Ahn (2026) as their stated limitation and future work: their framework isolates configuration heterogeneity and excludes sensor noise, communication noise, localization error, and latency; extending adaptation-free collaboration to such realistic deployment settings is the open target.

References

1. Hyunchul Bae and Heejin Ahn. Adaptation-Free Heterogeneous Collaborative Perception with Unseen Agent Configurations. arXiv:2605.26642, 2026. DOI: 10.48550/arxiv.2605.26642. https://arxiv.org/abs/2605.26642
2. Minwoo Song, Minhee Kang, and Heejin Ahn. HyDRA: Hybrid Domain-Aware Robust Architecture for Heterogeneous Collaborative Perception. arXiv:2603.23975, 2026. DOI: 10.48550/arxiv.2603.23975. https://arxiv.org/abs/2603.23975
3. Chi Li, Rui Lin, Aobo Ji, and Dongzhu Xu. CoAnchor: Robust Collaborative Perception under Spatio-Temporal Misalignment via Object-Level Anchors. arXiv:2608.21055, 2026 (ACM Multimedia 2026). DOI: 10.48550/arxiv.2608.21055. https://arxiv.org/abs/2608.21055
4. Sichao Wang, Ming Yuan, Chuang Zhang, Qing Xu, Lei He, and Jianqiang Wang. V2X-DGPE: Addressing Domain Gaps and Pose Errors for Robust Collaborative 3D Object Detection. arXiv:2501.02363, 2025. DOI: 10.48550/arxiv.2501.02363. https://arxiv.org/abs/2501.02363
5. Xi Zhou, Tao Huang, Qing-Long Han, Rana Abbas, and Mostafa Rahimi Azghadi. AFFormer: Adaptive Feature Fusion Transformer for V2X Cooperative Perception under Channel Impairments. arXiv:2605.01888, 2026. DOI: 10.48550/arxiv.2605.01888. https://arxiv.org/abs/2605.01888
6. Xiangbo Gao, Runsheng Xu, Jiachen Li, Ziran Wang, Zhiwen Fan, and Zhengzhong Tu. STAMP: Scalable Task And Model-agnostic Collaborative Perception. arXiv:2501.18616, 2025 (ICLR 2025). DOI: 10.48550/arxiv.2501.18616. https://arxiv.org/abs/2501.18616
7. Kang Yang, Tianci Bu, Peng Wang, Deying Li, and Yongcai Wang. BOLT: Online Lightweight Adaptation for Preparation-Free Heterogeneous Cooperative Perception. arXiv:2605.00405, 2026. DOI: 10.48550/arxiv.2605.00405. https://arxiv.org/abs/2605.00405
8. Yuchen Xia, Quan Yuan, Guiyang Luo, Xiaoyuan Fu, Yang Li, Xuanhan Zhu, Tianyou Luo, Siheng Chen, and Jinglin Li. One is Plenty: A Polymorphic Feature Interpreter for Immutable Heterogeneous Collaborative Perception. arXiv:2411.16799, 2025 (CVPR 2025). DOI: 10.48550/arxiv.2411.16799. https://arxiv.org/abs/2411.16799
9. Yifan Lu, Quanhao Li, Baoan Liu, Mehrdad Dianati, Chen Feng, Siheng Chen, and Yanfeng Wang. Robust Collaborative 3D Object Detection in Presence of Pose Errors. arXiv:2211.07214, 2023 (ICRA 2023). DOI: 10.48550/arxiv.2211.07214. https://arxiv.org/abs/2211.07214
10. Chen Li, Shengrong Yuan, Jialong Zuo, Xinzhong Zhu, Nong Sang, and Changxin Gao. INTACT: Ego-Guided Typed Sparse Evidence Retrieval for Heterogeneous Collaborative Perception. arXiv:2606.04437, 2026. DOI: 10.48550/arxiv.2606.04437. https://arxiv.org/abs/2606.04437
11. Yang Li, Weize Li, Quan Yuan, Congzhang Shao, Guiyang Luo, Yunqi Ba, Xuanhan Zhu, Xinyuan Ding, Xiaoyuan Fu, and Jinglin Li. One Model to Translate Them All: Universal Any-to-Any Translation for Heterogeneous Collaborative Perception. arXiv:2605.17907, 2026. DOI: 10.48550/arxiv.2605.17907. https://arxiv.org/abs/2605.17907
12. Armin Maleki and Hayder Radha. HeteroPROPMT: A Real-time and Privacy-Preserving Heterogeneous Collaborative Perception Framework. arXiv:2607.26283, 2026. DOI: 10.48550/arxiv.2607.26283. https://arxiv.org/abs/2607.26283
13. Wenkai Lin, Qiming Xia, Wen Li, Xun Huang, and Chenglu Wen. V2VLoc: Robust GNSS-Free Collaborative Perception via LiDAR Localization. arXiv:2511.14247, 2025 (AAAI 2026). DOI: 10.48550/arxiv.2511.14247. https://arxiv.org/abs/2511.14247
14. Fan Gao, Youzheng Wang, and Ning Ge. Domain-Generalized Adaptive Semantic Communication for Collaborative Perception. arXiv:2608.00056, 2026 (IEEE ICCC 2026). DOI: 10.48550/arxiv.2608.00056. https://arxiv.org/abs/2608.00056