The 2019 work of Zhen, Hu, Liu and Scherer fused a rotating LiDAR with a stereo camera by jointly solving a bundle adjustment problem and a point-cloud registration problem, recovering camera poses and the LiDAR–camera extrinsic transform in one optimization and reaching 2.7 mm mean reconstruction accuracy at 70 points per square centimeter against survey-scanner ground truth — but only from data collected station by station at fixed viewpoints. The paper's stated future work is to extend this joint-optimization approach to sequential data acquired while the sensor platform moves continuously (for example on a micro aerial vehicle), and to improve stereo reconstruction quality through stronger LiDAR–stereo fusion. Since 2019, the literature has addressed these threads: probabilistic and learned LiDAR–stereo depth fusion has matured into an active subfield, principled uncertainty estimation is now standard, and tightly coupled LiDAR–inertial–visual odometry systems handle continuous motion with online extrinsic calibration — but at centimeter-class SLAM accuracy rather than the millimetre-class, ground-truth-verified regime that defines the source method. What remains open is a jointly optimized (or online-joint) LiDAR–stereo pipeline that produces survey-grade dense models from a continuously moving platform, with the extrinsic recovered online and the sensor-noise weights estimated from the data rather than hand-tuned.

Background

LiDAR (Light Detection and Ranging) is an active time-of-flight sensor that returns sparse but metrically accurate 3D points, while a stereo camera pair triangulates dense depth from pixel disparities between the two images, with depth error growing quadratically in distance and failing on textureless surfaces. The two sensors are complementary (dense color and texture versus sparse metric range), which motivates fusing them. Fusing them requires knowing the extrinsic calibration — the rigid-body transform between the camera frame and the LiDAR frame. Estimating that transform from sensor motion alone, without artificial targets, is mathematically the classical hand-eye calibration problem AX=XBAX = XB of Tsai and Lenz (1989), which is uniquely solvable only if at least two motion pairs are observed and the camera rotation axes of those motions are not collinear. Structure-from-Motion (SfM) recovers camera poses and sparse scene geometry from image correspondences, and its core estimator, bundle adjustment (BA), jointly refines poses and 3D points by minimizing reprojection error. Probabilistic fusion, in this setting, treats each sensor's depth measurement as a noisy estimate with an explicit uncertainty — for example Gaussian depth posteriors — and combines them so that the more certain source dominates; Maddern and Newman (IROS 2016) demonstrated real-time probabilistic fusion of sparse LiDAR and dense stereo depths in this spirit, before the source paper. The source paper (Zhen, Hu, Liu, Scherer, IEEE Robotics and Automation Letters 2019) proposed an offline method that, at each of a set of discrete measurement stations, acquires one stereo pair and one LiDAR scan, and then jointly solves the camera BA problem and the LiDAR point-cloud registration problem to estimate camera poses and the LiDAR–camera extrinsic transform simultaneously; a two-fold fusion step then uses the LiDAR returns to correct stereo depths on locally planar, well-observed patches. On a 29-station dataset collected 2–3 m from a squared concrete pillar, the method reached an averaged accuracy of 2.7 mm and a density of 70 points per square centimeter against a ground truth captured with a FARO terrestrial laser scanner (a tripod-mounted, survey-grade scanner), and its extrinsic estimate outperformed the state-of-the-art calibration method compared against. The method has three acknowledged limitations that motivate the open problem. First, station-wise acquisition: the sensor pod must be placed at discrete stations, which is slow and infeasible for inaccessible viewpoints (the authors cite bridge deck beams that are too high to reach), and the paper's conclusion explicitly defers handling sequential data with the pod moving through the environment, possibly carried by a micro aerial vehicle (MAV). Second, the stereo-depth correction is restricted to locally flat, well-observed regions; recovering a dense map everywhere would call for the stronger probabilistic LiDAR–stereo fusion the paper points to but does not implement. Third, two practical gaps inside the optimization itself: the extrinsic is only observable when the station-to-station camera rotations have non-collinear axes (the hand-eye degeneracy condition, which casual operators violate by rotating only about the vertical axis), and the cost weights that balance camera and LiDAR terms — in particular the LiDAR noise parameter σl\sigma_l, hand-tuned so that the two cost magnitudes roughly match (with σp=1\sigma_p = 1, σd=5×103\sigma_d = 5\times10^3, σl∈[0.02,0.1]\sigma_l \in [0.02, 0.1] in the experiments) — lack a principled estimator. Later literature has transformed the surrounding landscape. LiDAR–stereo depth fusion is now a mature subfield, from the first unsupervised noise-aware fusion network (LidarStereoNet, CVPR 2019) through cost-volume-level guidance (LGSM, ISPRS Journal 2022; LiDAR-SGM, ICRA 2023) to real-time non-learning fusion achieving a 2.79% error rate on the KITTI benchmark (a standard dataset of synchronized stereo images and LiDAR scans recorded from a car; Yao, Ishikawa, Oishi, RA-L 2025) and LiDAR-guided RAFT-Stereo (2025), in which sparse LiDAR depths pre-fill the cost volume of the iterative RAFT-Stereo stereo-matching network. Tightly coupled LiDAR–inertial–visual odometry (e.g. FAST-LIVO2, IEEE Transactions on Robotics 2025) fuses data from an inertial measurement unit (IMU, a chip sensor measuring rotational rates and accelerations), raw LiDAR points and photometric image errors in an error-state iterated Kalman filter (a standard recursive estimator that linearizes the measurement model around the current state estimate and iterates the update) with online extrinsic refinement, and LiDAR–visual bundle adjustment (LVBA, 2024) adds global consistency for colored mapping — but these real-time systems operate at centimeter-class SLAM accuracy, typically with a monocular camera, and are not evaluated against survey-grade metrology ground truth. In the construction domain, handheld camera-LiDAR rigs with learned fusion (Trzeciak and Brilakis, EC3 2022) improve density and noise of indoor reconstructions, again without survey-scanner-verified millimetre accuracy. The authors' own follow-ups moved along different axes: a LiDAR-enhanced SfM pipeline for motion robustness in large-scale environments (ICRA 2020), learned per-pixel disparity uncertainty for high-resolution stereo (ICRA 2020), and targetless multi-sensor calibration in the wild (2021) — none extends the joint optimization to continuous acquisition at its original accuracy regime. What therefore remains open is not whether LiDAR and stereo can be fused from a moving platform (they can, at SLAM grade), but whether the source method's defining property — millimetre-class, survey-scanner-verified dense reconstruction with self-calibration — survives the transition from discrete stations to a continuously moving platform, where per-sensor ego-motion must be estimated continuously, the extrinsic must be recovered online under possibly degenerate motion, and the sensor-noise weighting must be estimated from the data rather than hand-tuned.

Problem Statement

Extend the joint-optimization LiDAR–stereo fusion approach of Zhen, Hu, Liu and Scherer (RA-L 2019) — which currently requires station-wise acquisition and jointly solves camera bundle adjustment and LiDAR point-cloud registration to estimate camera poses and the LiDAR–camera extrinsic transform — to sequential data acquired while the sensor platform moves continuously through the scene, for example carried by a micro aerial vehicle. The extended method should: (1) estimate continuous sensor ego-motion and poses from the streaming stereo images and LiDAR sweeps, handling LiDAR sweep motion distortion; (2) recover the LiDAR–camera extrinsic calibration online or within the global optimization, remaining valid under the hand-eye observability conditions (non-collinear camera rotation axes across motions) and degrading gracefully or flagging degeneracy when they are violated; (3) fuse the sparse LiDAR depths with the dense stereo depths probabilistically, with the sensor-noise weights (in particular the LiDAR noise weight σl\sigma_l) estimated from the acquired data by a stated, reported procedure whose values depend on those data, rather than fixed by hand; and (4) produce dense 3D reconstructions whose accuracy and density, evaluated against independent survey-grade ground truth in the manner of the original paper (which achieved 2.7 mm mean error and 70 points per square centimeter against a FARO scanner), match the station-wise pipeline's survey-grade regime — millimetre-class mean error within the reference scanner's precision, with a density of the same order as the original's — rather than reproducing the original figures exactly. These four components jointly define the objective — dense, self-calibrated, survey-grade 3D reconstruction from continuously moving LiDAR–stereo acquisition. The question likewise asks whether the station-wise regime can be carried over to continuous acquisition generally — not only for a specific degenerate motion pattern such as rotation about a single axis — or whether a fundamental obstacle prevents that carry-over.

References

  1. Weikun Zhen, Yaoyu Hu, Jingfeng Liu, Sebastian Scherer. A Joint Optimization Approach of LiDAR-Camera Fusion for Accurate Dense 3-D Reconstructions. IEEE Robotics and Automation Letters, 2019. DOI: 10.1109/LRA.2019.2928261. https://arxiv.org/abs/1907.00930
  2. W. Maddern, P. Newman. Real-time probabilistic fusion of sparse 3D LIDAR and dense stereo. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2016. DOI: 10.1109/IROS.2016.7759342. https://doi.org/10.1109/IROS.2016.7759342
  3. R. Y. Tsai, R. K. Lenz. A new technique for fully autonomous and efficient 3D robotics hand/eye calibration. IEEE Transactions on Robotics and Automation, 1989. DOI: 10.1109/70.34770. https://doi.org/10.1109/70.34770
  4. Weikun Zhen, Yaoyu Hu, Huai Yu, Sebastian Scherer. LiDAR-enhanced Structure-from-Motion. IEEE International Conference on Robotics and Automation (ICRA), 2020. DOI: 10.1109/ICRA40945.2020.9197030. https://arxiv.org/abs/1911.03369
  5. Yaoyu Hu, Weikun Zhen, Sebastian Scherer. Deep-Learning Assisted High-Resolution Binocular Stereo Depth Reconstruction. IEEE International Conference on Robotics and Automation (ICRA), 2020. DOI: 10.1109/ICRA40945.2020.9196655. https://arxiv.org/abs/1912.05012
  6. Taimeng Fu, Huai Yu, Wen Yang, Yaoyu Hu, Sebastian Scherer. Targetless Extrinsic Calibration of Stereo Cameras, Thermal Cameras, and Laser Sensors in the Wild. arXiv, 2021. arXiv:2109.13414. https://arxiv.org/abs/2109.13414
  7. Xuelian Cheng, Yiran Zhong, Yuchao Dai, Pan Ji, Hongdong Li. Noise-Aware Unsupervised Deep Lidar-Stereo Fusion. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. DOI: 10.1109/CVPR.2019.00650. https://arxiv.org/abs/1904.03868
  8. Yongjun Zhang, Siyuan Zou, Xinyi Liu, Xu Huang, Yi Wan, Yongxiang Yao. LiDAR-guided stereo matching with a spatial consistency constraint. ISPRS Journal of Photogrammetry and Remote Sensing, 2022. DOI: 10.1016/j.isprsjprs.2021.11.003. https://arxiv.org/abs/2202.09953
  9. Bianca Forkel, Hans-Joachim Wuensche. LiDAR-SGM: Semi-Global Matching on LiDAR Point Clouds and Their Cost-Based Fusion into Stereo Matching. IEEE International Conference on Robotics and Automation (ICRA), 2023. DOI: 10.1109/ICRA48891.2023.10160775. https://doi.org/10.1109/ICRA48891.2023.10160775
  10. Yasuhiro Yao, Ryoichi Ishikawa, Takeshi Oishi. Stereo-LiDAR Fusion by Semi-Global Matching With Discrete Disparity-Matching Cost and Semidensification. IEEE Robotics and Automation Letters, 2025. DOI: 10.1109/LRA.2025.3552236. https://arxiv.org/abs/2504.05148
  11. Jinsu Yoo, Sooyoung Jeon, Zanming Huang, Tai-Yu Pan, Wei-Lun Chao. Leveraging Sparse LiDAR for RAFT-Stereo: A Depth Pre-Fill Perspective. arXiv, 2025. arXiv:2507.19738. https://arxiv.org/abs/2507.19738
  12. Chunran Zheng, Wei Xu, Zuhao Zou, Tong Hua, Chongjian Yuan, Dongjiao He, Bingyang Zhou, Zheng Liu, Jiarong Lin, Fangcheng Zhu, Yunfan Ren, Rong Wang, Fanle Meng, Fu Zhang. FAST-LIVO2: Fast, Direct LiDAR–Inertial–Visual Odometry. IEEE Transactions on Robotics, 2025. DOI: 10.1109/TRO.2024.3502198. https://arxiv.org/abs/2408.14035
  13. Rundong Li, Xiyuan Liu, Haotian Li, Zheng Liu, Jiarong Lin, Yixi Cai, Fu Zhang. LVBA: LiDAR-Visual Bundle Adjustment for RGB Point Cloud Mapping. arXiv, 2024. arXiv:2409.10868. https://arxiv.org/abs/2409.10868
  14. Mathieu Cocheteux, Julien Moreau, Franck Davoine. Uncertainty-Aware Online Extrinsic Calibration: A Conformal Prediction Approach. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2025. arXiv:2501.06878. https://arxiv.org/abs/2501.06878
  15. Maciej Trzeciak, Ioannis Brilakis. Precise and dense AI-based mobile 3D reconstruction of indoor scenes by camera-lidar fusion and odometry. Proceedings of the European Conference on Computing in Construction, 2022. DOI: 10.35490/ec3.2022.150. https://doi.org/10.35490/ec3.2022.150