Abstract

Autonomous robots should maintain resilient spatial perception despite sensor degradation. Humans preserve spatial awareness when moving between well-lit and dark spaces or when vision is partially occluded. Neuroscience studies suggest this robustness relies in part on a proprioception-first sensory hierarchy: the vestibular system provides continuous inertial reference signals, while vision supplies corrective updates that refine spatial estimates (Velez-Fort et al., Cell, 2025). In most robotic perception systems, this hierarchy is reversed. Exteroceptive sensors (cameras, LiDAR) are treated as primary, while the IMU plays a supporting role via preintegration and gap filling. As a result, reliability often depends on environmental conditions favorable to exteroceptive sensing. In dense smoke, darkness, or geometrically repetitive environments, multiple exteroceptive modalities can degrade simultaneously, and adding more exteroceptive sensors may not resolve these correlated failure modes.

This thesis asks whether we can build state estimation systems that (i) architecture design for resilient odometry (ii) estimate their own uncertainty for different modalities, (iii) adapt fusion strategies based on current sensor reliability.

We first address architectural structure for continuity under exteroceptive degradation. We propose an IMU-centric factor graph (Super Odometry, IROS 2021) in which inertial sensing provides a continuous backbone and exteroceptive sensors contribute constraints only when their measurements are sufficiently informative. This design was evaluated on aerial, wheeled, and legged platforms during the DARPA Subterranean Challenge. However, enabling or disabling an entire modality is a coarse response: in a long corridor, for example, LiDAR scan matching can be well constrained perpendicular to the walls but poorly constrained along the corridor axis. To address this, we develop per-axis degeneracy prediction (SuperLoc, ICRA 2025), which decomposes each scan’s information matrix into six directions corresponding to the pose degrees of freedom, enabling the optimizer to weight each measurement direction according to its estimated observability.

Next, we consider conditions where image measurements themselves become unreliable, such as dense smoke, low light, or overexposure. We estimate per-feature covariance from the photometric Hessian across RGB and thermal modalities (MSO, arXiv 2025), allowing the factor graph to weight individual features based on their estimated reliability rather than relying on a fixed noise model. When exteroceptive sensing is severely degraded or unavailable, the IMU may be the only remaining signal. Standard preintegration models, which assume slowly varying biases, can accumulate substantial drift. We address this with a learned inertial odometry model (TartanIMU, CVPR 2025) pretrained on data from diverse robot platforms, and we enable online adaptation to previously unseen platforms through self-supervised learning.

Finally, we integrate these components into a unified system and evaluate it at scale. We present a hierarchical adaptation framework (Super Odometry 2.0, Science Robotics) with four levels: per-feature uncertainty weighting, per-axis observability adjustment, sensor-level reconfiguration, and learned inertial odometry as a fallback. We evaluate the system over approximately 200 km and 800 hours of operation across aerial, wheeled, legged, and handheld platforms. To support standardized evaluation of SLAM robustness, we also introduce SubT-MRS (CVPR 2024), a multi-robot, multi-degradation benchmark with associated robustness metrics, which served as the evaluation framework for the ICCV 2023 SLAM Challenge. All these works are presented on superodometry.com

Thesis Committee

Sebastian Scherer (Chair)
Michael Kaess
Shubham Tulsiani
Jakob Engel (Meta)

Please find the PhD thesis defense here.