Abstract:
Real-world autonomy requires perception systems that deliver rich, accurate information given the task and environment. However, as robots scale to diverse and rapidly evolving settings, maintaining this level of performance becomes increasingly brittle and labor-intensive, requiring significant human engineering and retraining for even small changes in environment and problem definition. To overcome this bottleneck, this thesis advances flexible robot perception by improving its generalizability, adaptivity, and uncertainty-awareness, enabling robots to operate effectively across more environments with minimal additional human intervention.

First, to enable stronger zero-shot generalization, we introduce MapItAnywhere, a scalable ecosystem for generalizable bird’s-eye view (BEV) mapping.
However, even generalizable perception systems face inevitable performance drops when deployed in new environments. To bridge this gap automatically, we develop ALTER, a perception system that adapts on-the-fly to new environments while mitigating catastrophic forgetting and label noise. Lastly, while an adaptive perception system can improve over time, collecting data in low-information regions leads to inefficient learning. To this end, we present MapEx, an exploration algorithm that builds an uncertainty-aware representation with an ensemble of world model predictors, to guide data collection toward high-information regions.

This thesis advances the core capabilities of generalization, adaptation, and uncertainty awareness needed for flexible and automated robot perception. Together, these capabilities address the fundamental bottlenecks of current engineering-intensive workflows and bring us closer to scalable real-world autonomy.

Thesis Committee Members:
Sebastian Scherer​ (Chair)
Matthew Johnson-Roberson
Deva Ramanan
Ali-akbar Agha-mohammadi​ (Field AI​)

More Information