Abstract:

Object navigation (ObjectNav) asks a robot to find an instance of a target object category in an unknown environment, which demands perception, spatial reasoning, and long-horizon decision making at once. Today’s autonomous robots excel at mapping and moving through space yet lack high-level semantic intelligence, while vision-language models (VLMs) offer rich commonsense reasoning but limited 3D spatial grounding and long-term spatial consistency. Most existing VLM-based navigation methods treat the model as a black-box oracle, querying it at every step on unstructured local observations, which leads to redundant backtracking, inefficient exploration, and brittle behavior outside clean simulation. This thesis argues that ObjectNav is a system-level problem rather than a single-policy learning task: its sub-challenges of semantic understanding, complex spatial structure, and long-horizon planning should be explicitly decoupled and handled by cooperating modules, with the VLM asked to reason only at the level where it is reliable. We build such a system and carry it step by step from simulation to floor-scale, cross-embodiment deployment in the real world.

We first develop the core of this system in simulation, where the robot incrementally organizes what it has seen into a structured scene representation and the VLM reasons only at a high level over it, while efficient geometry-based exploration handles fine-grained navigation. This design achieves state-of-the-art success rate and navigation efficiency across four widely used benchmarks. We then bring the system into the real world and extend it into three cooperating levels that decouple semantic reasoning, navigation planning, and motion control. At the high level, the structured scene representation summarizes the environment and the VLM provides semantically grounded navigation guidance over it. At the mid level, a hierarchical room-based navigation strategy reserves VLM guidance for room-level decisions, which makes effective use of its reasoning while keeping the system efficient. At the low level, planned waypoints are executed by embodiment-specific motion control. Because only the lowest level depends on the robot, the same system runs on a custom-built wheeled robot, the Unitree Go2 quadruped, and the Unitree G1 humanoid. Across 190 real-world experiments, it substantially improves success rate and navigates 4-5x more efficiently than existing baselines. To our knowledge, it is the first system to reliably and efficiently complete floor-scale, long-range object navigation in complex real-world environments. Together, these results show that real-world ObjectNav is solved not by a larger model or a single end-to-end policy, but by a system that balances semantic intelligence with spatial reliability and isolates embodiment-specific control from embodiment-invariant reasoning.

Committee:
Jean Oh (advisor)
Ji Zhang
Zhixuan Liu