Friday, August 29
11:00 am to 1:00 pm
GHC 4405
11:00 am to 1:00 pm
GHC 4405
Scalable Sim-to-Real Learning for General-Purpose Humanoid Skills
Abstract:
Thesis Committee Members:
Guanya Shi (co-chair)
Changliu Liu (co-chair)
Kris Kitani
Humanoids represent the most versatile robotic platform, capable of walking, manipulating, and collaborating with people in human-centered environments. Yet, despite recent advances, building humanoids that can operate reliably in the real world remains a fundamental challenge. Progress has been hindered by difficulties in whole-body control, robust perceptive reasoning, and bridging the sim-to-real gap.
This thesis proposal explores how scalable simulation and learning can systematically overcome these barriers. I will present a research trajectory that advances humanoid capabilities along three dimensions:
1. Sim-to-Real Motor Control (Chapter 2–5): From real-time teleoperation (H2O) to dexterous loco-manipulation (OmniH2O), to a versatile generalist controller (HOVER), and agile transfer via dynamics alignment (ASAP), these works demonstrate increasingly dexterous and adaptable control.
2. Sim-to-Real Perceptive Control (Chapter 6): With ABS, we show that robust real-world locomotion requires tightly coupling exteroceptive and proprioceptive sensing—shifting sim-to-real learning from blind skill execution to perception-driven control.
3. Future Directions (Chapter 7): I will outline next steps in (1) Perceptive Loco-Manipulation—end-to-end visuomotor policies unifying perception, locomotion, and manipulation—and (2) Real-to-Sim Evaluation—using high-fidelity simulators environments to provide consistent evaluation protocols and benchmarks for real-world policy evaluation.
Taken together, these works explore how far scalable sim-to-real learning can advance humanoid capabilities across control and perception. While sim-to-real is not the only path toward reliable humanoids, this thesis aims to rigorously test its limits—probing how scaling simulation and learning can push humanoids closer to functioning as capable partners in real-world environments.
Thesis Committee Members:
Guanya Shi (co-chair)
Changliu Liu (co-chair)
Kris Kitani
Marco Hutter (ETH Zurich)
Pieter Abbeel (UC Berkeley)