Learning Humanoid Control from Simulation to Real to Simulation

Abstract

Humanoid robots possess exceptional potential due to their capacity to perform versatile and human-like whole-body tasks. However, realizing agile and coordinated motions remains challenging, primarily due to the discrepancy in dynamics between simulations and real-world environments. Current methodologies, including system identification (SysID) and do- main randomization (DR), often demand extensive parameter tuning efforts or yield overly cautious policies, thus compromising agility. In this thesis, we introduce ASAP (Aligning Simulation and Real Physics), a two-stage framework aimed at effectively addressing dynamics mis- matches and enabling agile whole-body movements in humanoid robots. During the first stage, motion-tracking policies are pre-trained in simula- tion leveraging retargeted human motion data. In the second stage, policies are deployed on a real-world humanoid robot to gather data for training a delta (residual) action model, which compensates for simulation-to-reality dynamics differences. Subsequently, ASAP fine-tunes the pre-trained policies by incorporating the delta action model within the simulator, resulting in improved alignment with real-world dynamics. Our evaluation encompasses three distinct transfer scenarios: IsaacGym to IsaacSim, IsaacGym to Genesis, and IsaacGym to the physical Unitree G1 humanoid robot. Results demonstrate that ASAP significantly enhances agility and whole-body coordination compared to baseline approaches such as SysID, DR, and traditional delta dynamics learning techniques. By enabling highly agile and expressive motions previously challenging to achieve, our approach underscores the effectiveness of delta action learning in bridging the sim-to-real gap. This work highlights a promising direction for advancing agile humanoid robotics.