Date: Tuesday May 19, 2026
Time:  1:00 – 3:00PM (EST)
Location: NSH (Newell Simon Hall) 3305

Zoom Link

Type: PhD Thesis Defense

Title: Learning Geometric, Physical, and Semantic Priors for Embodied Planning and Control

Abstract: Embodied intelligence requires perceiving, predicting, and acting in environments with an understanding of the geometric, semantic, and physical structure of the world. Recently, the dominant trend in robotics has been to acquire such world understanding implicitly in a data-driven manner using end-to-end models. While these approaches have achieved impressive milestones, they often rely on substantial amounts of in-domain data and may remain brittle when success depends on long-horizon reasoning, precise physical interaction, or generalization from limited task-specific data. This thesis studies a perspective complementary to end-to-end approaches: when aspects of geometric, physical, and semantic structure are known and reusable, explicitly learning priors over them can improve the data efficiency, fidelity, and robustness of embodied learning systems.

This thesis instantiates this perspective across four embodied settings: scene-level forecasting in autonomous driving, learning deformable object dynamics from robot interaction videos, relational reasoning and cross-instance manipulation transfer, and zero-shot long-horizon manipulation. For scene-level prediction, we learn a predictive geometric prior over the future evolution of the full 3D scene representation. By modeling future motion at the scene level, the predictive geometric representation preserves coherence across agents and the environment, improving downstream prediction and planning under multi-agent uncertainty. Moving from passive scene forecasting to manipulation, physical priors become important to predict how state evolves in response to robot actions. To learn contact-rich dynamics and topological change directly from RGB-D robot interaction videos, we represent deformable objects as adaptive sets of 3D Gaussians inside a particle-filtering framework with physics-inspired interaction modeling and resampling mechanisms. Extending to multi-object manipulation problems, we develop semantic priors based on correspondence as a reusable representation for relational reasoning and cross-instance manipulation. These correspondence-based priors allow a robot to identify functionally meaningful object structure, reason about object alignment under geometric ambiguity, and transfer manipulation knowledge from demonstrated objects to novel instances. Finally, we integrate learned geometric, semantic, and physical priors into a zero-shot long-horizon manipulation system that connects high-level video and language planning with executable robot motion through geometric grounding of generated videos. Taken together, these works show that explicitly learned geometric, physical, and semantic priors can improve the data efficiency, fidelity, and robustness of embodied prediction, planning, and manipulation systems.

Committee:

Kris Kitani (Chair), Carnegie Mellon University

David Held, Carnegie Mellon University

Shubham Tulsiani, Carnegie Mellon University

Brian Okorn, Robotics and AI Institute

Thesis Draft