6:00 pm to 7:00 pm
Newell-Simon Hall 4305
Abstract:
Generalization is a core challenge in robotics, where the goal is to enable robots to handle novel objects, environments, and embodiments with minimal additional data. This thesis explores how human prior knowledge, captured through both passive observation and active demonstration, can be leveraged to improve generalization in manipulation tasks. We propose two complementary approaches that scale robot learning leveraging large-scale human-derived data.
First, we introduce HRP (Human Affordances for Robotic Pre-Training), where we learn actionable visual representations by extracting hand trajectories, contact points, and object labels from internet-scale human videos. These representations, when used to initialize control policies, lead to significant performance gains in downstream robot manipulation tasks and transfer effectively across viewpoints and robot morphologies.
Second, we present DexWild (Dexterous Human Interactions for In-the-Wild Robot Policies), a system that collects high-fidelity in-the-wild demonstrations using a human motion-capture device. A human-robot co-training algorithm combines this diverse human data with limited robot data, enabling robust policy transfer to unseen scenes, robot arms, and hands.
Together, these results show that human priors, whether learned through passive observation or active demonstration, can significantly enhance a robot’s ability to generalize.
Prof. Deepak Pathak (advisor)
Prof. Abhinav Gupta
Mihir Prabhudesai