Imitation learning has emerged as a leading paradigm for teaching manipulation skills to robots, but its success depends on the costly endeavour of collecting robot demonstrations through teleoperation. Generalizing to novel objects, environments, and task variations typically requires massive datasets that are expensive to scale. This thesis investigates an alternative lever: hierarchy—explicitly factorizing manipulation policies into high-level reasoning about where to go and low-level control about how to get there—and asks whether such a factorization can both improve data efficiency from robot demonstrations alone and enable the use of
cheaper sources of data such as human video.

We present GHOST, a framework for learning visuomotor manipulation policies that generalize beyond the training distribution. GHOST factorizes control into (i) a high-level policy that predicts the next sub-goal as a distribution over 3D end-effector poses from multi-view RGB-D observations, and (ii) a low-level goal-conditioned controller that executes embodiment-specific actions. To condition image-based policies on 3D goals, we introduce a simple spatial interface that projects predicted goals into the image plane and represents them as end-effector heatmaps. Across a suite of manipulation tasks, this hierarchical factorization consistently improves performance and robustness compared to a flat Diffusion Policy. We further show that this hierarchical interface makes it easy to incorporate human demonstrations without relying on (noisy) action retargeting. Because sub-goals are largely embodiment-agnostic, we train
the high-level policy on human video to specify how learned skills should be applied and composed, while keeping the low-level policy trained purely on robot data. This hierarchy enables adaptation to novel objects and task variations using only a small number of human demonstrations.

We evaluate GHOST on a suite of manipulation tasks spanning pick-and-place, cloth folding, and tool-use, demonstrating both improved in-distribution performance and meaningful cross-embodiment generalization from human video. We conduct ablations to further isolate the contributions of the hierarchical factorization, and identify the visual domain gap between human and robot observations as a significant bottleneck for long-horizon generalization.

Committee:
David Held (advisor)
Shubham Tulsiani
Aviral Kumar

Yilin Wu