Abstract:
Hierarchical imitation learning has become an effective approach for robotic manipulation: a high-level policy predicts a sub-goal end-effector pose, while a low-level policy executes the actions needed to reach it. This decomposition improves generalization and provides an interpretable interface, but the design of the high-level goal predictor remains an open question.

This thesis studies high-level goal prediction through three investigations. First, on the MimicGen benchmark, we compare dense predicted goal point clouds with a sparse four-point end-effector representation, finding that the sparse representation provides a more reliable conditioning signal across tasks. Second, on RLBench, we extend the four-point predictor with language, RGB features, gripper actions, and collision-ignore decisions, pairing it with motion planning to achieve competitive performance with state-of-the-art keyframe-action prediction methods. Third, in a sim-to-real setting, we make the high-level predictor steerable through prompting, allowing users to select among multiple valid targets, such as which drawer to open.

Together, these studies clarify how representation, capability, and controllability shape high-level goal prediction for hierarchical imitation learning in robotic manipulation.

Thesis Committee:
Prof. David Held (co-advisor)
Prof. Zackory Erickson (co-advisor)
Prof. Shubham Tulsiani
Yilin Wu