Title: Visual Imitation Learning for Robot Manipulation

Abstract:

Imitation learning has been successfully applied to solve a variety of tasks in complex domains where an explicit reward function is not available. However, most imitation learning methods require access to the robot’s actions during demonstration. This stands in a stark contrast to how we humans imitate: we acquire new skills by simply observing other humans perform a task, mostly relying on the visual information of the scene.

In this thesis, we examine how we can endow a robotic agent with this capability, i.e., how to acquire a new skill via visual imitation learning. A key challenge in learning from raw visual inputs is to extract meaningful information from the input scene, and enabling the robotic agent to learn the demonstrated skill based on the accessible input data.  We present a framework that encodes the visual input of a scene into a factorized graph representation, casting one-shot visual imitation of manipulation skills as a visual correspondence learning problem. Using different visual entities such as human keypoints and object-centric pixel features, we verify our approach through real robotic experiments, and we show how the proposed image graph encoding drives successful imitation of a variety of manipulation skills within minutes, using a single demonstration and without any environment instrumentation.

Committee:

Katerina Fragkiadaki (Advisor)

Oliver Kroemer (Advisor)

Ruslan Salakhutdinov

Lerrel Pinto