Learning Manipulation Graphs from Demonstrations Using Multimodal Sensory Signals

Zhe Su, Oliver Kroemer, Gerald E. Loeb, Gaurav S. Sukhatme and Stefan Schaal
Conference Paper, International Conference on Robotics and Automation (ICRA), January, 2018

Download Publication

Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.


Complex contact manipulation tasks can be decomposed into sequences of motor primitives. Individual primitives often end with a distinct contact state, such as inserting a screwdriver tip into a screw head or loosening it through twisting. To achieve robust execution, the robot should be able to verify that the primitive’s goal has been reached as well as disambiguate it from erroneous contact states. In this paper, we introduce and evaluate a framework to autonomously construct manipulation graphs from manipulation demonstrations. Our manipulation graphs include sequences of motor primitives for performing a manipulation task as well as corresponding contact state information. The sensory models for the contact states allow the robot to verify the goal of each motor primitive as well as detect erroneous contact changes. The proposed framework was experimentally evaluated on grasping, unscrewing, and insertion tasks on a Barrett arm and hand equipped with two BioTacs. The results of our experiments indicate that the learned manipulation graphs achieve more robust manipulation executions by confirming sensory goals as well as discovering and detecting novel failure modes.

author = {Zhe Su and Oliver Kroemer and Gerald E. Loeb and Gaurav S. Sukhatme and Stefan Schaal},
title = {Learning Manipulation Graphs from Demonstrations Using Multimodal Sensory Signals},
booktitle = {International Conference on Robotics and Automation (ICRA)},
year = {2018},
month = {January},
} 2019-03-15T12:55:58-04:00