Carnegie Mellon University
Imitation Learning for Natural Language Direction Following through Unknown Environments

Felix Duvallet, Thomas Kollar, and Anthony (Tony) Stentz
May, 2013.

  • Adobe portable document format (pdf) (1MB)
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

The use of spoken instructions in human-robot teams holds the promise of enabling untrained users to effectively control complex robotic systems in a natural and intuitive way. Providing robots with the capability to understand natural language directions would enable effortless coordination in human robot teams that operate in non-specialized unknown environments. However, natural language direction following through unknown environments requires understanding the meaning of language, using a partial semantic world model to generate actions in the world, and reasoning about the environment and landmarks that have not yet been detected. We address the problem of robots following natural language directions through complex unknown environments. By exploiting the structure of spatial language, we can frame direction following as a problem of sequential decision making under uncertainty. We learn a policy which predicts a sequence of actions that follow the directions by exploring the environment and discovering landmarks, backtracking when necessary, and explicitly declaring when it has reached the destination. We use imitation learning to train the policy, using demonstrations of people following directions. By training explicitly in unknown environments, we can generalize to situations that have not been encountered previously.

Natural language, imitation learning, decision making under uncertainty, human-robot interaction.

Associated Center(s) / Consortia: Field Robotics Center

Text Reference
Felix Duvallet, Thomas Kollar, and Anthony (Tony) Stentz , "Imitation Learning for Natural Language Direction Following through Unknown Environments," May, 2013.

BibTeX Reference
   author = "Felix Duvallet and Thomas Kollar and Anthony (Tony) {Stentz }",
   title = "Imitation Learning for Natural Language Direction Following through Unknown Environments",
   booktitle = "",
   month = "May",
   year = "2013",