/Inferring Maps and Behaviors from Natural Language Instructions

Inferring Maps and Behaviors from Natural Language Instructions

Felix Duvallet, Matthew Walter, Thomas Howard, Sachithra Hemachandra, Jean Hyaejin Oh, Seth Teller, Nicholas Roy and Anthony (Tony) Stentz
Conference Paper, International Symposium on Experimental Robotics, June, 2014

Download Publication (PDF)

Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author’s copyright. These works may not be reposted without the explicit permission of the copyright holder.


Natural language provides a exible, intuitive way for people to command robots, which is becoming increasingly important as robots transition to working alongside people in our homes and workplaces. To follow instructions in unknown environments, robots will be expected to reason about parts of the environments that were described in the instruction, but that the robot has no direct knowledge about. However, most existing approaches to natural language understanding require that the robot’s environment be known a priori. This paper proposes a probabilistic framework that enables robots to follow commands given in natural language, without any prior knowledge of the environment. The novelty lies in exploiting environment information implicit in the instruction, thereby treating language as a type of sensor that is used to formulate a prior distribution over the unknown parts of the environment. The algorithm then uses this learned distribution to infer a sequence of actions that are most consistent with the command, updating our belief as we gather more metric information. We evaluate our approach through simulation as well as experiments on two mobile robots; our results demonstrate the algorithm’s ability to follow navigation commands with performance comparable to that of a fully-known environment.

BibTeX Reference
author = {Felix Duvallet and Matthew Walter and Thomas Howard and Sachithra Hemachandra and Jean Hyaejin Oh and Seth Teller and Nicholas Roy and Anthony (Tony) Stentz},
title = {Inferring Maps and Behaviors from Natural Language Instructions},
booktitle = {International Symposium on Experimental Robotics},
year = {2014},
month = {June},