Interactive Multi-Modal Robot Programming

Abstract

As robots enter the human environment and come in contact with inexperienced users, they need to be able to interact with users in a multi-modal fashion-keyboard and mouse are no longer acceptable as the only input modalities. Humans should be able to communicate with robots using methods as similar as possible to the concise, rich, and diverse means they use to communicate with one another. This thesis is an investigation of how one can improve user's ability to control and program a mobile robot. The goal is a comprehensive multi-modal human-machine interface that allows non-experts to compose robot programs conveniently. Two key characteristics of this novel programming approach are that the system infer the user's intent to support interaction, and that the user can provide feedback interactively through intuitive interface, at any time. The framework takes a three-step approach to the problem: multi-modal recognition, intention interpretation, and prioritized task execution. The multi- modal recognition module translates hand gestures and spontaneous speech into a structured symbolic data stream without abstracting away the user's intent. The intention interpretation module selects the appropriate primitives based on the user input, current state, and robot sensor data. Finally, the prioritized task execution module selects and executes primitives based on current state, sensor input, and the task given by the previous step. Depending on the mode of operation, the system can provide interactive robot control and composition of robot programs. The framework is demonstrated by interactively controlling and programming a vacuum-cleaning robot. The demonstrations are used to exemplify the interactive programming and the plan recognition aspect of the research. The key contributions of this thesis are the introduction and implementation of the novel programming approach. It is expected to improve significantly the state-of-the-art in robot programming and interactive personal robotics.