/Autonomy Infused Teleoperation with Application to BCI Manipulation

Autonomy Infused Teleoperation with Application to BCI Manipulation

Katharina Muelling, Arun Venkatraman, Jean-Sebastien Valois, John Downey, Jeffrey Weiss, Shervin Javdani, Martial Hebert, Andrew B. Schwartz, Jennifer L. Collinger and J. Andrew (Drew) Bagnell
Conference Paper, Proceedings of Robotics: Science and Systems, July, 2015

Download Publication (PDF)

Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author’s copyright. These works may not be reposted without the explicit permission of the copyright holder.


Robot teleoperation systems face a common set of challenges including latency, low-dimensional user commands, and asymmetric control inputs. User control with Brain- Computer Interfaces (BCIs) exacerbates these problems through especially noisy and erratic low-dimensional motion commands due to the difficulty in decoding neural activity. We introduce a general framework to address these challenges through a combination of computer vision, user intent inference, and arbitration between the human input and autonomous control schemes. Adjustable levels of assistance allow the system to balance the operator’s capabilities and feelings of comfort and control while compensating for a task’s difficulty. We present experimental results demonstrating significant performance improvement using the shared-control assistance framework on adapted rehabilitation benchmarks with two subjects implanted with intracortical brain-computer interfaces controlling a seven degree-of-freedom robotic manipulator as a prosthetic. Our results further indicate that shared assistance mitigates perceived user difficulty and even enables successful performance on previously infeasible tasks. We showcase the extensibility of our architecture with applications to quality-of-life tasks such as opening a door, pouring liquids from containers, and manipulation with novel objects in densely cluttered environments.

BibTeX Reference
author = {Katharina Muelling and Arun Venkatraman and Jean-Sebastien Valois and John Downey and Jeffrey Weiss and Shervin Javdani and Martial Hebert and Andrew B. Schwartz and Jennifer L. Collinger and J. Andrew (Drew) Bagnell},
title = {Autonomy Infused Teleoperation with Application to BCI Manipulation},
booktitle = {Proceedings of Robotics: Science and Systems},
year = {2015},
month = {July},