Active Exploration for Robot Parameter Selection in Episodic Reinforcement Learning

Oliver Kroemer and Jan Peters
Conference Paper, Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), January, 2011

Download Publication

Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.


As the complexity of robots and other autonomous systems increases, it becomes more important that these systems can adapt and optimize their settings actively. However, such optimization is rarely trivial. Sampling from the system is often expensive in terms of time and other costs, and excessive sampling should therefore be avoided. The parameter space is also usually continuous and multi-dimensional. Given the inherent exploration-exploitation dilemma of the problem, we propose treating it as an episodic reinforcment learning problem. In this reinforcement learning framework, the policy is defined by the system’s parameters and the rewards are given by the system’s performance. The rewards accumulate during each episode of a task. In this paper, we present a method for efficiently sampling and optimizing in continuous multidimensional spaces. The approach is based on Gaussian process regression, which can represent continuous non-linear mappings from parameters to system performance. We employ an upper confidence bound policy, which explicitly manages the trade-off between exploration and exploitation. Unlike many other policies for this kind of problem, we do not rely on a discretization of the action space. The presented method was evaluated on a real robot. The robot had to learn grasping parameters in order to adapt its grasping execution to different objects. The proposed method was also tested on a more general gain tuning problem. The results of the experiments show that the presented method can quickly determine suitable parameters and is applicable to real online learning applications.

author = {Oliver Kroemer and Jan Peters},
title = {Active Exploration for Robot Parameter Selection in Episodic Reinforcement Learning},
booktitle = {Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL)},
year = {2011},
month = {January},
} 2019-03-12T14:25:29-04:00