A general convergence method for Reinforcement Learning in the continuous case

Remi Munos
European Conference on Machine Learning, 1998.


Download
  • Adobe portable document format (pdf) (579KB)
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

Abstract
In this paper, we propose a general method for designing convergent Reinforcement Learning algorithms in the case of continuous state space and time variables. The method is based on discretizations of the continuous process by convergent approximation schemes : the Hamilton-Jacobi-Bellman equation is replaced by a Dynamic Programming (DP) equation, for some Markovian Decision Process (MDP). If the data of the MDP were known, we could compute the value of the DP equation by using some DP updating rules. However, in the Reinforcement Learning (RL) approach, the state dynamics as well as the reinforcement functions are a priori unknown, leading impossible to use DP rules. Here we prove a general convergence theorem stating that if the values updated by some RL algorithm are close enough (in the sense that they satisfy a ''weak'' contraction property) to those of the DP, then they converge to the value function of the continuous process. The method is very general and is illustrated with a model-based algorithm build from a finite-difference approximation scheme.

Notes
Associated Lab(s) / Group(s): Auton Lab
Associated Project(s): Auton Project
Number of pages: 12

Text Reference
Remi Munos, "A general convergence method for Reinforcement Learning in the continuous case," European Conference on Machine Learning, 1998.

BibTeX Reference
@inproceedings{Munos_1998_2941,
   author = "Remi Munos",
   title = "A general convergence method for Reinforcement Learning in the continuous case",
   booktitle = "European Conference on Machine Learning",
   year = "1998",
}