Graphics enhanced version of this site

A general convergence method for Reinforcement Learning in the continuous case
R. Munos
European Conference on Machine Learning, 1998.

Jump to: Download | Abstract | Notes | Text Reference | BibTeX Reference

Adobe portable document format (pdf) [579 KB]
Compressed postscript (ps.gz) [237 KB]

Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

In this paper, we propose a general method for designing convergent Reinforcement Learning algorithms in the case of continuous state space and time variables. The method is based on discretizations of the continuous process by convergent approximation schemes : the Hamilton-Jacobi-Bellman equation is replaced by a Dynamic Programming (DP) equation, for some Markovian Decision Process (MDP). If the data of the MDP were known, we could compute the value of the DP equation by using some DP updating rules. However, in the Reinforcement Learning (RL) approach, the state dynamics as well as the reinforcement functions are a priori unknown, leading impossible to use DP rules. Here we prove a general convergence theorem stating that if the values updated by some RL algorithm are close enough (in the sense that they satisfy a ''weak'' contraction property) to those of the DP, then they converge to the value function of the continuous process. The method is very general and is illustrated with a model-based algorithm build from a finite-difference approximation scheme.

Associated lab/group: Auton Lab
Associated project: Auton Project

Number of pages: 12

R. Munos, "A general convergence method for Reinforcement Learning in the continuous case," European Conference on Machine Learning, 1998.

@inproceedings{Munos_1998_2941,
   author = "Remi Munos",
   title = "A general convergence method for Reinforcement Learning in the continuous case",
   booktitle = "European Conference on Machine Learning",
   year = "1998"
}