Long-term reward prediction in TD models of the dopamine system

Nathaniel Daw and David S. Touretzky
Neural Computation, Vol. 14, No. 11, November, 2002, pp. 2567 - 2583.

  • Adobe portable document format (pdf) (121KB)
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

This article addresses the relationship between long-term reward predictions and slow-timescale neural activity in temporal difference (TD) models of the dopamine system. Such models attempt to explain how the activity of dopamine (DA) neurons relates to errors in the prediction of future rewards. Previous models have been mostly restricted to short-term predictions of rewards expected during a single, somewhat artificially defined trial. Also, the models focused exclusively on the phasic pause-and-burst activity of primate DA neurons; the neurons' slower, tonic background activity was assumed to be constant. This has led to difficulty in explaining the results of neurochemical experiments that measure indications of DA release on a slow timescale, results that seem at first glance inconsistent with a reward prediction model. In this article, we investigate a TD model of DA activity modified so as to enable it to make longer-term predictions about rewards expected far in the future. We show that these predictions manifest themselves as slow changes in the baseline error signal, which we associate with tonic DA activity. Using this model, we make new predictions about the behavior of the DA system in a number of experimental situations. Some of these predictions suggest new computational explanations for previously puzzling data, such as indications from microdialysis studies of elevated DA activity triggered by aversive events.


Text Reference
Nathaniel Daw and David S. Touretzky, "Long-term reward prediction in TD models of the dopamine system," Neural Computation, Vol. 14, No. 11, November, 2002, pp. 2567 - 2583.

BibTeX Reference
   author = "Nathaniel Daw and David S Touretzky",
   title = "Long-term reward prediction in TD models of the dopamine system",
   journal = "Neural Computation",
   pages = "2567 - 2583",
   month = "November",
   year = "2002",
   volume = "14",
   number = "11",