Discrete coding of reward probability and uncertainty by dopamine neurons

Science

Volumn 299, Issue 5614, 2003, Pages 1898-1902

  Fiorillo, Christopher D ^a   Tobler, Philippe N ^a   Schultz, Wolfram ^a  

^a NONE

Author keywords

[No Author keywords available]

Indexed keywords

INFORMATION DISSEMINATION; PROBABILITY; RISK MANAGEMENT; SIGNAL ENCODING;

DISCRETE CODINGS; DOPAMINE NEURONS;

BIOMECHANICS;

BIOCHEMISTRY; RISK;

ACCURACY; ANIMAL BEHAVIOR; ARTICLE; CELL ACTIVATION; DOPAMINERGIC NERVE CELL; LEARNING; MESENCEPHALON; NERVE CELL; NONHUMAN; PREDICTION; PRIORITY JOURNAL; REWARD; RISK; SIGNAL TRANSDUCTION; STATISTICAL ANALYSIS; VISUAL STIMULATION;

ANIMALIA;

EID: 0037459319     PISSN: 00368075     EISSN: None     Source Type: Journal    
DOI: 10.1126/science.1077349     Document Type: Article

References (32)

1. R. P. N. Rao, D. H. Ballard, Nature Neurosci. 2, 79 (1999).
2. D. M. Wolpert, Z. Ghahramani, Nature Neurosci. 3 (suppl.), 1212 (2000).
3. E. K. Engel, P. Fries, W. Singer, Nature Rev. Neurosci. 2, 704 (2001).
4. R. P. N. Rao, B. A. Olshausen, M. S. Lewicki, Eds., Probabilistic Models of the Brain (MIT Press, Cambridge, MA, 2002).
5. C. R. Gallistel, The Organization of Learning (MIT Press, Cambridge, MA, 1990).
6. A. Dickinson, Contemporary Animal Learning Theory (Cambridge Univ. Press, Cambridge, 1980).
7. J. M. Pearce, An Introduction to Animal Cognition (Lawrence Erlbaum, Hove, UK, 1987).
8. W. Schultz, P. Apicella, T. Ljungberg, J. Neurosci. 13, 900 (1993).
9. P. Waelti, A. Dickinson, W. Schultz, Nature 412, 43 (2001).
10. W. Schultz, J. Neurophysiol. 80, 1 (1998).
11. Materials and methods are available as supporting material on Science Online.
12. Simple linear regression coefficients for each type of phasic response were calculated for each set of data for which all probabilities were tested (P = 0.0, 0.25, 0.5, 0.75, and 1.0 in Fig. 2, C and E; P = 0.0, 0.25, 0.5, and 0.75 in Fig. 2D). This was done only as an approximation and does not imply linearity in the response functions. In addition to the nonlinear factors discussed in (13), there is imprecision in the subjective timing of the 2-s interval between stimulus onset and potential reward (15). This probably accounts for the small but significant activation to "fully" predicted reward in monkey A (Figs. 2C and 3B).
13. Unpublished data (30), as well as Figs. 3 and 4, suggest that the responses of dopamine neurons multiplicatively combine the probability and magnitude of reward. Thus, it is not necessarily the case that the maximal responses observed in this study for a given reward magnitude (those at P = 0.0, 0.5, or 1.0, depending on the type of response) are actually the maximal evoked responses of a given neuron. One would expect that, like other neurons coding the intensity of a signal, dopamine neurons have a stimulus-response function that is sigmoid, being insensitive to values above or below a particular range. The likelihood that individual neurons have distinct thresholds has critical implications for understanding the shape of the probability- response functions presented in Figs. 2 and 3 and could explain why many neurons shown in Fig. 3D appear to be unresponsive. The shape of the probability functions that we measured would depend on the range of values to which most of the neurons are sensitive. Because these ranges are unknown, the only interpretation that should be given to the data at this time is that dopamine neuronal responses follow probability or uncertainty in a monotonic fashion.
14. The present experiments were performed with a standard delay conditioning procedure, meaning that the conditioned stimulus remained on for the full 2-s delay until the potential time of reward. In a separate experiment, a smaller number of neurons (n = 22) were tested with trace conditioning in which the conditioned stimulus indicating the probability of reward was on for 1 s, and potential reward occurred following an additional 1-s interval after stimulus offset. Although there may have been some sustained activation in the trace condition at P = 0.5 (P < 0.1), the activity preceding potential reward (during either 250- or 500-ms periods) was significantly less than that in experiments with delay conditioning (P < 0.05, Mann-Whitney test). Furthermore, a distinct behavioral pattern emerged with trace conditioning; the likelihood of licking increased before stimulus offset, decreased subsequently, and then increased again before reward. The explanation for the apparent discrepancy between trace and delay conditioning is unclear, but it could be related to the presence of temporal information provided by the continued presence of the delay stimulus; that is, as long as the delay stimulus is present, the time of potential reward must not have passed, and this information could suppress incoming inhibitory signals that are (imprecisely) timed to coincide with potential reward (15).
15. Objectively, potential reward always occurred after a 2-s delay. However, it is known that subjective timing is imprecise. Thus, the time course of the slowly developing sustained activation could reflect the increasing likelihood that the interval is nearing completion. Unpublished data (30) on the phasic activation of dopamine neurons to the delivery of reward earlier or later than predicted suggest a similar degree of temporal imprecision in the prediction. It is therefore reasonable to hypothesize that dopamine neurons code the uncertainty in reward in the subsequent moment (the very near future).
16. A. L. Fairhall, G. D. Lewin, W. Bialek, R. R. de Ruyter van Steveninck, Nature 412, 787 (2001).
17. R. R. Montague, P. Dayan, T. J. Sejnowski, J. Neurosci. 16, 1936 (1996).
18. W. Schultz, P. Dayan, R. R. Montague, Science 275, 1593 (1997).
19. R. A. Rescorla, A. R. Wagner, in Classical Conditioning II: Current Research and Theory, A. H, Black, W. S. Prokasy, Eds. (Appleton-Century-Crofts, New York, 1972), pp. 64-69.
20. N. J. Mackintosh, Psychol. Rev. 82, 276 (1975).
21. J. M. Pearce, G. A. Hall, Psychol. Rev. 87, 532 (1980).
22. H. Kaye, J. M. Pearce, J. Exp. Psychol. Anim. Behav. Process. 10, 90 (1984).
23. The fact that there are two distinct dopamine signals, each with unique properties, suggests two distinct functions for dopamine. However, this does not necessarily imply that the two signals must be processed independently. Thus, each signal may contribute to the performance of two or more functions. Furthermore, questions concerning the functions of dopamine in target areas (such as reinforcement and attention) are distinct from questions about the qualitative nature of stimuli (rewarding versus attention-inducing) that contribute to the activation of dopamine neurons (31).
24. P. Dayan, S. Kakade, P. R. Montague, Nature Neurosci. 3 (suppl.), 1218 (2000).
25. In the artificial, impoverished conditions of a laboratory setting or a casino, the probabilities associated with particular stimuli or actions are fixed, and there is nothing else useful to be learned. However, the natural environment contains a high degree of correlation between a multitude of events; this is implicit in the adaptive utility of associative learning. Thus, an animal should not assume that uncertainty signals the objective absence of accurate predictors but rather that it is ignorant of those predictors, Although accurate predictors of reward may not always be present in the environment, one would not expect the learning machinery of the brain to assume their absence, In fact, there is a period of uncertainty about all rewards before accurate predictors are found. If subjective uncertainty is assumed to result from ignorance of predictors rather than absence of predictors, then it would be appropriate for subjective uncertainty to have attention-inducing and reinforcing properties that would ultimately enhance teaming and reduce uncertainty.
26. C. E. Shannon, Bell Syst. Tech. J. 27, 379 (1948).
27. C. R. Gallistel, J. Gibbon, Psychol. Rev. 107, 289
28. J. E. Mazur, Psychol. Rev. 108, 96 (2001).
29. Alternative attempts to explain gambling behavior focus on the fact that people (particularly those with prefrontal deficits) may misperceive reward probabilities or magnitudes or combine them in an inappropriate manner. This "cognitive" hypothesis fails to explain why gambling is appealing (and sometimes addictive) to a large number of otherwise healthy people, most of whom are aware that the odds are against them and that they have lost and will continue to lose money. In addition, any attempt to explain gambling behavior must address the fact that gambling is common at all probabilities (except P = 0 or 1, by definition) and all reward magnitudes. The present work suggests that activation of dopamine neurons may occur to a comparable extent during the expectation of a small reward at intermediate probabilities or a large reward at low probabilities. Thus, dopamine could contribute to the appeal of gambling in general. A behavior as prevalent as gambling must be explained in terms that are consistent with natural selection, The present hypothesis does so by pointing out that risk-taking promotes learning in natural environments.
30. C. D. Fiorillo, P. N. Tobler, W, Schultz, unpublished data.
31. P. Redgrave, T. J. Prescott, K. Gurney, Trends Neurosci. 22, 146 (1999).
32. We thank A. Dickinson, S. Baker, R. Moreno, K. Tsutsui, I. Hernadi, P. Dayan, and two anonymous reviewers for helpful comments on the manuscript. Funding was provided by the Human Frontiers Science Program (C.D.F.), Swiss National Science Funds (W.S. and P.N.T.), and Wellcome Trust (W.S.).