Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration

Philosophical Transactions of the Royal Society B: Biological Sciences

Volumn 362, Issue 1481, 2007, Pages 933-942

  Cohen, Jonathan D ^a,b   McClure, Samuel M ^a   Yu, Angela J ^a  

^a PRINCETON UNIVERSITY   (United States)

^b UNIVERSITY OF PITTSBURGH SCHOOL OF MEDICINE   (United States)

Author keywords

Decision making; Exploration; Learning; Neurotransmitters; Prefrontal cortex; Uncertainty

Indexed keywords

BRAIN; EXPERIMENTAL STUDY; EXPLORATION; LEARNING; TRADE-OFF;

ANIMALIA;

BRAIN; DECISION MAKING; EXPLORATORY BEHAVIOR; HUMAN; PHYSIOLOGY; REVIEW;

BRAIN; DECISION MAKING; EXPLORATORY BEHAVIOR; HUMANS;

EID: 34250348767     PISSN: 09628436     EISSN: None     Source Type: Journal    
DOI: 10.1098/rstb.2007.2098     Document Type: Conference Paper

References (52)

1. Ainslie, G. 1975 Specious reward: a behavioral theory of impulsiveness and impulse control. Psychol. Bull. 82, 463-496. (doi:10.1037/h0076860)
2. Allport, A., Styles, E. & Hsieh, S. 1994 Shifting intentional set: exploring the dynamic control of task. In Attention and performance XV (eds C. Umilta & M. Moscovitch), pp. 421-452. Cambridge, MA: MIT Press.
3. Aston-Jones, G. & Cohen, J. D. 2005 An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. Annu. Rev. Neurosci. 28, 403-450. (doi:10.1146/annurev.neuro.28. 061604.135709)
4. Aston-Jones, G., Rajkowski, J., Kubiak, P. & Alexinsky, T. 1994 Locus coeruleus neurons in monkey are selectively activated by attended cues in a vigilance task. J. Neurosci. 14, 4467-4480.
5. Aston-Jones, G., Rajkowski, J. & Kubiak, P. 1997 Conditioned responses in monkey locus coeruleus neurons anticipate acquisition of discriminative behavior in a vigilance task. Neuroscience 80, 697-715. (doi:10.1016/S0306-4522(97)00060-2)
6. Aston-Jones, G., Rajkowski, J., Lu, W., Zhu, Y., Cohen, J. D. & Morecraft, R. J. 2002 Prominent projections from the orbital prefrontal cortex to the locus coeruleus in monkeys. Soc. Neurosci. Abstr. 28, 86-89.
7. Banks, J. S. & Sundaram, R. K. 1994 Switching costs and the Gittins index. Econometrica: J. Econ. Soc. 62, 687-694.
8. Berry, D. A. & Fristedt, B. 1985 Bandit problems: sequential allocation of experiments. London, UK: Chapman and Hall.
9. Botvinick, M. M., Braver, T. S., Barch, D. M., Carter, C. S. & Cohen, J. D. 2001 Conflict monitoring and cognitive control. Psychol. Rev. 108, 624-652. (doi:10.1037/0033-295X.108.3.624)
10. Brown, E., Gao, J., Bogacz, R., Gilzenrat, M. & Cohen, J. D. 2005 Simple neural networks that optimize decisions. Int. J. Bifurc. Chaos 15, 803-826. (doi:10.1142/S021812 7405012478)
11. Carstensen, L. L., Isaacowitz, D. & Charles, S. T. 1999 Taking time seriously: a theory of socioemotional selectivity. Am. Psychol. 54, 165-181. (doi:10.1037/0003-066X.54.3.165)
12. Carter, C. S., Braver, T. S., Barch, D. M., Botvinick, M. M., Noll, D. C. & Cohen, J. D. 1998 Anterior cingulate cortex, error detection and the on-line monitoring of performance. Science 280, 747-749. (doi:10.1126/science. 280.5364.747)
13. Carter, C. S., Macdonald, A. M., Botvinick, M., Ross, L. L., Stenger, V. A., Noll, D. & Cohen, J. D. 2000 Parsing executive processes: strategic vs. evaluative functions of the anterior cingulate cortex. Proc. Natl Acad. Sci. USA 97, 1944-1948. (doi:10.1073/pnas.97.4.1944)
14. Daw, N. D., O'Doherty, J. P., Seymour, B., Dayan, P. & Dolan, R. J. 2006 Cortical substrates for exploratory decisions in humans. Nature 441, 876-879. (doi:10.1038/nature04766)
15. Gilzenrat M. S. & Cohen J. D. In preparation. The role of locus coeruleus in mediating between exploration and exploitation in nonstationary environments: an empirical test in a changing utility task.
16. Gittins, J. C. 1979 Bandit processes and dynamic allocation indices. J. R. Stat. Soc. B 41, 148-177.
17. Gittins, J. C. & Jones, D. M. 1974 A dynamic allocation index for the sequential design of experiments. In Progress in statisics (ed. J. Gans), pp. 241-266. Amsterdam, The Netherlands: North-Holland.
18. Gratton, G., Coles, M. G. H. & Donchin, E. 1992 Optimization in the use of information: strategic control of activation and responses. J. Exp. Psychol. Gen. 4, 480-506. (doi:10.1037/0096-3445.121.4.480)
19. Holroyd, C. B. & Coles, M. G. H. 2002 The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychol. Rev. 109, 679-709. (doi:10.1037/0033-295X.109.4.679)
20. Kaelbling, L. P. 1996 Gittins Allocation Indices. See http://www.cs.cmu.edu/afs/cs/project/jair/pub/volume4/kaelbling96a-html/node9. html.
21. Kaelbling, L. P. 2003 Learning in embedded systems. Cambridge, MA: MIT Press.
22. Kaelbling, L. P., Littman, M. L. & Moore, A. W. 1996 Reinforcement learning: a survey. J. Artif. Intell. Res. 4, 237-285.
23. Kerns, J. G., Cohen, J. D., MacDonald, A. W., Cho, R. Y., Stenger, V. A. & Carter, C. S. 2004 Anterior cingulated conflict monitoring and adjustments in control. Science 303, 1023-1026. (doi:10.1126/science.1089910)
24. Knutson, B., Fong, G. W., Bennett, S. M., Adams, C. M. & Hommer, D. 2003 A region of the mesial prefrontal cortex tracks monetary rewarding outcomes: characterization with rapid event-related fMRI. Neuroimage 18, 263-272. (doi:10.1016/S1053-8119(02)00057-5)
25. Krebs, J. R., Kacelnik, A. & Taylor, P. 1978 Tests of optimal sampling by foraging great tits. Nature 275, 27-31. (doi:10.1038/275027a0)
26. Laming, D. R. J. 1979 Choice reaction performance following an error. Acta Psychologica 43, 199-224. (doi:10.1016/0001-6918(79)90026-X)
27. Leonard, N. E., Paley, D., Lekien, F., Sepulchre, R., Fratantoni, D. M. & Davis, R. E. In press. Collective motion, sensor networks and ocean sampling. Proc. IEEE, 95.
28. McClure, S. M., Daw, N. D. & Montague, P. R. 2003 A computational substrate for incentive salience. Trends Neurosci. 26, 423-428. (doi:10.1016/S0166-2236(03) 00177-2)
29. McClure, S. M., Laibson, D. I., Loewenstein, G. & Cohen, J. D. 2004 Separate neural systems value immediate and delayed monetary reward. Science 306, 503-507. (doi:10. 1126/science.1100907)
30. McClure, S. M., Gilzenrat, M. S. & Cohen, J. D. 2006 An exploration-exploitation model based on norepinephrine and dopamine activity. In Advances in neural information processing systems, vol. 18 (eds Y. Weiss, B. Sholkopf & J. Piatt), pp. 867-874. Cambridge, MA: MIT Press.
31. Miller, E. K. & Cohen, J. D. 2001 An integrative theory of prefrontal cortex function. Annu. Rev. Neurosci. 24, 167-202. (doi:10.1146/annurev.neuro.24.1.167)
32. Miltner, W. H. R., Braun, C. H. & Coles, M. G. H. 1997 Event-related potentials following incorrect feedback in a time-estimation task: evidence for a 'generic' neural system for error detection. J. Cogn. Neurosci. 9, 788-798.
33. Montague, P. R., Dayan, P. & Sejnowski, T. J. 1996 A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936-1947.
34. Montague, P. R., Hyman, S. E. & Cohen, J. D. 2004 Computational roles for dopamine in behavioral control. Nature 431, 760-767. (doi:10.1038/nature03015)
35. Montague, P. R., King-Casas, B. & Cohen, J. D. In press. Imaging valuation models in human choice. Annu. Rev. Neurosci. 29, 417-448. (doi:10.1146/annurev.neuro.29. 051605.112903)
36. Newell, A. & Simon, H. A. 1972 Human problem solving. Englewood Cliffs, NJ: Prentice-Hall.
37. O'Doherty, J., Kringelback, M. L., Rolls, E. T., Hornak, J. & Andrews, C. 2001 Abstract reward and punishment representation in the human orbitofrontal cortex. Nat. Neurosci. 4, 95-102. (doi:10.1038/82959)
38. Padoa-Schioppa, C. & Assad, J. A. 2006 Neurons in the orbitofrontal cortex encode economic value. Nature 441, 223-226. (doi:10.1038/nature04676)
39. Peyron, R., Laurent, B. & Garcia-Larrea, L. 2000 Functional imaging of brain responses to pain: a review and meta-analysis. Neurophysiol. Clin. 30, 263-288. (doi:10.1016/S0987-7053(00)00227-6)
40. Posner, M. I., Snyder, C. R. R. & Davidson, B. J. 1980 Attention and the detection of signals. J. Exp. Psychol. Gen. 109, 160-174. (doi:10.1037/0096-3445.109.2.160)
41. Pratt, S. C. & Sumpter, D. J. T. 2006 A tunable algorithm for collective decision-making. Proc. Natl Acad. Sci. USA 103, 15 906-15 910. (doi:10.1073/pnas.0604801103)
42. Rabbitt, P. M. A. 1966 Errors and error-correction in choice-response tasks. J. Exp. Psychol. 71, 264-272. (doi:10.1037/h0022853)
43. Rajkowski, J., Lu, W., Zhu, Y., Cohen, J. D. & Aston-Jones, G. 2000 Prominent projections from the anterior cingulate cortex to the locus coeruleus in Rhesus monkey. Soc. Neurosci. Abstr. 26, 838.15.
44. Rogers, R. & Monsell, S. 1995 The costs of a predictable switch between simple cognitive tasks. J. Exp. Psychol. Gen. 124, 207-231. (doi:10.1037/0096-3445.124.2. 207)
45. Schultz, W., Dayan, P. & Montague, P. R. 1997 A neural substrate of prediction and reward. Science 275, 1593-1599. (doi:10.1126/science.275.5306.1593)
46. Sugrue, L. P., Corrado, G. S. & Newsome, W. T. 2004 Matching behavior and the representation of value in the parietal cortex. Science 304, 1782-1787. (doi:10.1126/science.1094765)
47. Sutton, R. S. & Barto, A. G. 1998 Reinforcement learning: an introduction. Cambridge, MA: MIT Press.
48. Thrun, S. B. 1992 The role of exploration in learning control. In Handbook of intelligent control: neural, fuzzy, and adaptive approaches (eds D. A. White & D. A. Sofge), pp. 527-559. Florence, KY: Van Nostrand Reinhold.
49. Usher, M., Cohen, J. D., Rajkowski, J. & Aston-Jones, G. 1999 The role of the locus coeruleus in the regulation of cognitive performance. Science 283, 549-554. (doi:10. 1126/science.283.5401.549)
50. Watkinson, S. C., Boddy, L., Burton, K., Darrah, P. R., Eastwood, D., Fricker, M. D. & Tlalka, M. 2005 New approaches to investigating the function of mycelial networks. Mycologist 19, 11-17. (doi:10.1017/S0269915 X05001023)
51. Yeung, N., Botvinick, M. M. & Cohen, J. D. 2004 The neural basis of error detection: conflict monitoring and the error-related negativity. Psychol. Rev. 111, 931-959. (doi:10. 1037/0033-295X.111.4.939)
52. Yu, A. & Dayan, P. 2005 Uncertainty, neuromodulation and attention. Neuron 46, 681-692. (doi:10.1016/j.neuron. 2005.04.026)