The anatomy of choice: Dopamine and decision-making

Philosophical Transactions of the Royal Society B: Biological Sciences

Volumn 369, Issue 1655, 2014, Pages

  Friston, Karl ^a   Schwartenbeck, Philipp ^a   FitzGerald, Thomas ^a   Moutoussis, Michael ^a   Behrens, Timothy ^a,b   Dolan, Raymond J ^a  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

^b JOHN RADCLIFFE HOSPITAL   (United Kingdom)

Author keywords

Active inference; Agency; Bayesian inference; Bounded rationality; Free energy; Utility theory

Indexed keywords

ANATOMY; BAYESIAN ANALYSIS; COMPUTER SIMULATION; DECISION MAKING; ERROR ANALYSIS; OPTIMIZATION; PHYSIOLOGICAL RESPONSE;

DOPAMINE;

BAYES THEOREM; BIOLOGICAL MODEL; BRAIN; DECISION MAKING; ENTROPY; HUMAN; PHYSIOLOGY;

BAYES THEOREM; BRAIN; CHOICE BEHAVIOR; DOPAMINE; ENTROPY; HUMANS; MODELS, NEUROLOGICAL;

EID: 84907494737     PISSN: 09628436     EISSN: 14712970     Source Type: Journal    
DOI: 10.1098/rstb.2013.0481     Document Type: Article

References (81)

1. Schmidhuber J. 1991 Curious model-building control systems. In Proc. IEEE Int. Joint Conf. on Neural Networks, Singapore, 18–21 Nov. 1991. Neural Networks, vol. 2, pp. 1458–1463. (doi:10. 1109/IJCNN.1991.170605)
2. Klyubin AS, Polani D, Nehaniv CL. 2005 Empowerment: a universal agent-centric measure of control. In Proc. IEEE Congress on Evolutionary Computation, Edinburgh, UK, 5 September 2005, vol. 1, pp. 128-135. (doi:10.1109/CEC.2005.1554676)
3. Tishby N, Polani D. 2010 Information theory of decisions and actions. In Perception–reason–action cycle: models, algorithms and systems (eds V Cutsuridis, A Hussain, J Taylor), pp. 1–37. Berlin, Germany: Springer.
4. Ortega PA, Braun DA. 2011 Information, utility and bounded rationality. In Artificial general intelligence (eds J Schmidhuber, KR Thorisson, M Looks). Lecture Notes in Computer Science, vol. 6830, pp. 269–274. Berlin, Germany: Springer.
5. Wissner-Gross AD, Freer CE. 2013 Causal entropic forces. Phys. Rev. Lett. 110, 168702. (doi:10.1103/ PhysRevLett.110.168702)
6. McKelvey R, Palfrey T. 1995 Quantal response equilibria for normal form games. Games Econ. Behav. 10, 6–38. (doi:10.1006/game.1995.1023)
7. Sharot T, Guitart-Masip M, Korn CW, Chowdhury R, Dolan RJ. 2012 How dopamine enhances an optimism bias in humans. Curr. Biol. 22, 1477–1481. (doi:10.1016/j.cub.2012.05.053)
8. Friston K, Schwartenbeck P, FitzGerald T, Moutoussis M, Behrens T, Raymond J, Dolan RJ. 2013 The anatomy of choice: active inference and agency. Front. Hum. Neurosci. 7, 598. (doi:10.3389/fnhum. 2013.00598)
9. Friston K. 2012 A free energy principle for biological systems. Entropy 14, 2100–2121. (doi:10.3390/ e14112100)
10. Helmholtz H. 1866/1962 Concerning the perceptions in general. In Treatise on physiological optics, 3rd edn. New York, NY: Dover.
11. Ashby WR. 1947 Principles of the self-organizing dynamic system. J. Gen. Psychol. 37, 125–128. (doi:10.1080/00221309.1947.9918144)
12. Conant RC, Ashby RW. 1970 Every good regulator of a system must be a model of that system. Int. J. Syst. Sci. 1, 89–97. (doi:10.1080/ 00207727008920220)
13. Dayan P, Hinton GE, Neal R. 1995 The Helmholtz machine. Neural Comput. 7, 889–904. (doi:10. 1162/neco.1995.7.5.889)
14. Friston K. 2010 The free-energy principle: a unified brain theory? Nat. Rev. Neurosci. 11, 127–138. (doi:10.1038/nrn2787)
15. Camerer CF. 2003 Behavioural studies of strategic thinking in games. Trends Cogn. Sci. 7, 225–231. (doi:10.1016/S1364-6613(03)00094-9)
16. Daw ND, Doya K. 2006 The computational neurobiology of learning and reward. Curr. Opin. Neurobiol. 16, 199–204. (doi:10.1016/j.conb.2006. 03.006)
17. Dayan P, Daw ND. 2008 Decision theory, reinforcement learning, and the brain. Cogn. Affect Behav. Neurosci. 8, 429–453. (doi:10.3758/CABN.8.4.429)
18. Savage LJ. 1954 The foundations of statistics. New York, NY: Wiley.
19. Von Neumann J, Morgenstern O. 1944 Theory of games and economic behavior. Princeton, NJ: Princeton University Press.
20. Simon HA. 1956 Rational choice and the structure of the environment. Psychol. Rev. 63, 129–138. (doi:10.1037/h0042769)
21. Ortega PA, Braun DA. 2013 Thermodynamics as a theory of decision-making with information-processing costs. Proc. R. Soc. A 469, 20120683. (doi:10.1098/rspa.2012.0683)
22. Haile PA, Hortaçsu A, Kosenok G. 2008 On the empirical content of quantal response equilibrium. Am. Econ. Rev. 98, 180–200. (doi:10.1257/aer. 98.1.180)
23. Luce RD. 1959 Individual choice behavior. Oxford, UK: Wiley.
24. Fudenberg D, Kreps D. 1993 Learning mixed equilibria. Games Econ. Behav. 5, 320–367. (doi:10. 1006/game.1993.1021)
25. Sutton RS, Barto AG. 1998 Reinforcement learning: an introduction. Cambridge, MA: MIT Press.
26. Cohen JD, McClure SM, Yu AJ. 2007 Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration. Phil. Trans. R. Soc. B 362, 933–942. (doi:10.1098/rstb. 2007.2098)
27. Beal MJ. 2003 Variational algorithms for approximate bayesian inference. PhD thesis, University College London, London, UK.
28. MacKay DJC. 2003 Information theory, inference and learning algorithms. Cambridge, UK: Cambridge University Press.
29. Schultz W, Dayan P, Montague PR. 1997 A neural substrate of prediction and reward. Science 275, 1593–1599. (doi:10.1126/science.275.5306.1593)
30. Kapur S. 2003 Psychosis as a state of aberrant salience: a framework linking biology, phenomenology, and pharmacology in schizophrenia. Am. J. Psychiatry 160, 13–23. (doi:10.1176/appi.ajp.160.1.13)
31. Fletcher PC, Frith CD. 2009 Perceiving is believing: a Bayesian approach to explaining the positive symptoms of schizophrenia. Nat. Rev. Neurosci. 10, 48–58. (doi:10.1038/nrn2536)
32. Pellicano E, Burr D. 2012 When the world becomes ‘too real’: a Bayesian explanation of autistic perception. Trends Cogn. Sci. 16, 504–510. (doi:10. 1016/j.tics.2012.08.009)
33. Adams RA, Stephan KE, Brown HR, Frith CD, Friston KJ. 2013 The computational anatomy of psychosis. Front. Psychiatry 4, 47. (doi:10.3389/ fpsyt.2013.00047)
34. Friston KJST et al. 2012 Dopamine, affordance and active inference. PLoS Comput. Biol. 8, e1002327. (doi:10.1371/journal.pcbi.1002327)
35. Frank MJ, Scheres A, Sherman SJ. 2007 Understanding decision-making deficits in neurological conditions: insights from models of natural action selection. Phil. Trans. R. Soc. B 362, 1641–1654. (doi:10.1098/rstb.2007.2058)
36. Cisek P. 2007 Cortical mechanisms of action selection: the affordance competition hypothesis. Phil. Trans. R. Soc. B 362, 1585–1599. (doi:10. 1098/rstb.2007.2054)
37. Thompson WR. 1933 On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika 25, 285–294. (doi:10.1093/biomet/25.3-4.285)
38. Ortega PADA. 2010 A minimum relative entropy principle for learning and acting. 38, 475–511.
39. Dayan P, Hinton GE. 1997 Using expectation maximization for reinforcement learning. Neural Comput. 9, 271–278. (doi:10.1162/neco.1997. 9.2.271)
40. Rao RP, Ballard DH. 1999 Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat. Neurosci. 2, 79–87. (doi:10.1038/4580)
41. Watkins CJ, Dayan P. 1992 Q-learning. Mach. Learn. 8, 279–292. (doi:10.1007/BF00992698)
42. van den Broek JL, Wiegerinck WAJJ, Kappen HJ. 2010 Risk-sensitive path integral control. UAI 6, 1–8.
43. Jaynes ET. 1957 Information theory and statistical mechanics. Phys. Rev. Ser. II 106, 620–630.
44. Kakade S, Dayan P. 2002 Dopamine: generalization and bonuses. Neural Netw. 15, 549–559. (doi:10. 1016/S0893-6080(02)00048-5)
45. Kappen HJ, Gomez Y, Opper M. 2012 Optimal control as a graphical model inference problem. Mach. Learn. 87, 159–182. (doi:10.1007/s10994-012-5278-7)
46. Fox C, Roberts S. 2011 A tutorial on variational Bayes. In Artificial intelligence review. New York, NY: Spinger.
47. Friston K, Samothrakis S, Montague R. 2012 Active inference and agency: optimal control without cost functions. Biol. Cybern. 106, 523–541. (doi:10. 1007/s00422-012-0512-8)
48. Canolty RT et al. 2006 High gamma power is phase-locked to theta oscillations in human neocortex. Science 313, 1626–1628. (doi:10.1126/science. 1128115)
49. Goldman-Rakic PS. 1997 The cortical dopamine system: role in memory and cognition. Adv. Pharmacol. 42, 707–711. (doi:10.1016/S1054-3589(08)60846-7)
50. Moran RJ, Symmonds M, Stephan KE, Friston K., Dolan RJ. 2011 An in vivo assay of synaptic function mediating human cognition. Curr. Biol. 21, 1320-1325. (doi:10.1016/j.cub.2011.06.053)
51. O’Doherty J, Dayan P, Schultz J, Deichmann R, Friston K, Dolan RJ. 2004 Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304, 452–454. (doi:10.1126/science.1094285)
52. Surmeier D, Plotkin J, Shen W. 2009 Dopamine and synaptic plasticity in dorsal striatal circuits controlling action selection. Curr. Opin. Neurobiol. 19, 621–628. (doi:10.1016/j.conb.2009.10.003)
53. Goldman-Rakic PS, Lidow MS, Smiley JF, Williams MS. 1992 The anatomy of dopamine in monkey and human prefrontal cortex. J. Neural Transm. Suppl. 36, 163–177.
54. Solway A, Botvinick M. 2012 Goal-directed decision making as probabilistic inference: a computational framework and potential neural correlates. Psychol. Rev. 119, 120–154. (doi:10.1037/a0026435)
55. Jocham G, Hunt LT, Near J, Behrens TE. 2012 A mechanism for value-guided choice based on the excitation–inhibition balance in prefrontal cortex. Nat. Neurosci. 15, 960–961. (doi:10.1038/nn.3140)
56. Geisler S, Derst C, Veh RW, Zahm DS. 2007 Glutamatergic afferents of the ventral tegmental area in the rat. J. Neurosci. 27, 5730–5743. (doi:10. 1523/JNEUROSCI.0012-07.2007)
57. Smith Y, Bevan MD, Shink E, Bolam JP. 1998 Microcircuitry of the direct and indirect pathways of the basal ganglia. Neuroscience 86, 353–387. (doi:10.1016/S0306-4522(97)00608-8)
58. 3H]SCH23390. Neuroscience 40, 657–671. (doi:10.1016/0306-4522(91)90003-7)
59. Redgrave P, Gurney K. 2006 The short-latency dopamine signal: a role in discovering novel actions? Nat. Rev. Neurosci. 7, 967–975. (doi:10. 1038/nrn2022)
60. Fiorillo CD, Tobler PN, Schultz W. 2003 Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299, 1898–1902. (doi:10.1126/science.1077349)
61. Schultz W. 2007 Multiple dopamine functions at different time courses. Annu. Rev. Neurosci. 30, 259–288. (doi:10.1146/annurev.neuro.28.061604.135722)
62. Schultz W. 1998 Predictive reward signal of dopamine neurons. J. Neurophysiol. 80, 1–27.
63. Mathys C, Daunizeau J, Friston K., Stephan KE. 2011 A Bayesian foundation for individual learning under uncertainty. Front. Hum. Neurosci. 5, 39. (doi:10. 3389/fnhum.2011.00039)
64. Gurney K, Prescott TJ, Redgrave P. 2001 A computational model of action selection in the basal ganglia. I. A new functional anatomy. Biol. Cybern. 84, 401–410. (doi:10.1007/PL00007984)
65. Niv Y. 2007 Cost, benefit, tonic, phasic: what do response rates tell us about dopamine and motivation? Ann. NY Acad. Sci. 1104, 357–376. (doi:10.1196/annals.1390.018)
66. Beeler JA, Frank MJ, McDaid J, Alexander E, Turkson S, Bernandez MS, McGehee DS, Zhuang X. 2012 A role for dopamine-mediated learning in the pathophysiology and treatment of Parkinson’s disease. Cell Rep. 2, 1747–1761. (doi:10.1016/j. celrep.2012.11.014)
67. Dayan P. 2012 Twenty-five lessons from computational neuromodulation. Neuron 76, 240–256. (doi:10.1016/j.neuron.2012.09.027)
68. Friston K. 2011 What is optimal about motor control? Neuron 72, 488-498. (doi:10.1016/j. neuron.2011.10.018)
69. Fuster JM. 2004 Upper processing stages of the perception–action cycle. Trends Cogn. Sci. 8, 143–145.(doi:10.1016/j.tics.2004.02.004)
70. Samejima K, Ueda Y, Doya K, Kimura M. 2005 Representation of action-specific reward values in the striatum. Science 310, 1337–1340.(doi:10. 1126/science.1115270)
71. Lau B, Glimcher PW. 2007 Action and outcome encoding in the primate caudate nucleus. J. Neurosci. 27, 14502–14514. (doi:10.1523/ JNEUROSCI.3060-07.2007)
72. Tai LH, Lee AM, Benavidez N, Bonci A, Wilbrecht L. 2012 Transient stimulation of distinct subpopulations of striatal neurons mimics changes in action value. Nat. Neurosci. 15, 1281-1289. (doi:10.1038/nn.3188)
73. Kravitz AV, Tye LD, Kreitzer AC. 2012 Distinct roles for direct and indirect pathway striatal neurons in reinforcement. Nat. Neurosci. 15, 816–818. (doi:10. 1038/nn.3100)
74. Schultz W et al. 2008 Explicit neural signals reflecting reward uncertainty. Phil. Trans. R. Soc. B 363, 3801–3811. (doi:10.1098/rstb.2008.0152)
75. Fiorillo CD, Newsome WT, Schultz W. 2008 The temporal precision of reward prediction in dopamine neurons. Nat. Neurosci. 11, 966–973. (doi:10.1038/nn.2159)
76. Zokaei N, Gorgoraptis N, Husain M. 2012 Dopamine modulates visual working memory precision. J. Vis. 12, 350. (doi:10.1167/12.9.350)
77. Galea JM, Bestmann S, Beigi M, Jahanshahi M, Rothwell JC. 2012 Action reprogramming in Parkinson’s disease: response to prediction error is modulated by levels of dopamine. J. Neurosci. 32, 542–550. (doi:10.1523/JNEUROSCI. 3621-11.2012)
78. Coull JT, Cheng RK, Meck W. 2011 Neuroanatomical and neurochemical substrates of timing. Neuropsychopharmacology 36, 3–25. (doi:10.1038/ npp.2010.113)
79. Toussaint M, Storkey A. 2006 Probabilistic inference for solving discrete and continuous state Markov Decision Processes. In Proc. 23nd Int. Conf. on Machine Learning, pp. 945–952. New York, NY: Association for Computing Machinery. (doi:10.1145/ 1143844.1143963)
80. Gläscher J, Daw N, Dayan P, O’Doherty JP. 2010 States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66, 585–595. (doi:10.1016/j.neuron.2010.04. 016)
81. De Martino B, Fleming SM, Garrett N, Dolan RJ. 2012 Confidence in value-based choice. Nat. Neurosci. 16, 105–110. (doi:10.1038/nn.3279)