Dopamine-associated cached values are not sufficient as the basis for action selection

Proceedings of the National Academy of Sciences of the United States of America

Volumn 111, Issue 51, 2014, Pages 18357-18362

  Hollon, Nick G ^a,b   Arnold, Monica M ^a   Gan, Jerylin O ^a,b   Walton, Mark E ^a,c   Phillips, Paul E M ^a,b  

^a UNIVERSITY OF WASHINGTON SCHOOL OF MEDICINE   (United States)

^b UNIVERSITY OF WASHINGTON   (United States)

^c UNIVERSITY OF OXFORD   (United Kingdom)

Author keywords

Action selection; Cached values; Decision making; Dopamine

Indexed keywords

DOPAMINE;

ANIMAL BEHAVIOR; ANIMAL EXPERIMENT; ARTICLE; ASSOCIATION; CACHED VALUE; CONTROLLED STUDY; COST BENEFIT ANALYSIS; CYCLIC POTENTIOMETRY; DECISION MAKING; DOPAMINE RELEASE; DOPAMINERGIC TRANSMISSION; EXERCISE; FAST SCAN CYCLIC VOLTAMMETRY; MALE; MESOLIMBIC DOPAMINERGIC SYSTEM; NONHUMAN; NUCLEUS ACCUMBENS; RAT; REWARD; ANIMAL; METABOLISM; PHYSIOLOGY; SPRAGUE DAWLEY RAT;

RATTUS;

ANIMALS; DECISION MAKING; DOPAMINE; NUCLEUS ACCUMBENS; RATS; RATS, SPRAGUE-DAWLEY; REWARD;

EID: 84919905513     PISSN: 00278424     EISSN: 10916490     Source Type: Journal    
DOI: 10.1073/pnas.1419770111     Document Type: Article

References (60)

1. Sutton RS, Barto AG (1998) Reinforcement Learning: An Introduction (MIT Press, Cambridge, MA).
2. Daw ND, Doya K (2006) The computational neurobiology of learning and reward. Curr Opin Neurobiol 16(2):199-204.
3. Rangel A, Camerer C, Montague PR (2008) A framework for studying the neurobiology of value-based decision making. Nat Rev Neurosci 9(7):545-556.
4. Kable JW, Glimcher PW (2009) The neurobiology of decision: Consensus and controversy. Neuron 63(6):733-745.
5. Lee D, Seo H, Jung MW (2012) Neural basis of reinforcement learning and decision making. Annu Rev Neurosci 35:287-308.
6. Samejima K, Ueda Y, Doya K, Kimura M (2005) Representation of action-specific reward values in the striatum. Science 310(5752):1337-1340.
7. Lau B, Glimcher PW (2008) Value representations in the primate striatum during matching behavior. Neuron 58(3):451-463.
8. Cai X, Kim S, Lee D (2011) Heterogeneous coding of temporally discounted values in the dorsal and ventral striatum during intertemporal choice. Neuron 69(1):170-182.
9. Tai LH, Lee AM, Benavidez N, Bonci A, Wilbrecht L (2012) Transient stimulation of distinct subpopulations of striatal neurons mimics changes in action value. Nat Neurosci 15(9):1281-1289.
10. Reynolds JN, Hyland BI, Wickens JR (2001) A cellular mechanism of reward-related learning. Nature 413(6851):67-70.
11. Shen W, Flajolet M, Greengard P, Surmeier DJ (2008) Dichotomous dopaminergic control of striatal synaptic plasticity. Science 321(5890):848-851.
12. Pawlak V, Kerr JN (2008) Dopamine receptor activation is required for corticostriatal spike-timing-dependent plasticity. J Neurosci 28(10):2435-2446.
13. Houk JC, Adams JL, Barto AG (1995) A model of how the basal ganglia generate and use neural signals that predict reinforcement. Models of Information Processing in the Basal Ganglia, eds Houk JC, Davis JL, Beiser DG (MIT Press, Cambridge, MA), pp 249-270.
14. Montague PR, Dayan P, Sejnowski TJ (1996) A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci 16(5):1936-1947.
15. SchultzW, Dayan P, Montague PR (1997) A neural substrate of prediction and reward. Science 275(5306):1593-1599.
16. Glimcher PW (2011) Understanding dopamine and reinforcement learning: The dopamine reward prediction error hypothesis. Proc Natl Acad Sci USA 108(Suppl 3):15647-15654.
17. Schultz W (2013) Updating dopamine reward signals. Curr Opin Neurobiol 23(2):229-238.
18. Waelti P, Dickinson A, Schultz W (2001) Dopamine responses comply with basic assumptions of formal learning theory. Nature 412(6842):43-48.
19. Bayer HM, Glimcher PW (2005) Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47(1):129-141.
20. Cohen JY, Haesler S, Vong L, Lowell BB, Uchida N (2012) Neuron-type-specific signals for reward and punishment in the ventral tegmental area. Nature 482(7383):85-88.
21. Steinberg EE, et al. (2013) A causal link between prediction errors, dopamine neurons and learning. Nat Neurosci 16(7):966-973.
22. Hart AS, Rutledge RB, Glimcher PW, Phillips PEM (2014) Phasic dopamine release in the rat nucleus accumbens symmetrically encodes a reward prediction error term. J Neurosci 34(3):698-704.
23. Stopper CM, Tse MT, Montes DR, Wiedman CR, Floresco SB (2014) Overriding phasic dopamine signals redirects action selection during risk/reward decision making. Neuron 84(1):177-189.
24. Roesch MR, Calu DJ, Schoenbaum G (2007) Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat Neurosci 10(12):1615-1624.
25. Fiorillo CD, Tobler PN, Schultz W (2003) Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299(5614):1898-1902.
26. Morris G, Arkadir D, Nevet A, Vaadia E, Bergman H (2004) Coincident but distinct messages of midbrain dopamine and striatal tonically active neurons. Neuron 43(1):133-143.
27. Tobler PN, Fiorillo CD, Schultz W (2005) Adaptive coding of reward value by dopamine neurons. Science 307(5715):1642-1645.
28. Kobayashi S, Schultz W (2008) Influence of reward delays on responses of dopamine neurons. J Neurosci 28(31):7837-7846.
29. Day JJ, Jones JL, Wightman RM, Carelli RM (2010) Phasic nucleus accumbens dopamine release encodes effort- and delay-related costs. Biol Psychiatry 68(3):306-309.
30. Sugam JA, Day JJ, Wightman RM, Carelli RM (2012) Phasic nucleus accumbens dopamine encodes risk-based decision-making behavior. Biol Psychiatry 71(3):199-205.
31. Lak A, Stauffer WR, Schultz W (2014) Dopamine prediction error responses integrate subjective value from different reward dimensions. Proc Natl Acad Sci USA 111(6):2343-2348.
32. Stauffer WR, Lak A, Schultz W (2014) Dopamine reward prediction error responses reflect marginal utility. Curr Biol 24(21):2491-2500.
33. Nasrallah NA, et al. (2011) Risk preference following adolescent alcohol use is associated with corrupted encoding of costs but not rewards by mesolimbic dopamine. Proc Natl Acad Sci USA 108(13):5466-5471.
34. Bromberg-Martin ES, Hikosaka O (2009) Midbrain dopamine neurons signal preference for advance information about upcoming rewards. Neuron 63(1):119-126.
35. Gan JO, Walton ME, Phillips PEM (2010) Dissociable cost and benefit encoding of future rewards by mesolimbic dopamine. Nat Neurosci 13(1):25-27.
36. Pasquereau B, Turner RS (2013) Limited encoding of effort by dopamine neurons in a cost-benefit trade-off task. J Neurosci 33(19):8288-8300.
37. Ravel S, Richmond BJ (2006) Dopamine neuronal responses in monkeys performing visually cued reward schedules. Eur J Neurosci 24(1):277-290.
38. Wanat MJ, Kuhnen CM, Phillips PEM (2010) Delays conferred by escalating costs modulate dopamine release to rewards but not their predictors. J Neurosci 30(36):12020-12027.
39. Fiorillo CD (2013) Two dimensions of value: Dopamine neurons represent reward but not aversiveness. Science 341(6145):546-549.
40. Clark JJ, et al. (2010) Chronic microsensors for longitudinal, subsecond dopamine detection in behaving animals. Nat Methods 7(2):126-129.
41. Kepecs A, Uchida N, Zariwala HA, Mainen ZF (2008) Neural correlates, computation and behavioural impact of decision confidence. Nature 455(7210):227-231.
42. Green DM, Swets JA (1966) Signal Detection Theory and Psychophysics (Wiley, New York).
43. Akaike H (1974) A new look at the statistical model identification. IEEE Trans Automat Contr 19(6):716-723.
44. Burnham KP, Anderson DR (2002) Model Selection and Multimodal Inference: A Practical Information-Theoretic Approach (Springer, New York).
45. Nomoto K, Schultz W, Watanabe T, Sakagami M (2010) Temporally extended dopamine responses to perceptually demanding reward-predictive stimuli. J Neurosci 30(32):10692-10702.
46. de Lafuente V, Romo R (2011) Dopamine neurons code subjective sensory experience and uncertainty of perceptual decisions. Proc Natl Acad Sci USA 108(49):19767-19771.
47. Bromberg-Martin ES, Matsumoto M, Hong S, Hikosaka O (2010) A pallidus-habenuladopamine pathway signals inferred stimulus values. J Neurophysiol 104(2):1068-1076.
48. Daw ND, Niv Y, Dayan P (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 8(12):1704-1711.
49. Ghods-Sharifi S, St Onge JR, Floresco SB (2009) Fundamental contribution by the basolateral amygdala to different forms of decision making. J Neurosci 29(16):5251-5259.
50. Walton ME, Bannerman DM, Alterescu K, Rushworth MF (2003) Functional specialization within medial frontal cortex of the anterior cingulate for evaluating effort-related decisions. J Neurosci 23(16):6475-6479.
51. Rudebeck PH, Walton ME, Smyth AN, Bannerman DM, Rushworth MF (2006) Separate neural pathways process different decision costs. Nat Neurosci 9(9):1161-1168.
52. Hillman KL, Bilkey DK (2010) Neurons in the rat anterior cingulate cortex dynamically encode cost-benefit in a spatial decision-making task. J Neurosci 30(22):7705-7713.
53. Kennerley SW, Behrens TE, Wallis JD (2011) Double dissociation of value computations in orbitofrontal and anterior cingulate neurons. Nat Neurosci 14(12):1581-1589.
54. Amemori K, Graybiel AM (2012) Localized microstimulation of primate pregenual cingulate cortex induces negative decision-making. Nat Neurosci 15(5):776-785.
55. Prévost C, Pessiglione M, Météreau E, Cléry-Melin ML, Dreher JC (2010) Separate valuation subsystems for delay and effort decision costs. J Neurosci 30(42):14080-14090.
56. Palminteri S, et al. (2012) Critical roles for anterior insula and dorsal striatum in punishment-based avoidance learning. Neuron 76(5):998-1009.
57. McHugh SB, et al. (2014) Aversive prediction error signals in the amygdala. J Neurosci 34(27):9024-9033.
58. Padoa-Schioppa C (2011) Neurobiology of economic choice: A good-based model. Annu Rev Neurosci 34:333-359.
59. Kim S, Hwang J, Lee D (2008) Prefrontal coding of temporally discounted values during intertemporal choice. Neuron 59(1):161-172.
60. Louie K, Glimcher PW (2010) Separating value from choice: Delay discounting activity in the lateral intraparietal area. J Neurosci 30(16):5498-5507.