Navigating complex decision spaces: Problems and paradigms in sequential choice

Psychological Bulletin

Volumn 140, Issue 2, 2014, Pages 466-486

  Walsh, Matthew M ^a   Anderson, John R ^b  

^a AIR FORCE RESEARCH LABORATORY   (United States)

^b CARNEGIE MELLON UNIVERSITY   (United States)

Author keywords

Reinforcement learning; Sequential choice; Temporal credit assignment

Indexed keywords

DOPAMINE;

ANIMAL; ARTICLE; BASAL GANGLION; BIOLOGICAL MODEL; BRAIN; CEREBELLUM; DECISION MAKING; HUMAN; LEARNING; PHYSIOLOGY; PREFRONTAL CORTEX; PROBABILITY; PSYCHOLOGICAL MODEL; REINFORCEMENT; TEMPORAL LOBE; TIME;

ANIMALS; BASAL GANGLIA; BRAIN; CEREBELLUM; CHOICE BEHAVIOR; DECISION MAKING; DOPAMINE; HUMANS; LEARNING; MARKOV CHAINS; MODELS, NEUROLOGICAL; MODELS, PSYCHOLOGICAL; PREFRONTAL CORTEX; REINFORCEMENT (PSYCHOLOGY); TEMPORAL LOBE; TIME FACTORS;

EID: 84894423717     PISSN: 00332909     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0033455     Document Type: Article

References (210)

1. Abler, B., Walter, H., Erk, S., Kammerer, H., & Spitzer, M. (2006). Prediction error as a linear function of reward probability is coded in human nucleus accumbens. NeuroImage, 31, 790-795
2. Adams, C. D., & Dickinson, A. (1981). Instrumental responding following reinforcer devaluation. The Quarterly Journal of Experimental Psychology B: Comparative and Physiological Psychology, 33B, 109-121.
3. Anderson, J. R. (2007). How can the human mind occur in the physical universe? New York, NY: Oxford University Press.
4. Anderson, J. R., Albert, M. V., & Fincham, J. M. (2005). Tracing problem solving in real time: fMRI analysis of the subject-paced tower of Hanoi. Journal of Cognitive Neuroscience, 17, 1261-1274
5. Asaad, W. F., Rainer, G., & Miller, E. K. (2000). Task-specific neural activity in the primate prefrontal cortex. Journal of Neurophysiology, 84, 451-459
6. Baker, T. E., & Holroyd, C. B. (2009). Which way do I go? Neural activation in response to feedback and spatial processing in a virtual T-Maze. Cerebral Cortex, 19, 1708-1722
7. Balleine, B. W. (2005). Neural bases of food-seeking: Affect, arousal and reward in corticostriatolimbic circuits. Physiology & Behavior, 86, 717- 730
8. Balleine, B. W., Garner, C., Gonzalez, F., & Dickinson, A. (1995). Motivational control of heterogeneous instrumental chains. Journal of Experimental Psychology: Animal Behavior Processes, 21, 203-217
9. Balleine, B. W., & O'doherty, J. P. (2010). Human and rodent homologies in action control: Corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology, 35, 48-69
10. Barto, A. G., & Mahadevan, S. (2003). Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems: Theory and Applications, 13, 41-77
11. Berns, G. S., Mcclure, S. M., Pagnoni, G., & Montague, P. R. (2001). Predictability modulates human brain response to reward. The Journal of Neuroscience, 21, 2793-2798.
12. Berridge, K. C. (2007). The debate over dopamine's role in reward: The case for incentive salience. Psychopharmacology, 191, 391-431
13. Blodgett, H. C. (1929). The effect of the introduction of reward upon the maze performance of rats. University of California Publications in Psychology, 4, 113-134.
14. Bogacz, R., Mcclure, S. M., Li, J., Cohen, J. D., & Montague, P. R. (2007). Short-term memory traces for action bias in human reinforcement learning. Brain Research, 1153, 111-121
15. Botvinick, M. (2007). Conflict monitoring and decision making: Reconciling two perspectives on anterior cingulate function. Cognitive, Affective & Behavioral Neuroscience, 7, 356-366
16. Botvinick, M. M., Braver, T. S., Barch, D. M., Carter, C. S., & Cohen, J. D. (2001). Conflict monitoring and cognitive control. Psychological Review, 108, 624-652
17. Botvinick, M. M., Niv, Y., & Barto, A. C. (2009). Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective. Cognition, 113, 262-280
18. Bray, S., Shimojo, S., & O'Doherty, J. P. (2010). Human medial orbitofrontal cortex is recruited during experience of imagined and real rewards. Journal of Neurophysiology, 103, 2506-2512
19. Brogden, W. J. (1939). Sensory pre-conditioning. Journal of Experimental Psychology, 25, 323-332
20. Brown, J. W., & Braver, T. S. (2005). Learned predictions of error likelihood in the anterior cingulate cortex. Science, 307, 1118-1121
21. Buel, J. (1935). Differential errors in animal mazes. Psychological Bulletin, 32, 67-99
22. Bunge, S. A. (2004). How we use rules to select actions: A review of evidence from cognitive neuroscience. Cognitive, Affective & Behavioral Neuroscience, 4, 564-579
23. Bunsey, M., & Eichenbaum, H. (1996). Conservation of hippocampal memory function in rats and humans. Nature, 379, 255-257
24. Busemeyer, J. R., & Pleskac, T. J. (2009). Theoretical tools for understanding and aiding dynamic decision making. Journal of Mathematical Psychology, 53, 126-138
25. Bush, R. R., & Mosteller, F. (1955). Stochastic models for learning. Oxford, England: Wiley.
26. Bussey, T. J., Wise, S. P., & Murray, E. A. (2001). The role of ventral and orbital prefrontal cortex in conditional visuomotor learning and strategy use in Rhesus monkeys (Macaca mulatta). Behavioral Neuroscience, 115, 971-982
27. Capaldi, E. J. (1957). The effect of different amounts of alternating partial reinforcement on resistance to extinction. The American Journal of Psychology, 70, 451-452
28. Cardinal, R. N., Parkinson, J. A., Hall, J., & Everitt, B. J. (2002). Emotion and motivation: The role of the amygdala, ventral striatum, and prefrontal cortex. Neuroscience and Biobehavioral Reviews, 26, 321-352
29. Carter, C. S., Braver, T. S., Barch, D. M., Botvinick, M. M., Noll, D., & Cohen, J. D. (1998). Anterior cingulate cortex, error detection, and the online monitoring of performance. Science, 280, 747-749
30. Cohen, M. X., & Ranganath, C. (2007). Reinforcement learning signals predict future decisions. The Journal of Neuroscience, 27, 371-378
31. Cohen, N. J., & Eichenbaum, H. (1993). Memory, amnesia, and the hippocampal system. Cambridge, MA: MIT Press.
32. Courville, A. C., Daw, N. D., & Touretzky, D. S. (2006). Bayesian theories of conditioning in a changing world. Trends in Cognitive Sciences, 10, 295-300
33. Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011). Model-based influences on humans' choices and striatal prediction errors. Neuron, 69, 1204-1215
34. Daw, N. D., Niv, Y., & Dayan, P. (2005). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience, 8, 1704-1711
35. Daw, N. D., O'doherty, J. P., Dayan, P., Seymour, B., & Dolan, R. J. (2006). Cortical substrates for exploratory decisions in humans. Nature, 441, 876-879
36. Dayan, P. (2012). Instrumental vigour in punishment and reward. European Journal of Neuroscience, 35, 1152-1168
37. Dayan, P., & Daw, N. D. (2008). Decision theory, reinforcement learning, and the brain. Cognitive, Affective & Behavioral Neuroscience, 8, 429- 453
38. Dayan, P., & Niv, Y. (2008). Reinforcement learning: The good, the bad and the ugly. Current Opinion in Neurobiology, 18, 185-196
39. de Groot, A. D. (1978). Thought and choice in chess (2nd ed.). The Hague, the Netherlands: Mouton. (Original work published 1946).
40. Delgado, M. R., Locke, H. M., Stenger, V. A., & Fiez, J. A. (2003). Dorsal striatum responses to reward and punishment: Effects of valence and magnitude manipulations. Cognitive, Affective & Behavioral Neuroscience, 3, 27-38.
41. Dickinson, A., Squire, S., Varga, Z., & Smith, J. W. (1998). Omission learning after instrumental pretraining. The Quarterly Journal of Experimental Psychology B: Comparative and Physiological Psychology, 51B, 271-286.
42. Dietterich, T. G. (2000). Hierarchical reinforcement learning with the MAXQ value function decomposition. Journal of Artificial Intelligence Research, 13, 227-303.
43. Donahoe, J. W., Burgos, J. E., & Palmer, D. C. (1993). A selectionist approach to reinforcement. Journal of the Experimental Analysis of Behavior, 60, 17-40.
44. Doshi-Velez, F., & Ghahramani, Z. (2011). A comparison of human and agent reinforcement learning. In L. Carlson, C. Hoelscher, & T. F. Shipley (Eds.), Proceedings of the 33rd annual conference of the Cognitive Science Society (pp. 2703-2708). Austin, TX: Cognitive Sciences Society.
45. Doya, K. (1999). What are the computations of the cerebellum, the basal ganglia and the cerebral cortex? Neural Networks, 12, 961-974.
46. Doya, K. (2000). Metalearning, neuromodulation, and emotion. In G. Hatano, N. Okada, & H. Tanabe (Eds.), Affective minds (pp. 101-104). Amsterdam, the Netherlands: Elsevier Science.
47. Drewnowski, A., & Murdock, B. B. (1980). The role of auditory features in memory span for words. Journal of Experimental Psychology: Human Learning and Memory, 6, 319-332.
48. Dunning, J. P., & Hajcak, G. (2007). Error-related negativities elicited by monetary loss and cues that predict loss. NeuroReport, 18, 1875-1878.
49. Elliott, R., Newman, J. L., Longe, O. A., & Deakin, J. F. W. (2004). Instrumental responding for rewards is associated with enhanced neuronal response in subcortical reward systems. NeuroImage, 21, 984- 990.
50. Falkenstein, M., Hohnsbein, J., Hoormann, J., & Blanke, L. (1991). Effects of crossmodal divided attention on late ERP components: II. Error processing in choice reaction tasks. Electroencephalography & Clinical Neurophysiology, 78, 447-455.
51. Fermin, A., Yoshida, T., Ito, M., Yoshimoto, J., & Doya, K. (2010). Evidence for model-based action planning in a sequential finger movement task. Journal of Motor Behavior, 42, 371-379.
52. Fiorillo, C. D., Tobler, P. N., & Schultz, W. (2003). Discrete coding of reward probability and uncertainty by dopamine neurons. Science, 299, 1898-1902.
53. Foster, D. J., & Wilson, M. A. (2006). Reverse replay of behavioural sequences in hippocampal place cells during the awake state. Nature, 440, 680-683.
54. Frank, M. J., & Claus, E. D. (2006). Anatomy of a decision: Striatoorbitofrontal interactions in reinforcement learning, decision making, and reversal. Psychological Review, 113, 300-326.
55. Frederick, S., Loewenstein, G., & O'Donoghue, T. (2002). Time discounting and time preference: A critical review. Journal of Economic Literature, 40, 351-401.
56. Fu, W. T., & Anderson, J. R. (2006). From recurrent choice to skill learning: A reinforcement-learning model. Journal of Experimental Psychology: General, 135, 184-206.
57. Fu, W. T., & Anderson, J. R. (2008a). Dual learning processes in interactive skill acquisition. Journal of Experimental Psychology: Applied, 14, 179-191.
58. Fu, W. T., & Anderson, J. R. (2008b). Solving the credit assignment problem: Explicit and implicit learning of action sequences with probabilistic outcomes. Psychological Research, 72, 321-330.
59. Fu, W. T., & Gray, W. D. (2004). Resolving the paradox of the active user: Stable suboptimal performance in interactive tasks. Cognitive Science, [28,] 901-935.
60. Fuster, J. M. (1997). The prefrontal cortex: Anatomy, physiology, and neuropsychology of the frontal lobe. Philadelphia, PA: Lippincott- Raven.
61. Gehring, W. J., Goss, B., Coles, M. G. H., Meyer, D. E., & Donchin, E. (1993). A neural system for error detection and compensation. Psychological Science, 4, 385-390.
62. Gershman, S. J., Blei, D., & Niv, Y. (2010). Context, learning, and extinction. Psychological Review, 117, 197-209.
63. Gershman, S. J., Markman, A. B., & Otto, A. R. (in press). Retrospective revaluation in sequential decision making: A tale of two systems. Journal of Experimental Psychology: General.
64. Gershman, S. J., & Niv, Y. (2012). Exploring a latent cause theory of classical conditioning. Learning & Behavior, 40, 255-268.
65. Gläscher, J., Daw, N., Dayan, P., & O'doherty, J. P. (2010). States versus rewards: Dissociable neural prediction error signals underlying modelbased and model-free reinforcement learning. Neuron, 66, 585-595.
66. Gläscher, J., Hampton, A. N., & O'doherty, J. P. (2008). Determining a role for ventromedial prefrontal cortex in encoding action-based value signals during reward-related decision making. Cerebral Cortex, 19, 483-495.
67. Gray, W. D., Sims, C. R., Fu, W. T., & Schoelles, M. J. (2006). The soft constraints hypothesis: A rational analysis approach to resource allocation for interactive behavior. Psychological Review, 113, 461-482.
68. Gureckis, T. M., & Love, B. C. (2009). Short-term gains, long-term pains: How cues about state aid learning in dynamic environments. Cognition, 113, 293-313.
69. Haber, S. N., Fudge, J. L., & McFarland, N. R. (2000). Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. Journal of Neuroscience, 20, 2369-2382.
70. Haber, S. N., Lynd-Balta, E., Klein, C., & Groenewegen, H. J. (1990). Topographic organization of the ventral striatal efferent projections in the rhesus monkey: An anterograde tracing study. Journal of Comparative Neurology, 293, 282-298.
71. Hampton, A. N., Bossaerts, P., & O'doherty, J. P. (2006). The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. The Journal of Neuroscience, 26, 8360- 8367.
72. Hayes, J. R. (1965). Problem topology and the solution process. Journal of Verbal Learning & Verbal Behavior, 4, 371-379.
73. Herrnstein, R. J., Loewenstein, G. F., Prelec, D., & Vaughan, W. (1993). Utility maximization and melioration: Internalities in individual choice. Journal of Behavioral Decision Making, 6, 149-185.
74. Herrnstein, R. J., & Prelec, D. (1991). Melioration: A theory of distributed choice. Journal of Economic Perspectives, 5, 137-156.
75. Heyman, G. M., & Dunn, B. (2002). Decision biases and persistent illicit drug use: An experimental study of distributed choice and addiction. Drug and Alcohol Dependence, 67, 193-203.
76. Holland, P. C., & Rescorla, R. (1975). The effect of two ways of devaluing the unconditioned stimulus after first- and second-order appetitive conditioning. Journal of Experimental Psychology: Animal Behavior Processes, 1, 355-363.
77. Holroyd, C. B., & Coles, M. G. H. (2002). The neural basis of human error processing: Reinforcement learning, dopamine, and the error-relatednegativity. Psychological Review, 109, 679-709.
78. Holroyd, C. B., Krigolson, O. E., & Lee, S. (2011). Reward positivity elicited by predictive cues. NeuroReport, 22, 249-252.
79. Hoshi, E., Shima, K., & Tanji, J. (2000). Neuronal activity in the primate prefrontal cortex in the process of motor selection based on two behavioral rules. Journal of Neurophysiology, 83, 2355-2373.
80. Hull, C. L.. (1932) The goal gradient hypothesis and maze learning. Psychological Review, 39, 25-43.
81. Hull, C. L.. (1943). Principles of behavior: An introduction to behavioral theory. New York, NY: Appleton-Century-Crofts.
82. Huys, Q. J. M., Eshel, N., O'Nions, E., Sheridan, L., Dayan, P., & Roiser, J. P.. (2012). Bonsai trees in your head: How the Pavlovian system sculpts goal-directed choices by pruning decision trees. PLOS Computational Biology, 8, e1002410.
83. Hyman, S. E., & Malenka, R. C.. (2001). Addiction and the brain: The neurobiology of compulsion and its persistence. Nature Reviews Neuroscience, 2, 695-703.
84. Imamizu, H., & Kawato, M.. (2009). Brain mechanisms for predictive control by switching internal models: Implications for higher-order cognitive functions. Psychological Research, 73, 527-544.
85. Ito, M. (2008). Control of mental activities by internal models in the cerebellum. Nature Reviews Neuroscience, 9, 304-313.
86. Izquierdo, A. D., Suda, R. K., & Murray, E. A.. (2004). Bilateral orbital prefrontal cortex lesions in rhesus monkeys disrupt choices guided by both reward value and reward contingency. The Journal of Neuroscience, 24, 7540-7548.
87. James, W.. (1950). The principles of psychology. New York, NY: Dover. (Original work published 1890).
88. Janssen, C. P., & Gray, W. D.. (2012). When, what, and how much to reward in reinforcement learning-based models of cognition. Cognitive Science, 36, 333-358.
89. Joel, D., Niv, Y., & Ruppin, E.. (2002). Actor-critic models of the basal ganglia: New anatomical and computational perspectives. Neural Networks, 15, 535-547.
90. Johnson, A., & Redish, A. D.. (2007). Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. The Journal of Neuroscience, 27, 12176 -12189.
91. Kaelbling, L. P., Littman, M. L., & Cassandra, A. R.. (1998). Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101, 99-134.
92. Kemp, C., & Tenenbaum, J. B.. (2008). The discovery of structural form. PNAS Proceedings of the National Academy of Sciences of the United States of America, 105, 10687-10692.
93. Kennerley, S. W., Walton, M. E., Behrens, T. E. J., Buckley, M. J., & Rushworth, M. F. S.. (2006). Optimal decision making and the anterior cingulate cortex. Nature Neuroscience, 9, 940-947.
94. Keramati, M., Dezfouli, A., & Piray, P.. (2011). Speed/accuracy trade-off between the habitual and the goal-directed processes. PLOS Computational Biology, 7, e1002055.
95. Kerns, J. G., Cohen, J. D., Macdonald, A. W., Cho, R. Y., Stenger, V. A., & Carter, C. S. (2004). Anterior cingulate conflict monitoring and adjustments in control. Science, 303, 1023-1026.
96. Killcross, S., & Coutureau, E. (2003). Coordination of actions and habits in the medial prefrontal cortex of rats. Cerebral Cortex, 13, 400-408.
97. Knutson, B., Taylor, J., Kaufman, M., Peterson, R., & Glover, G. (2005). Distributed neural representation of expected value. The Journal of Neuroscience, 25, 4806-4812.
98. Kobayashi, S., & Schultz, W. (2008). Influences of reward delays on responses of dopamine neurons. The Journal of Neuroscience, 28, 7837- 7846.
99. Liao, Y., Gramann, K., Feng, W., Deák, G. O., & Li, H. (2011). This ought to be good: Brain activity accompanying positive and negative expectations and outcomes. Psychophysiology, 48, 1412-1419.
100. Liljeholm, M., Tricomi, E., O'doherty, J. P., & Balleine, B. W. (2011). Neural correlates of instrumental contingency learning: Differential effects of action-reward conjunction and disjunction. The Journal of Neuroscience, 31, 2474-2480.
101. Lin, L. J. (1992). Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning, 8, 293-321.
102. Loch, J., & Singh, S. (1998). Using eligibility traces to find the best memoryless policy in partially observable Markov decision processes. In J. W. Shavlik (Ed.), Proceedings of the fifteenth international conference on machine learning (pp. 323-331). San Francisco, CA: Morgan Kaufmann.
103. Lubow, R. (1989). Latent inhibition and conditioned attention theory. Cambridge, England: Cambridge University Press.
104. Luce, R. D. (1977). The choice axiom after twenty years. Journal of Mathematical Psychology, 15, 215-233.
105. Madden, G. J., Petry, N. M., Badger, G. J., & Bickel, W. K. (1997). Impulsive and self-control choices in opioid-dependent patients and non-drug-using control participants: Drug and monetary rewards. Experimental and Clinical Psychopharmacology, 5, 256-262.
106. Maia, T. V. (2009). Fear conditioning and social groups: Statistics, not genetics. Cognitive Science, 33, 1232-1251.
107. Maia, T. V. (2010). Two-factor theory, the actor-critic model, and conditioned avoidance. Learning & Behavior, 38, 50-67.
108. Maia, T. V., & Frank, M. J. (2011). From reinforcement learning models to psychiatric and neurological disorders. Nature Neuroscience, 14, 154-162.
109. Mansouri, F. A., Matsumoto, K., & Tanaka, K. (2006). Prefrontal cell activities related to monkeys' success and failure in adapting to rule changes in a Wisconsin Card Sorting test analog. The Journal of Neuroscience, 26, 2745-2756.
110. Mccallum, R. A. (1995). Instance-based utile distinctions for reinforcement learning with hidden states. In A. Prieditis & S. J. Russell (Eds.), The proceedings of the twelfth international machine learning conference (pp. 387-395). San Francisco, CA: Morgan Kaufmann.
111. Mcclure, S. M., Berns, G. S., & Montague, P. R. (2003). Temporal prediction errors in a passive learning task activate human striatum. Neuron, 38, 339-346.
112. McDannald, M. A., Lucantonio, F., Burke, K. A., Niv, Y., & Schoenbaum, G. (2011). Ventral striatum and orbitofrontal cortex are both required for model-based, but not model-free, reinforcement learning. The Journal of Neuroscience, 31, 2700-2705.
113. McDowell, J. J. (2004). A computational model of selection by consequences. Journal of the Experimental Analysis of Behavior, 81, 297- 317.
114. Miltner, W. H. R., Braun, C. H., & Coles, M. G. H. (1997). Event-related brain potentials following incorrect feedback in a time-estimation task: Evidence for a "generic" neural system for error detection. Journal of Cognitive Neuroscience, 9, 788-798.
115. Minsky, M. (1963). Steps toward artificial intelligence. In E. A. Feigenbaum & J. Feldman (Eds.), Computers and thought (pp. 406-450). New York, NY: McGraw-Hill.
116. Mobini, S., Chiang, T. J., Ho, M. Y., Bradshaw, C. M., & Szabadi, E. (2000). Effects of central 5-hydroxytrptamine depletion on sensitivity to delayed and probabilistic reinforcement. Psychopharmacology, 152, 390-397.
117. Montague, P. R., & Berns, G. S. (2002). Neural economics and the biological substrates of valuation. Neuron, 36, 265-284.
118. Montague, P. R., Dayan, P., & Sejnowski, T. J. (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning. The Journal of Neuroscience, 16, 1936-1947.
119. Montague, P. R., Hyman, S. E., & Cohen, J. D. (2004). Computational roles for dopamine in behavioural control. Nature, 431, 760-767.
120. Morris, G., Nevet, A., Arkadir, D., Vaadia, E., & Bergman, H. (2006). Midbrain dopamine neurons encode decisions for future action. Nature Neuroscience, 9, 1057-1063.
121. Muhammad, R., Wallis, J. D., & Miller, E. K. (2006). A comparison of abstract rules in the prefrontal cortex, premotor cortex, inferior temporal cortex, and striatum. Journal of Cognitive Neuroscience, 18, 974-989.
122. Mushiake, H., Saito, N., Sakamoto, K., Itoyama, Y., & Tanji, J. (2006). Activity in the lateral prefrontal cortex reflects multiple steps of future events in action plans. Neuron, 50, 631-641.
123. Newell, A., & Simon, H. (1972). Human problem solving. Englewood Cliffs, NJ: Prentice-Hall.
124. Niv, Y. (2009). Reinforcement learning in the brain. Journal of Mathematical Psychology, 53, 139-154.
125. O'doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003). Temporal difference models and reward-related learning in the human brain. Neuron, 38, 329 -337.
126. O'doherty, J. P., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004). Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science, 304, 452-454.
127. O'doherty, J. P., Hampton, A., & Kim, H. (2007). Model-based fMRI and its application to reward learning and decision making. Annals of the New York Academy of Science, 1104, 35-53.
128. O'Keefe, J., & Nadel, L. (1978). The hippocampus as a cognitive map. Oxford, England: Clarendon.
129. O'Reilly, R. C., & Frank, M. J. (2006). Making working memory work: A computational model of learning in prefrontal cortex and basal ganglia. Neural Computation, 18, 283-328.
130. Ostlund, S. B., & Balleine, B. W. (2005). Lesions of medial prefrontal cortex disrupt the acquisition but not the expression of goal-directed learning. The Journal of Neuroscience, 25, 7763-7770.
131. Otto, A. R., Gershman, S. J., Markman, A. B., & Daw, N. D. (in press). The curse of planning: Dissecting multiple reinforcement learning systems by taxing the central executive. Psychological Science.
132. Owen, A. M. (1997). Cognitive planning in humans: Neuropsychological, neuroanatomical and neuropharmacological perspectives. Progress in Neurobiology, 53, 431-450.
133. Owen, A. M., Sahakian, B. J., Hodges, J. R., Summers, B. A., Polkey, C. E., & Robbins, T. W. (1995). Dopamine-dependent fronto-striatalplanning deficits in early Parkinson's disease. Neuropsychology, 9, 126-140.
134. Packard, M. G., & Knowlton, B. J. (2002). Learning and memory functions of the basal ganglia. Annual Review of Neuroscience, 25, 563-593.
135. Pagnoni, G., Zink, C. F., Montague, P. R., & Berns, G. S. (2002). Activity in human ventral striatum locked to errors of reward prediction. Nature Neuroscience, 5, 97-98.
136. Pan, W. X., Schmidt, R., Wickens, J. R., & Hyland, B. I. (2005). Dopamine cells respond to predicted events during classical conditioning: Evidence for eligibility traces in the reward learning network. The Journal of Neuroscience, 25, 6235-6242.
137. Pavlov, I. P. (1927). Conditioned reflexes (G. V. Anrep, Trans.). London, England: Oxford University Press.
138. Plassmann, H., O'doherty, J., & Rangel, A. (2007). Orbitofrontal cortex encodes willingness to pay in everyday economic transactions. The Journal of Neuroscience, 27, 9984-9988.
139. Pritchard, W. S., Shappell, S. A., & Brandt, M. E. (1991). Psychophysiology of N200/N400: A review and classification scheme. In J. R. Jennings, P. K. Ackles, & M. G. H. Coles (Eds.), Advances in psychophysiology (Vol. 4, pp. 43-106). London, England: Jessica Kingsley.
140. Rachlin, H. (1995). Self-control: Beyond commitment. Behavioral and Brain Sciences, 18, 109-159.
141. Rangel, A., Camerer, C., & Montague, P. R. (2008). A framework for studying the neurobiology of value-based decision making. Nature Reviews Neuroscience, 9, 545-556.
142. Rao, R. P. N. (2010). Decision making under uncertainty: A neural model based on partially observable Markov decision processes. Frontiers in Computational Neuroscience, 4, 146.
143. Redgrave, P., Rodriguez, M., Smith, Y., Rodriguez-Oroz, M. C., Lehericy, S., Bergman, H.,... Obeso, J. A. (2010). Goal-directed and habitual control in the basal ganglia: Implications for Parkinson's disease. Nature Reviews Neuroscience, 11, 760-772.
144. Redish, A. D., Jensen, S., & Johnson, A. (2008). A unified framework for addiction: Vulnerabilities in the decision process. Behavioral and Brain Sciences, 31, 415-437.
145. Redish, A. D., Jensen, S., Johnson, A., & Kurth-Nelson, Z. (2007). Reconciling reinforcement learning models with behavioral extinction and renewal: Implications for addiction, relapse, and problem gambling. Psychological Review, 114, 784-805.
146. Rescorla, R. A., & Wagner, A. R. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In A. H. Black & W. F. Prokasy (Eds.), Classical conditioning II: Current research and theory (pp. 64-99). New York, NY: Appleton-Century-Crofts.
147. Reynolds, B. (2006). A review of delay-discounting research with humans: Relations to drug use and gambling. Behavioural Pharmacology, 17, 651-667.
148. Reynolds, J. N. J., Hyland, B. I., & Wickens, J. R. (2001). A cellular mechanism of reward-related learning. Nature, 413, 67-70.
149. Reynolds, J. N. J., & Wickens, J. R. (2002). Dopamine-dependent plasticity of corticostriatal synapses. Neural Networks, 15, 507-521
150. Ribas-Fernandes, J. J. F., Solway, A., Diuk, C., McGuire, J. T., Barto, A. G., Niv, Y., & Botvinick, M. M. (2011). A neural signature of hierarchical reinforcement learning. Neuron, 71, 370-379.
151. Rizley, R. C., & Rescorla, R. A. (1972). Associations in second-order conditioning and sensory preconditioning. Journal of Comparative and Physiological Psychology, 81, 1-11.
152. Roesch, M. R., Calu, D. J., & Schoenbaum, G. (2007). Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nature Neuroscience, 10, 1615-1624.
153. Rolls, E. T., Kringelbach, M. L., & de Araujo, I. E. T. (2003). Different representations of pleasant and unpleasant odours in the human brain. European Journal of Neuroscience, 18, 695-703.
154. Rutledge, R. B., Dean, M., Caplin, A., & Glimcher, P. W. (2010). Testing the reward prediction error hypothesis with an axiomatic model. The Journal of Neuroscience, 30, 13525-13536.
155. Saito, N., Mushiake, H., Sakamoto, K., Itoyama, Y., & Tanji, J. (2005). Representation of immediate and final behavioral goals in the monkey prefrontal cortex during an instructed delay period. Cerebral Cortex, 15, 1535-1546.
156. Samuel, A. L. (1995). Some studies in machine learning using the game of checkers. In E. A. Feigenbaum & J. Feldman (Eds.), Computers and thought (pp. 71-105). New York, NY: McGraw-Hill. (Reprinted from 1959, IBM Journal of Research and Development, 3, pp. 211-229)
157. Satoh, T., Nakai, S., Sato, T., & Kimura, M. (2003). Correlated coding of motivation and outcome of decision by dopamine neurons. The Journal of Neuroscience, 23, 9913-9923.
158. Schacter, D. L., Addis, D. R., & Buckner, R. L. (2007). Remembering the past to imagine the future: The prospective brain. Nature Reviews Neuroscience, 8, 657-661.
159. Schultz, W. (1998). Predictive reward signal of dopamine neurons. Journal of Neurophysiology, 80, 1-27.
160. Schultz, W., Apicella, P., & Ljungberg, T. (1993). Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. The Journal of Neuroscience, 13, 900-913.
161. Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275, 1593-1599.
162. Schweighofer, N., Bertin, M., Shishida, K., Okamoto, Y., Tanaka, S. C., Yamawaki, S., & Doya, K. (2008). Low-serotonin levels increase delayed reward discounting in humans. The Journal of Neuroscience, 28, 4528-4532.
163. Shallice, T. (1982). Specific impairments of planning. Philosophical Transactions of the Royal Society of London Series B, Biological Sciences, 298, 199-209.
164. Simon, D. A., & Daw, N. D. (2011). Neural correlates of forward planning in a spatial decision task in humans. The Journal of Neuroscience, 31, 5526-5539.
165. Sims, C. R., Neth, H., Jacobs, R. A., & Gray, W. D. (2013). Melioration as rational choice: Sequential decision making in uncertain environments. Psychological Review, 120, 139-154.
166. Singh, S. P., & Sutton, R. S. (1996). Reinforcement learning with replacing eligibility traces. Machine Learning, 22, 123-158.
167. Skinner, B. F. (1938). The behavior of organisms: An experimental analysis. Oxford, England: Appleton-Century.
168. Smith, A., Li, M., Becker, S., & Kapur, S. (2006). Dopamine, prediction error and associative learning: A model-based account. Network: Computation in Neural Systems, 17, 61- 84.
169. Solway, A., & Botvinick, M. M. (2012). Goal-directed decision making as probabilistic inference: A computational framework and potential neural correlates. Psychological Review, 119, 120-154.
170. Spence, K. W. (1932). The order of eliminating blinds in maze learning by the rat. Journal of Comparative Psychology, 14, 9-27.
171. Spragg, S. D. S. (1934). Anticipatory responses in the maze. Journal of Comparative Psychology, 18, 51-73.
172. Stankiewicz, B. J., Legge, G. E., Mansfield, J. S., & Schlicht, E. J. (2006). Lost is virtual space: Studies in human and ideal spatial navigation. Journal of Experimental Psychology: Human Perception and Performance, 32, 688-704.
173. Stocco, A., Lebiere, C., & Anderson, J. R. (2010). Conditional routing of information to the cortex: A model of the basal ganglia's role in cognitive coordination. Psychological Review, 117, 541-574.
174. Stoodley, C. J. (2012). The cerebellum and cognition: Evidence from functional imaging studies. The Cerebellum, 11, 352-365.
175. Strick, P. L., Dum, R. P., & Fiez, J. A. (2009). Cerebellum and nonmotor function. Annual Review of Neuroscience, 32, 413-434.
176. Sutton, R. S. (1990). Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In B. W. Porter & R. J. Mooney (Eds.), Proceedings of the seventh international conference on machine learning (pp. 216-224). San Francisco, CA: Morgan Kaufmann.
177. Sutton, R. S., & Barto, A. G. (1990). Time-derivative models of Pavlovian reinforcement. In M. Gabriel & J. Moore (Eds.), Learning and computational neuroscience: Foundations of adaptive networks (pp. 497-537). Cambridge, MA: MIT Press.
178. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
179. Taatgen, N. A., Huss, D., Dickison, D., & Anderson, J. R. (2008). The acquisition of robust and flexible cognitive skills. Journal of Experimental Psychology: General, 137, 548-565.
180. Tanaka, S. C., Balleine, B. W., & O'doherty, J. P. (2008). Calculating consequences: Brain systems that encode the causal effects of actions. The Journal of Neuroscience, 28, 6750-6755.
181. Tanaka, S. C., Shishida, K., Schweighofer, N., Okamoto, Y., Yamawaki, S., & Doya, K. (2009). Serotonin affects association of aversive outcomes to past actions. The Journal of Neuroscience, 29, 15669-15674.
182. Thistlethwaite, D. (1951). A critical review of latent learning and related experiments. Psychological Bulletin, 48, 97-129.
183. Thrun, S. B. (1992). The role of exploration in learning control. In D. A. White & D. A. Sofge (Eds.), Handbook of intelligent control: Neural, fuzzy and adaptive approaches (pp. 527-554). Florence, KY: Van Nostrand Reinhold.
184. Tobler, P. N., Fiorillo, C. D., & Schultz, W. (2005). Adaptive coding of reward value by dopamine neurons. Science, 307, 1642-1645.
185. Tobler, P. N., O'doherty, J. P., Dolan, R. J., & Schultz, W. (2005). Human neural learning depends on reward prediction errors in the blocking paradigm. Journal of Neurophysiology, 95, 301-310.
186. Todd, M. T., Niv, Y., & Cohen, J. D. (2009). Learning to use working memory in partially observable environments through dopaminergic reinforcement. In D. Koller, D. Schuurmans, Y. Bengio, & L. Bottou (Eds.), Advances in neural information processing systems (pp. 1689- 1696). Cambridge, MA: MIT Press.
187. Tolman, E. C. (1932). Purposive behavior in animals and men. New York, NY: Century.
188. Tolman, E. C., & Honzik, C. H. (1930). Degrees of hunger, reward and non-reward, and maze learning in rats. University of California Publications in Psychology, 4, 241-256.
189. Tricomi, E., Balleine, B. W., & O'doherty, J. P. (2009). A specific role for posterior dorsolateral striatum in human habit learning. European Journal of Neuroscience, 29, 2225-2232.
190. Tricomi, E. M., Delgado, M. R., & Fiez, J. A. (2004). Modulation of caudate activity by action contingency. Neuron, 41, 281-292.
191. Tunney, R. J., & Shanks, D. R. (2002). A re-examination of melioration and rational choice. Journal of Behavioral Decision Making, 15, 291- 311.
192. Valentin, V. V., Dickinson, A., & O'doherty, J. P. (2007). Determining the neural substrates of goal-directed learning in the human brain. The Journal of Neuroscience, 27, 4019-4026
193. van der Meer, M. A. A., & Redish, A. D. (2009). Covert expectation-ofreward in rat ventral striatum at decision points. Frontiers in Integrative Neuroscience, 3, 1-15.
194. van Veen, V., & Carter, C. S. (2002). The timing of action-monitoring processes in the anterior cingulate cortex. Journal of Cognitive Neuroscience, 14, 593-602.
195. Waelti, P., Dickinson, A., & Schultz, W. (2001). Dopamine responses comply with basic assumptions of formal learning theory. Nature, 412, 43-48.
196. Walsh, M. M., & Anderson, J. R. (2009). The strategic nature of changing your mind. Cognitive Psychology, 58, 416-440.
197. Walsh, M. M., & Anderson, J. R. (2011a). Learning from delayed feedback: Neural responses in temporal credit assignment. Cognitive, Affective & Behavioral Neuroscience, 11, 131-143.
198. Walsh, M. M., & Anderson, J. R. (2011b). Modulation of the feedbackrelated negativity by instruction and experience. PNAS Proceedings of the National Academy of Sciences of the United States of America, 108, 19048-19053.
199. Walsh, M. M., & Anderson, J. R. (2012). Learning from experience: Event-related potential correlates of reward processing, neural adaptation, and behavioral choice. Neuroscience and Biobehavioral Reviews, 36, 1870-1884.
200. Wang, Y., & Laird, J. E. (2007). The importance of action history in decision making and reinforcement learning. In R. L. Lewis, T. A. Polk, & J. E. Laird (Eds.), Proceedings of the eighth international conference on cognitive modeling (pp. 85-90). Ann Arbor, MI.
201. Warry, C. J., Remington, B., & Sonuga-Barke, E. J. S. (1999). When more means less: Factors affecting human self-control in a local versus global choice paradigm. Learning and Motivation, 30, 53-73.
202. Wassum, K. M., Ostlund, S. B., & Maidment, N. T. (2012). Phasic mesolimbic dopamine signaling precedes and predicts performance of a self-initiated action sequence task. Biological Psychiatry, 71, 846-854.
203. White, I. M., & Wise, S. P. (1999). Rule-dependent neuronal activity in the prefrontal cortex. Experimental Brain Research, 126, 315-335.
204. Wickens, J. R., Begg, A. J., & Arbuthnott, G. W. (1996). Dopamine reverses the depression of rat corticostriatal synapses which normally follows high-frequency stimulation of cortex in vitro. Neuroscience, 70, 1-5.
205. Wunderlich, K., Dayan, P., & Dolan, R. J. (2012). Mapping value based planning and extensively trained choice in the human brain. Nature Neuroscience, 15, 786-791.
206. Yarkoni, T., Braver, T. S., Gray, J. R., & Green, L. (2005). Prefrontal brain activity predicts temporally extended decision-making behavior. Journal of the Experimental Analysis of Behavior, 84, 537-554.
207. Yechiam, E., Erev, I., Yehene, V., & Gopher, D. (2003). Melioration and the transition from touch-typing training to everyday use. Human Factors, 45, 671-684.
208. Yeung, N., Botvinick, M. M., & Cohen, J. D. (2004). The neural basis of error detection: Conflict monitoring and the error-related negativity. Psychological Review, 111, 931-959.
209. Yin, H. H., Knowlton, B. J., & Balleine, B. W. (2004). Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. European Journal of Neuroscience, 19, 181-189.
210. Yoshida, W., & Ishii, S. (2006). Resolution of uncertainty in prefrontal cortex. Neuron, 50, 781-789.