Retrospective revaluation in sequential decision making: A tale of two systems

Journal of Experimental Psychology: General

Volumn 143, Issue 1, 2014, Pages 182-194

  Gershman, Samuel J ^a   Markman, Arthur B ^b   Otto, A Ross ^b,c  

^a PRINCETON UNIVERSITY   (United States)

^b UNIVERSITY OF TEXAS AT AUSTIN   (United States)

^c NEW YORK UNIVERSITY   (United States)

Author keywords

Decision making; DYNA; Reinforcement learning; Retrospective revaluation

Indexed keywords

ARTICLE; CONTROLLED CLINICAL TRIAL; CONTROLLED STUDY; DECISION MAKING; HUMAN; NEUROPSYCHOLOGICAL TEST; PSYCHOLOGICAL MODEL; RANDOMIZED CONTROLLED TRIAL; REINFORCEMENT;

DECISION MAKING; HUMANS; MODELS, PSYCHOLOGICAL; NEUROPSYCHOLOGICAL TESTS; REINFORCEMENT (PSYCHOLOGY);

EID: 84893351295     PISSN: 00963445     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0030844     Document Type: Article

References (59)

1. Adams, C. D. (1982). Variations in the sensitivity of instrumental responding to reinforcer devaluation. Quarterly Journal of Experimental Psychology: Section B: Comparative and Physiological Psychology, 34, 77-98.
2. Baddeley, A. (1992). Working memory. Science, 255, 556-559. doi: 10.1126/science.1736359
3. Balleine, B. W., & O'Dohety, J. P. (2010). Human and rodent homologies in action control: Corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology, 35, 48-69. doi:10.1038/npp.2009.131
4. Bellman, R. E. (1957). Dynamic programming. Princeton, NJ: Princeton University Press.
5. Bornstein, A. M., & Daw, N. D. (2012). Dissociating hippocampal and striatal contributions to sequential prediction learning. European Journal of Neuroscience, 35, 1011-1023. doi:10.1111/j.1460-9568.2011.07920.x
6. Buckner, R. L. (2010). The role of the hippocampus in prediction and imagination. Annual Review of Psychology, 61, 27-48. doi:10.1146/annurev.psych.60.110707.163508
7. Chapman, G. B. (1991). Trial order affects cue interaction in contingency judgment. Journal of Experimental Psychology: Learning, Memory, and Cognition, 17, 837-854. doi:10.1037/0278-7393.17.5.837
8. Conway, A. R., Kane, M. J., & Engle, R. W. (2003). Working memory capacity and its relation to general intelligence. Trends in Cognitive Sciences, 7, 547-552.
9. Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011). Model-based influences on humans' choices and striatal prediction errors. Neuron, 69, 1204-1215. doi:10.1016/j.neuron.2011.02.027
10. Daw, N. D., Niv, Y., & Dayan, P. (2005). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience, 8, 1704-1711. doi:10.1038/nn1560
11. Dayan, P. (2009). Goal-directed control and its antipodes. Neural Networks, 22, 213-219. doi:10.1016/j.neunet.2009.03.004
12. Diba, K., & Buzsáki, G. (2007). Forward and reverse hippocampal placecell sequences during ripples. Nature Neuroscience, 10, 1241-1242. doi:10.1038/nn1961
13. Dickinson, A. (1985). Actions and habits: The development of behavioural autonomy. Philosophical Transactions of the Royal Society: B, 308, 67-78. doi:10.1098/rstb.1985.0010
14. Dickinson, A., & Balleine, B. (2002). The role of learning in the operation of motivational systems. In H. Pashler & R. Gallistel (Eds.), Stevens' handbook of experimental psychology, third edition, Vol. 3: Learning, motivation, and emotion ( pp. 497-534). New York, NY: Wiley.
15. Dickinson, A., & Burke, J. (1996). Within-compound associations mediate the retrospective revaluation of causality judgements. Quarterly Journal of Experimental Psychology: Section B: Comparative and Physiological Psychology, 37, 60-80.
16. Dragoi, G., & Tonegawa, S. (2011). Preplay of place cell sequences by hippocampal cellular assemblies. Nature, 469, 397-401. doi:10.1038/nature09633
17. Engle, R. W. (2002). Working memory capacity as executive attention. Current Directions in Psychological Science, 11, 19-23.
18. Foerde, K., Knowlton, B. J., & Poldrack, R. A. (2006). Modulation of competing memory systems by distraction. Proceedings of the National Academy of Sciences, 103, 11778-11783.
19. Foster, D. J., & Wilson, M. A. (2006). Reverse replay of behavioural sequences in hippocampal place cells during the awake state. Nature, 440, 680-683.
20. Fu, W.-T., & Anderson, J. R. (2008). Dual learning processes in interactive skill acquisition. Journal of Experimental Psychology: Applied, 14, 179-191. doi:10.1037/1076-898X.14.2.179
21. Fuhs, M. C., & Touretzky, D. S. (2007). Context learning in the rodent hippocampus. Neural Computation, 19, 3173-3215. doi:10.1162/neco.2007.19.12.3173
22. Gershman, S. J., Blei, D. M., & Niv, Y. (2010). Context, learning and extinction. Psychological Review, 117, 197-209. doi:10.1037/a0017808
23. Gluck, M. A., & Myers, C. E. (1993). Hippocampal mediation of stimulus representation: A computational theory. Hippocampus, 3, 491-516. doi: 10.1002/hipo.450030410
24. Gupta, A. S., van der Meer, M. A. A., Touretzky, D. S., & Redish, A. D. (2010). Hippocampal replay is not a simple function of experience. Neuron, 65, 695-705. doi:10.1016/j.neuron.2010.01.034
25. Harrison, L. M., Duggins, A., & Friston, K. J. (2006). Encoding uncertainty in the hippocampus. Neural Networks, 19, 535-546. doi:10.1016/j.neunet.2005.11.002
26. Hölscher, C., Jacob, W., & Mallot, H. A. (2003). Reward modulates neuronal activity in the hippocampus of the rat. Behavioural Brain Research, 142, 181-191. doi:10.1016/S0166-4328(02)00422-9
27. Jeffreys, H. (1961). Theory of probability (3rd ed.). Oxford, England: Oxford University Press.
28. Johnson, A., & Redish, A. D. (2005). Hippocampal replay contributes to within session learning in a temporal difference reinforcement learning model. Neural Networks, 18, 1163-1171. doi:10.1016/j.neunet.2005.08.009
29. Johnson, A., & Redish, A. D. (2007). Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. Journal of Neuroscience, 27, 12176 -12189. doi:10.1523/JNEUROSCI.3761-07.2007
30. Johnson, A., van der Meer, M. A. A., & Redish, A. D. (2007). Integrating hippocampus and striatum in decision making. Current Opinion in Neurobiology, 17, 692-697. doi:10.1016/j.conb.2008.01.003
31. Keramati, M., Dezfouli, A., & Piray, P. (2011). Speed/accuracy trade-off between the habitual and the goal-directed processes. PLoS Computational Biology, 7, e1002055. doi:10.1371/journal.pcbi.1002055
32. Lansink, C. S., Goltstein, P. M., Lankelma, J. V., McNaughton, B. L., & Pennartz, C. M. A. (2009). Hippocampus leads ventral striatum in replay of place-reward information. PLoS Biology, 7, e1000173. doi:10.1371/journal.pbio.1000173
33. Larkin, M., Aitken, M., & Dickinson, A. (1998). Retrospective revaluation of causal judgments under positive and negative contingencies. Journal of Experimental Psychology: Learning, Memory, and Cognition, 24, 1331-1352. doi:10.1037/0278-7393.24.6.1331
34. Levy, W. B., Hocking, A. B., & Wu, X. (2005). Interpreting hippocampal function as recoding and forecasting. Neural Networks, 18, 1242-1264. doi:10.1016/j.neunet.2005.08.005
35. Markman, A. B. (1989). LMS rules and the inverse base-rate effect: Comment on Gluck and Bower (1988). Journal of Experimental Psychology: General, 118, 417-421. doi:10.1037/0096-3445.118.4.417
36. McDonald, R. J., & White, N. M. (1993). A triple dissociation of memory systems: Hippocampus, amygdala, and dorsal striatum. Behavioral Neuroscience, 107, 3-22. doi:10.1037/0735-7044.107.1.3
37. Melchers, K. G., Lachnit, H., & Shanks, D. R. (2004). Within-compound associations in retrospective revaluation and in direct learning: A challenge for comparator theory. The Quarterly Journal of Experimental Psychology: Section B: Comparative and Physiological Psychololgy, 57, 25-53. doi:10.1080/02724990344000042
38. Niv, Y. (2009). Reinforcement learning in the brain. The Journal of Mathematical Psychology, 53, 139-154. doi:10.1016/j.jmp.2008.12.005
39. Otto, A. R., Gershman, S. G., Markman, A. B., & Daw, N. D. (2013). The curse of planning: Dissecting multiple reinforcement learning systems by taxing the central executive. Psychological Science, 24, 751-761.
40. Otto, A. R., Taylor, E. G., & Markman, A. B. (2011). There are at least two kinds of probability matching: Evidence from a secondary task. Cognition, 118, 274-279. doi:10.1016/j.cognition.2010.11.009
41. Prokofiev, S. (2008). Gavotte Op. 12 No. 2 [Recorded by Sergei Prokofiev]. On Prokofiev: The Pianist [CD]. London, England: Future Noise Music.
42. Rescorla, R. A., & Wagner, A. R. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In A. Black & W. Prokasy (Eds.), Classical conditioning II: Current research and theory (pp. 64-99). New York, NY: Appleton-Century-Crofts.
43. Rouder, J. N., Speckman, P. L., Dongchu, S., & Morey, R. D. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16, 225-237. doi:10.3758/PBR.16.2.225
44. Shanks, D. (1985). Forward and backward blocking in human contingency judgement. Quarterly Journal of Experimental Psychology: Section B: Comparative and Physiological Psychology, 37, 1-21.
45. Shinners, P. (2011). PyGame-Python game development. Available from http://www.pygame.org
46. Shohamy, D., & Wagner, A. D. (2008). Integrating memories in the human brain: Hippocampal-midbrain encoding of overlapping events. Neuron, 60, 378-389. doi:10.1016/j.neuron.2008.09.023
47. Simon, D. A., & Daw, N. D. (2011). Environmental statistics and the trade-off between model-based and TD learning in humans. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, & K. Weinberger (Eds.), Advances in neural information processing systems (Vol. 24, pp. 127-135). Retrieved from http://books.nips.cc/nips24.html
48. Strange, B. A., Duggins, A., Penny, W., Dolan, R. J., & Friston, K. J. (2008). Information theory, novelty and hippocampal responses: Unpredicted or unpredictable? Neural Networks, 18, 225-230. doi:10.1016/j.neunet.2004.12.004
49. Sutton, R. S. (1990). Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In B. W. Porter & R. J. Mooney (Eds.), Proceedings of the Seventh International Conference on Machine Learning (pp. 216-224). San Francisco, CA: Morgan Kaufmann.
50. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
51. Thorndike, E. L. (1911). Animal Intelligence. New York, NY: The Macmillian Company.
52. Tolman, E. C. (1948). Cognitive maps in rats and men. Psychological Review, 55, 189-208. doi:10.1037/h0061626
53. Tricomi, E., Balleine, B. W., & O'Doherty, J. P. (2009). A specific role for posterior dorsolateral striatum in human habit learning. European Journal of Neuroscience, 29, 2225-2232. doi:10.1111/j.1460-9568.2009.06796.x
54. Turner, M. L., & Engle, R. W. (1989). Is working memory capacity task dependent? Journal of Memory and Language, 28, 127-154.
55. Unsworth, N., Heitz, R. P., Schrock, J. C., & Engle, R. W. (2005). An automated version of the operation span task. Behavior Research Methods, 37, 498-505.
56. Valentin, V. V., Dickinson, A., & O'Doherty, J. P. (2007). Determining the neural substrates of goal-directed learning in the human brain. Journal of Neuroscience, 27, 4019-4026. doi:10.1523/JNEUROSCI.0564-07.2007
57. Van Hamme, L. J., & Wasserman, E. A. (1994). Cue competition in causality judgments: The role of nonpresentation of compound stimulus elements. Learning and Motivation, 25, 127-151. doi:10.1006/lmot.1994.1008
58. Watkins, C. J. C. H. (1989). Learning from delayed rewards (Unpublished doctoral dissertation). Cambridge University, Cambridge, England.
59. Zeithamova, D., & Maddox, W. T. (2006). Dual-task interference in perceptual category learning. Memory & Cognition, 34, 387-398.