Short-term gains, long-term pains: How cues about state aid learning in dynamic environments

Cognition

Volumn 113, Issue 3, 2009, Pages 293-313

  Gureckis, Todd M ^a   Love, Bradley C ^b  

^a NEW YORK UNIVERSITY   (United States)

^b UNIVERSITY OF TEXAS AT AUSTIN   (United States)

Author keywords

Decision making; Dynamic control task; Learning; Q learning; Reinforcement learning; Self control; State; Temporal difference; Temporal discounting

Indexed keywords

ARTICLE; ASSOCIATION; DECISION MAKING; ENVIRONMENT; HUMAN; LEARNING; MOTIVATION; NORMAL HUMAN; PRIORITY JOURNAL; REINFORCEMENT; REWARD; TASK PERFORMANCE;

ANALYSIS OF VARIANCE; CHOICE BEHAVIOR; CONFLICT (PSYCHOLOGY); CUES; ENVIRONMENT; HUMANS; MODELS, PSYCHOLOGICAL; NEUROPSYCHOLOGICAL TESTS; PSYCHOLOGICAL THEORY; PSYCHOMOTOR PERFORMANCE; REINFORCEMENT (PSYCHOLOGY); TIME FACTORS; YOUNG ADULT;

EID: 70350572378     PISSN: 00100277     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.cognition.2009.03.013     Document Type: Article

References (50)

1. Akaike H. A new look at the statistical model identification. IEEE Transaction on Automatic Control 19 6 (1974) 716-723
2. Axelrod R. The evolution of cooperation (1984), Basic Books, New York
3. Bagnell J., and Schneider J. Autonomous helicopter control using reinforcement learning policy search methods. International conference on robotics and automation (2001), IEEE 1615-1620
4. Bechara A., and Damasio H. Decision-making and addition (part I): Impaired activation of somatic states in substance dependent individuals when pondering decisions with negative future consequences. Neuropsychologia 40 10 (2002) 1675-1689
5. Bechara A., Dolan S., Denburg N., Hindes A., Anderson S., and Nathan P. Decision-making deficits, linked to a dysfunctional ventromedial prefrontal cortex, revealed in alcohol and stimulant abusers. Neuropsychologia 39 (2001) 376-389
6. Berry D.C., and Broadbent D.E. Interactive tasks and the implicit-explicit distinction. British Journal of Psychology 79 (1988) 251-272
7. Bogacz R., McClure S., Li J., Cohen J., and Montague P. Short-term memory traces for action bias in human reinforcement learning. Brain Research 1153 (2007) 111-121
8. Busemeyer J., and Stout J. A contribution of cognitive decision models to clinical assessment: Decomposing performance on the bechara gambling task. Psychological Assessment 14 3 (2002) 253-262
9. Carpenter G.A., and Grossberg S. The art of adaptive pattern recognition by a self-organizing neural network. Computer 21 3 (1988) 77-88
10. Chapman, D., & Kaelbling, L. (1991). Input generalization in delayed reinforcement learning: An algorithm and performance comparisons. In Proceedings of IJCAI.
11. Chhabra M., and Jacobs R. Near-optimal human adaptive control across different noise environments. The Journal of Neuroscience 26 42 (2006) 10883-10887
12. Daw N., Niv Y., and Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience 8 12 (2005) 1704-1711
13. Daw N., O'Doherty J., Seymour B., Dayan P., and Dolan R. Cortical substrates for exploratory decision in humans. Nature 441 (2006) 876-879
14. Daw N., and Touretzky D. Long-term reward prediction in td models of the dopamine system. Neural Computation 14 (2002) 603-616
15. Dehaene S., Bossini S., and Giraux P. The mental representation of parity and numerical magnitude. Journal of Experimental Psychology: General 122 (1993) 371-396
16. Egelman D., Person C., and Montague P. A computational role for dopamine delivery in human decision making. Journal of Cognitive Neuroscience 10 (1998) 623-630
17. Fu W., and Anderson J. From recurrent choice to skill learning: A reinforcement-learning model. Journal of Experimental Psychology: General 135 2 (2006) 184-206
18. Grant S., Controreggi C., and London E. Drug abusers show impaired performance in a laboratory test of decision making. Neuropsychologia 38 (2000) 1180-1187
19. Herrnstein R. Experiments on stable suboptimality in individual behavior. The American Economic Review 81 2 (1991) 360-364
20. Herrnstein R., and Prelec D. Melioration: A theory of distributed choice. The Journal of Economic Perspectives 5 3 (1991) 137-156
21. Herrnstein R., Loewenstein G.F., Prelec D., and Vaughan W. Utility maximization and melioration: Internalities in individual choice. Journal of Behavioral Decision Making 6 (1993) 149-185
22. Littman, M., Sutton, R., & Sigh, S. (2002). Predictive representations of state. In Advances in neural information processing systems (Vol. 14, pp. 1555-1561).
23. Love B., Medin D., and Gureckis T. Sustain: A network model of category learning. Psychological Review 111 2 (2004) 309-332
24. Luce R.D. Individual choice behavior: A theoretical analysis (1959), Greenwood Press, Westport, CT
25. McCallum, R. (1993). Overcoming incomplete perception with utile distinction memory. In The proceedings of the tenth international machine learning conference (ML'93). Amherst, MA.
26. McCallum, A. (1995). Reinforcement learning with selective perception and hidden state. Unpublished doctoral dissertation. University of Rochester.
27. Montague P., and Berns G. Neural economics and the biological substrates of valuation. Neuron 36 (2002) 265-284
28. Montague P., Dayan P., Person C., and Sejnowski T. Bee foraging in uncertain environments using predictive hebbian learning. Nature 377 6551 (1995) 725-728
29. Montague P., Dayan P., and Sejnowski T. A framework for mesencephalic dopamine system based on predictive hebbian learning. Journal of Neuroscience 16 5 (1996) 1936-1947
30. Nelder J., and Mead R. A simple method for function minimization. Computer Journal 7 (1965) 308-313
31. Neth H., Sims C., and Gray W. Melioration dominates maximization: Stable suboptimal performance despite global feedback. In: Sun R., and Miyake N. (Eds). Proceedings of the 28th annual meeting of the cognitive science society (2006), Lawrence Erlbaum Associates, Hillsdale, NJ
32. Redish A., Jensen S., Johnson A., and Kurth-Nelson Z. Reconciling reinforcement learning models with behavioral extinction and renewal: Implications for addition, relapse, and problem gamling. Psychological Review 114 3 (2007) 784-805
33. Rescorla R., and Wagner A. A theory of pavolvian conditioning: Variations in the effectiveness of reinforcement and non-reinforcement. In: Black A., and Prokasy W. (Eds). Classical conditioning II: Current research and theory (1972), Appleton-Century-Crofts, New York 64-99
34. Sanabria F., Baker F., and Rachlin H. Learning by pigeons playing against tit-for-tat in an operant prisoner's dilemma. Learning and Behavior 31 4 (2003) 318-331
35. Schultz W., Dayan P., and Montague P.R. A neural substrate of prediction and reward. Science 275 (1997) 1593-1598
36. Schwartz G. Estimating the dimension of a model. The Annals of Statistics 5 (1978) 461-464
37. Stanley W., Mathew R., Russ R., and Kotler-Cope S. Insight without awareness: On the interaction of verbalization, instruction, and practice in a simulated process control task. Quarterly Journal of Experimental Psychology 41A 3 (1989) 553-577
38. Sun R., Slusarz P., and Terry C. The interaction of the explicit and the implicit in skill learning: A dual-process approach. Psychological Review 112 1 (2005) 159-192
39. Suri R., Bargas J., and Arbib M. Modeling functions of striatal dopamine modulation in learning and planning. Neuroscience 103 (2001) 65-85
40. Sutton R. Generalization in reinforcement learning: Successful examples using sparse coarse coding. In: Touretzky D., Mozer M., and Hasselmo M. (Eds). Advances in neural information processing systems: Proceedings of the 1995 conference (1996), MIT Press, Cambridge, MA 1038-1044
41. Sutton R., and Barto A. Reinforcement learning: An introduction (1998), MIT Press, Cambridge, MA
42. Tesauro G. Td-gammon, a self-teaching backgammon program, achieves master-level play. Neural Computation 6 2 (1994) 215-219
43. Triesch J., Ballard D.H., Hayhoe M.M., and Sullivan B.T. What you see is what you need. Journal of Vision 3 1 (2003) 86-94
44. Tunney R.J., and Shanks D.R. A re-examination of melioration and rational choice. Journal of Behavioral Decision Making 15 (2002) 291-311
45. Wagner A., and Rescorla R. Inhibition in pavlovian conditioning: Application of a theory. In: Boake R., and Halliday M. (Eds). Inhibition and learning (1972), Academic Press, London 301-336
46. Watkins, C. (1989). Learning from delayed rewards. Unpublished doctoral dissertation. Cambridge, England: Cambridge University.
47. Whitehead S., and Ballard D. Learning to perceive and act by trial and error. Machine Learning 7 1 (1991) 45-83
48. Widrow B., and Hoff M. Adaptive switching circuits. Institute of Radio Engineers, Western Electronic Show and Convention Record 4 (1960) 96-104
49. Worthy D., Maddox W., and Markman A. Regulatory fit effects in a choice task. Psychonomic Bulletin and Review (2007)
50. Yechiam E., and Busemeyer J. Comparison of basic assumptions embedded in learning models for experience based decision-making. Psychonomic Bulletin and Review 12 3 (2005) 387-402