When, What, and How Much to Reward in Reinforcement Learning-Based Models of Cognition

Cognitive Science

Volumn 36, Issue 2, 2012, Pages 333-358

  Janssen, Christian P ^a,b,c   Gray, Wayne D ^b  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

^b RENSSELAER POLYTECHNIC INSTITUTE   (United States)

^c UNIVERSITY OF GRONINGEN   (Netherlands)

Author keywords

Adaptive behavior; Choice; Cognitive architecture; Expected utility; Expected value; Reinforcement learning; Skill acquisition and learning; Strategy selection

Indexed keywords

ADAPTIVE BEHAVIOR; ARTICLE; COGNITION; HUMAN; PSYCHOLOGICAL MODEL; REINFORCEMENT;

ADAPTATION, PSYCHOLOGICAL; COGNITION; HUMANS; MODELS, PSYCHOLOGICAL; REINFORCEMENT (PSYCHOLOGY); REINFORCEMENT SCHEDULE;

EID: 84857974114     PISSN: 03640213     EISSN: None     Source Type: Journal    
DOI: 10.1111/j.1551-6709.2011.01222.x     Document Type: Article

References (52)

1. Ahn, W.-Y., Busemeyer, J. R., Wagenmakers, E., & Stout, J. C. (2008). Comparison of decision learning models using the generalization criterion method. Cognitive Science, 32, 1376-1402.
2. Anderson, J. R. (1993). Rules of the mind. Hillsdale, NJ: Lawrence Erlbaum Associates.
3. Anderson, J. R. (2007). How can the human mind occur in the physical universe? New York: Oxford University Press.
4. Anderson, J. R., Bothell, D., Byrne, M. D., Douglass, S., Lebiere, C., & Qin, Y. (2004). An integrated theory of the mind. Psychological Review, 111, 1036-1060.
5. Anderson, J. R., & Lebiere, C. J. (1998). The atomic components of thought. Mahwah, NJ: Lawrence Erlbaum Associates.
6. Anderson, J. R., & Schooler, L. J. (1991). Reflections of the environment in memory. Psychological Science, 2, 396-408.
7. Ballard, D. H., Hayhoe, M. M., & Pelz, J. B. (1995). Memory representations in natural tasks. Journal of Cognitive Neuroscience, 7, 66-80.
8. Ballard, D. H., Hayhoe, M. M., Pook, P. K., & Rao, R. P. N. (1997). Deictic codes for the embodiment of cognition. Behavioral and Brain Sciences, 20, 723-742.
9. Ballard, D. H., & Sprague, N. (2007). On the role of embodiment in modeling natural behaviors. In W. D. Gray (Ed.), Integrated models of cognitive systems (pp. 283-296). New York: Oxford University Press.
10. Barto, A. G., Sutton, R., & Anderson, C. W. (1983). Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics, 13, 835-846.
11. Bothell, D. (Producer). (2008) ACT-R 6 reference manual. Accessed June 2008.
12. Botvinick, M., Niv, Y., & Barto, A. G. (2009). Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective. Cognition, 113, 262-280.
13. Cohen, M. X. (2008). Neurocomputational mechanisms of reinforcement-guided learning in humans: A review. Cognitive, Affective & Behavioral Neuroscience, 8, 113-125.
14. Davis, D. G., Staddon, J. E., Machado, A., & Palmer, R. G. (1993). The process of recurrent choice. Psychological Review, 100, 320-341.
15. Daw, N. D., & Frank, M. J. (2009). Reinforcement learning and higher level cognition: Introduction to special issue. Cognition, 113, 259-261.
16. Edwards, W. (1965). Optimal strategies for seeking information: Models for statistics, choice reaction times, and human information processing. Journal of Mathematical Psychology, 2, 312-329.
17. Erev, I., & Barron, G. (2005). On adaptation, maximization, and reinforcement learning among cognitive strategies. Psychological Review, 112, 912-931.
18. Fitts, P. M. (1954). The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology, 47, 381-391.
19. Fu, W. T., & Anderson, J. R. (2004). Extending the computational abilities of the procedural learning mechanism in ACT-R. In: K. D. Forbus, D. Gentner, & T. Regier (Eds.), Proceedings of the 26th Annual Meeting of the Cognitive Science Society (pp. 416-421). Austin, TX: Cognitive Science Society.
20. Fu, W. T., & Anderson, J. R. (2006). From recurrent choice to skill learning: A reinforcement-learning model. Journal of Experimental Psychology: General, 135, 184-206.
21. Fu, W. T., & Gray, W. D. (2006). Suboptimal tradeoffs in information seeking. Cognitive Psychology, 52, 195-242.
22. Gray, W. D., & Boehm-Davis, D. A. (2000). Milliseconds matter: An introduction to microstrategies and to their use in describing and predicting interactive behavior. Journal of Experimental Psychology: Applied, 6, 322-335.
23. Gray, W. D., & Fu, W. T. (2004). Soft constraints in interactive behavior: The case of ignoring perfect knowledge in-the-world for imperfect knowledge in-the-head. Cognitive Science, 28, 359-382.
24. Gray, W. D., Schoelles, M. J., & Sims, C. R. (2005). Adapting to the task environment: Explorations in expected value. Cognitive Systems Research, 6, 27-40.
25. Gray, W. D., Sims, C. R., Fu, W. T., & Schoelles, M. J. (2006). The soft constraints hypothesis: A rational analysis approach to resource allocation for interactive behavior. Psychological Review, 113, 461-482.
26. Gureckis, T. M., & Love, B. C. (2009). Short-term gains, long-term pains: How cues about state aid learning in dynamic environments. Cognition, 113, 293-313.
27. Herrnstein, R. J. (1990). Behavior, reinforcement and utility. Psychological Science, 1, 217-224.
28. Holroyd, C. B., & Coles, M. G. H. (2002). The neural basis of human error processing: Reinforcement learning, dopamine, and the error-related negativity. Psychological Review, 109, 679-709.
29. Howes, A., Lewis, R. L., & Vera, A. (2009). Rational adaptation under task and processing constraints: Implications for testing theories of cognition and action. Psychological Review, 116, 717-751.
30. Janssen, C. P., & Brumby, D. P. (2010). Strategic adaptation to performance objectives in a dual-task setting. Cognitive Science, 34, 1548-1560.
31. Janssen, C. P., Brumby, D. P., Dowell, J., Chater, N., & Howes, A. (2011). Identifying optimum performance trade-offs using a cognitively bounded rational analysis model of discretionary task interleaving. Topics in Cognitive Science, 3, 123-139.
32. Lovett, M. C. (1998). Choice. In J. R. Anderson & C. Lebiere (Eds.), The atomic components of thought (pp. 255-296). Mahwah, NJ: Erlbaum.
33. Montague, P. R., Dayan, P., & Sejnowski, T. J. (1996). A framework for mesencephalic dopamine systems based on predictive hebbian learning. Journal of Neuroscience, 16, 1936-1947.
34. Morgan, P. L., Patrick, J., Waldron, S. M., King, S. L., & Patrick, T. (2009). Improving memory after interruption: Exploiting soft constraints and manipulating information access cost. Journal of Experimental Psychology: Applied, 15, 291-306.
35. Napoli, A., & Fum, D. (2010). Rewards and punishments in iterated decision making: An explanation for the frequency of the contingent event effect. In D. D. Salvucci & G. Gunzelmann (Eds.), Proceedings of the 10th International Conference on Cognitive Modeling (pp. 175-180). Philadelphia, PA: Drexel University.
36. Nason, S., & Laird, J. E. (2005). SOAR-RL: Integrating reinforcement learning with SOAR. Cognitive Systems Research, 6, 51-59.
37. Neth, H., Khemlani, S. S., & Gray, W. D. (2008). Feedback design for the control of a dynamic multitasking system: Dissociating outcome feedback from control feedback. Human Factors, 50, 643-651.
38. Neth, H., Sims, C. R., & Gray, W. D. (2006). Melioration dominates maximization: Stable suboptimal performance despite global feedback. In R. Sun (Ed.), Proceedings of the 28th Annual Meeting of the Cognitive Science Society (pp. 627-632). Austin, TX: Cognitive Science Society.
39. Rieskamp, J., & Otto, P. E. (2006). SSL: A theory of how people learn to select strategies. Journal of Experimental Psychology: General, 135, 207-236.
40. Schultz, W. (2006). Behavioral theories and the neurophysiology of reward. Annual Review of Psychology, 57, 87-115.
41. Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275, 1593-1599.
42. Shanks, D. R., Tunney, R. J., & McCarthy, J. D. (2002). A re-examination of probability matching and rational choice. Journal of Behavioral Decision Making, 15, 233-250.
43. Sims, C. R., & Gray, W. D. (2004). Episodic versus semantic memory: An exploration of models of memory decay in the serial attention paradigm. In M. Lovett, C. Schunn, & C. Lebiere (Eds.), Proceedings of the 6th International Conference on Cognitive Modeling (pp. 279-284). Mahwah, NJ: Lawrence Erlbaum Associates.
44. Singh, S., Lewis, R., & Barto, A. G. (2009). Where do rewards come from? In N. Taatgen & H. Van Rijn (Eds.), Proceedings of the 31st Annual Meeting of the Cognitive Science Society (pp. 2601-2606). Austin, TX: Cognitive Science Society.
45. Soukoreff, R. W., & MacKenzie, I. S. (2004). Towards a standard for pointing device evaluation, perspectives on 27 years of fitts' law research in HCI. International Journal of Human-Computer Studies, 61, 751-789.
46. Sun, R., Merrill, E., & Peterson, T. (2001). From implicit skills to explicit knowledge: A bottom-up model of skill learning. Cognitive Science, 25, 203-244.
47. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
48. Waldron, S. M., Patrick, J., Morgan, P. L., & King, S. (2007). Influencing cognitive strategy by manipulating information access. The Computer Journal, 50, 694-702.
49. Walsh, M. M., & Anderson, J. R. (2009). The strategic nature of changing your mind. Cognitive Psychology, 58, 416-440.
50. Wang, D. D., Proctor, R. W., & Pick, D. F. (2007). Acquisition and transfer of attention allocation strategies in a multiple-task work environment. Human Factors, 49, 995-1004.
51. Watkins, C., & Dayan, P. (1992). Q-learning. Machine Learning, 8, 279-292.
52. Yechiam, E., & Busemeyer, J. R. (2005). Comparison of basic assumptions embedded in learning models for experience-based decision making. Psychonomic Bulletin & Review, 12, 387-402.