Novelty and Inductive Generalization in Human Reinforcement Learning

Topics in Cognitive Science

Volumn 7, Issue 3, 2015, Pages 391-415

  Gershman, Samuel J ^a   Niv, Yael ^b  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^b PRINCETON UNIVERSITY   (United States)

Author keywords

Bayesian inference; Exploration exploitation dilemma; Neophilia; Neophobia; Reinforcement learning

Indexed keywords

DOPAMINE;

ALGORITHM; ANIMAL; COGNITION; DECISION MAKING; ENVIRONMENT; EXPLORATORY BEHAVIOR; HUMAN; LEARNING; METABOLISM; PHOBIA; PHYSIOLOGY; PRISONER DILEMMA; PSYCHOLOGICAL MODEL; PSYCHOLOGY; REINFORCEMENT; REWARD;

ALGORITHMS; ANIMALS; COGNITION; DECISION MAKING; DOPAMINE; ENVIRONMENT; EXPLORATORY BEHAVIOR; HUMANS; LEARNING; MODELS, PSYCHOLOGICAL; PHOBIC DISORDERS; PRISONER DILEMMA; REINFORCEMENT (PSYCHOLOGY); REWARD;

EID: 84939246893     PISSN: 17568757     EISSN: 17568765     Source Type: Journal    
DOI: 10.1111/tops.12138     Document Type: Article

References (89)

1. Acuña, D., & Schrater, P. (2010). Structure learning in human sequential decision-making. PLoS Computational Biology, 6(12), 221-229.
2. Anderson, J. (1991). The adaptive nature of human categorization. Psychological Review, 98(3), 409-429.
3. Bardo, M., & Bevins, R. (2000). Conditioned place preference: What does it add to our preclinical understanding of drug reward? Psychopharmacology, 153(1), 31-43.
4. Barnett, S. (1958). Experiments on "neophobia' in wild and laboratory rats. British Journal of Psychology, 49(3), 195-201.
5. Barto, A. (1995). Adaptive critics and the basal ganglia. In J. Houk, J. Davis, & D. Beiser, (Eds.), Models of information processing in the basal ganglia (pp. 215-232). Cambridge, MA: MIT Press.
6. Bayer, H., & Glimcher, P. (2005). Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron, 47(1), 129-141.
7. Behrens, T., Woolrich, M., Walton, M., & Rushworth, M. (2007). Learning the value of information in an uncertain world. Nature Neuroscience, 10(9), 1214-1221.
8. Berlyne, D. (1960). Conflict, arousal, and curiosity. New York: McGraw-Hill.
9. Berlyne, D. (1966). Curiosity and exploration. Science, 153(3731), 25-33.
10. Berlyne, D., Koenig, I., & Hirota, T. (1966). Novelty, arousal, and the reinforcement of diversive exploration in the rat. Journal of Comparative and Physiological Psychology, 62(2), 222-226.
11. Bevins, R. (2001). Novelty seeking and reward: Implications for the study of high-risk behaviors. Current Directions in Psychological Science, 10(6), 189-193.
12. Blanchard, R., Kelley, M., & Blanchard, D. (1974). Defensive reactions and exploratory behavior in rats. Journal of Comparative and Physiological Psychology, 87(6), 1129-1133.
13. Botvinick, M., Niv, Y., & Barto, A. (2009). Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective. Cognition, 113(3), 262-280.
14. Bouton, M., & King, D. (1983). Contextual control of the extinction of conditioned fear: Tests for the associative value of the context. Journal of Experimental Psychology: Animal Behavior Processes, 9(3), 248-265.
15. Brafman, R., & Tennenholtz, M. (2003). R-max-a general polynomial time algorithm for nearoptimal reinforcement learning. The Journal of Machine Learning Research, 3(3), 213-231.
16. Brainard, D. (1997). The psychophysics toolbox. Spatial Vision, 10(4), 433-436.
17. Cohen, J., McClure, S., & Yu, A. (2007). Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration. Philosophical Transactions of the Royal Society B: Biological Sciences, 362(1481), 933-942.
18. Corey, D. (1978). The determinants of exploration and neophobia. Neuroscience & Biobehavioral Reviews, 2(4), 235-253.
19. Courville, A., Daw, N., & Touretzky, D. (2006). Bayesian theories of conditioning in a changing world. Trends in Cognitive Sciences, 10(7), 294-300.
20. Cowan, P. (1976). The new object reaction of Rattus rattus L.: The relative importance of various cues. Behavioral Biology, 16(1), 31-44.
21. Daw, N., Courville, A., & Touretzky, D. (2006a). Representation and timing in theories of the dopamine system. Neural Computation, 18(7), 1637-1677.
22. Daw, N., O'Doherty, J., Dayan, P., Seymour, B., & Dolan, R. (2006b). Cortical substrates for exploratory decisions in humans. Nature, 441(7095), 876-879.
23. Day, J., Roitman, M., Wightman, R., & Carelli, R. (2007). Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens. Nature Neuroscience, 10(8), 1020-1028.
24. Dayan, P., Niv, Y., Seymour, B., & Daw, N. D. (2006). The misbehavior of value and the discipline of the will. Neural Networks, 19(8), 1153-1160.
25. Dearden, R., Friedman, N., & Russell, S. (1998). Bayesian q-learning. In J. Mostow & C. Rich (Eds.), Proceedings of the National Conference on Artificial Intelligence (pp. 761-768). Madison, WI.
26. Domjan, M. (2003). The principles of learning and behavior. Stamford, CT: Thomson/Wadsworth.
27. Engel, Y., Mannor, S., & Meir, R. (2003). Bayes meets Bellman: The Gaussian process approach to temporal difference learning. In T. Fawcett & N. Mishra (Eds.), International Conference on Machine Learning (Vol. 20, pp. 154-162). Washington, DC.
28. Ennaceur, A., & Delacour, J. (1988). A new one-trial test for neurobiological studies of memory in rats. 1: Behavioral data. Behavioural Brain Research, 31(1), 47-59.
29. Fehrer, E. (1956). The effects of hunger and familiarity of locale on exploration. Journal of Comparative and Physiological Psychology, 49(6), 549-552.
30. File, S., & Day, S. (1972). Effects of time of day and food deprivation on exploratory activity in the rat. Animal Behaviour, 20(4), 758-762.
31. Friston, K. (2008). Hierarchical models in the brain. PLoS Computational Biology, 4(11), e1000211.
32. Gershman, S., Blei, D., & Niv, Y. (2010). Context, learning, and extinction. Psychological Review, 117(1), 197-209.
33. Gershman, S. & Niv, Y. (2010). Learning latent structure: Carving nature at its joints. Current Opinion in Neurobiology, 20(2), 251-256.
34. Gershman, S., Pesaran, B., & Daw, N. (2009). Human reinforcement learning subdivides structured action spaces by learning effector-specific values. Journal of Neuroscience, 29(43), 13524-13531.
35. Gittins, J. (1989). Multi-armed bandit allocation indices. Chichester, England: John Wiley & Sons Inc.
36. Gluck, M., (1991). Stimulus generalization and representation in adaptive network models of category learning. Psychological Science, 2(1), 50-55.
37. Gluck, M., & Bower, G. (1988a). Evaluating an adaptive network model of human learning. Journal of Memory and Language, 27(2), 166-195.
38. Gluck, M., & Bower, G. (1988b). From conditioning to category learning: An adaptive network model. Journal of Experimental Psychology: General, 117(3), 227-247.
39. Glymour, C. (2003). Learning, prediction and causal bayes nets. Trends in Cognitive Sciences, 7(1), 43-48.
40. Gopnik, A., & Sobel, D. (2000). Detecting blickets: How young children use information about novel causal powers in categorization and induction. Child Development, 7(5), 1205-1222.
41. Griffiths, T., Chater, N., Kemp, C., Perfors, A., & Tenenbaum, J. (2010). Probabilistic models of cognition: Exploring representations and inductive biases. Trends in Cognitive Sciences, 14(8), 357-364.
42. Hennessy, J., Levin, R., & Levine, S. (1977). Influence of experiential factors and gonadal hormones on pituitary-adrenal response of the mouse to novelty and electric shock. Journal of Comparative and Physiological Psychology, 91(4), 770-777.
43. Hollerman, J., & Schultz, W. (1998). Dopamine neurons report an error in the temporal prediction of reward during learning. Nature Neuroscience, 1(4), 304-309.
44. Horvitz, J., Stewart, T., & Jacobs, B. (1997). Burst activity of ventral tegmental dopamine neurons is elicited by sensory stimuli in the awake cat. Brain Research, 759(2), 251-258.
45. Houk, J., Adams, J., & Barto, A. (1995). A model of how the basal ganglia generate and use neural signals that predict reinforcement. In J. Houk, J. Davis, & D. Beiser (Eds.), Models of information processing in the basal ganglia (pp. 249-270). Cambridge, MA: MIT Press.
46. Howard, R. (1966). Information value theory. IEEE Transactions on Systems Science and Cybernetics, 2(1), 22-26.
47. Hughes, R. (2007). Neotic preferences in laboratory rodents: Issues, assessment and substrates. Neuroscience & Biobehavioral Reviews, 31(3), 441-464.
48. Kakade, S., & Dayan, P. (2002a). Acquisition and extinction in autoshaping. Psychological Review, 109(3), 533-544.
49. Kakade, S., & Dayan, P. (2002b). Dopamine: Generalization and bonuses. Neural Networks, 15(4-6), 549-559.
50. Kemp, C., Goodman, N., & Tenenbaum, J. (2010). Learning to learn causal models. Cognitive Science, 34(7), 1185-1243.
51. Kemp, C., Perfors, A., & Tenenbaum, J. (2007). Learning overhypotheses with hierarchical Bayesian models. Developmental Science, 10(3), 307-321.
52. King, D., & Appelbaum, J. (1973). Effect of trials on "emotionality' behavior of the rat and mouse. Journal of Comparative and Physiological Psychology, 85(1), 186-194.
53. Lien, Y., & Cheng, P. (2000). Distinguishing genuine from spurious causes: A coherence hypothesis. Cognitive Psychology, 40(2), 87-137.
54. Love, B., Medin, D., & Gureckis, T. (2004). Sustain: A network model of category learning. Psychological Review, 111(2), 309.
55. Lucas, C., & Griffiths, T. (2010). Learning the form of causal relationships using hierarchical Bayesian models. Cognitive Science, 34(1), 113-147.
56. Marr, D. (1982). Vision. Cambridge, MA: Freeman.
57. Mitchell, T. (1997). Machine learning. Boston: McGraw-Hill.
58. Montague, P., Dayan, P., & Sejnowski, T. (1996). A framework for mesencephalic dopamine systems based on predictive hebbian learning. The Journal of Neuroscience, 16(5), 1936-1947.
59. Myers, A., & Miller, N. (1954). Failure to find a learned drive based on hunger; evidence for learning motivated by exploration. Journal of Comparative and Physiological Psychology, 47(6), 428-436.
60. Ng, A., Harada, D., & Russell, S. (1999). Policy invariance under reward transformations: Theory and application to reward shaping. In I. Bratko, & S. Dzeroski (Eds.), Proceedings of the Sixteenth International Conference on Machine Learning. Bled, Slovenia.
61. Nissen, H. (1930). A study of exploratory behavior in the white rat by means of the obstruction method. Journal of Genetic Psychology, 37(3), 361-376.
62. Niv, Y. (2009). Reinforcement learning in the brain. Journal of Mathematical Psychology, 53(3), 139-154.
63. Nosofsky, R. (1986). Attention, similarity, and the identification-categorization relationship. Journal of Experimental Psychology: General, 115(1), 39-57.
64. Payzan-LeNestour, E., & Bossaerts, P. (2011). Risk, unexpected uncertainty, and estimation uncertainty: Bayesian learning in unstable settings. PLoS Computational Biology, 7(1), e1001048.
65. Pearl, J. (1988). Probabilistic reasoning in intelligent systems: Networks of plausible inference. San Francisco, CA: Morgan Kaufmann.
66. Rajecki, D. (1974). Effects of prenatal exposure to auditory or visual stimulation on postnatal distress vocalizations in chicks. Behavioral Biology, 11(4), 525-536.
67. Rao, R. (2010). Decision making under uncertainty: A neural model based on partially observable markov decision processes. Frontiers in Computational Neuroscience, 4(4), 146.
68. Redish, A., Jensen, S., Johnson, A., & Kurth-Nelson, Z. (2007). Reconciling reinforcement learning models with behavioral extinction and renewal: implications for addiction, relapse, and problem gambling. Psychological Review, 114(3), 784-805.
69. Reed, P., Mitchell, C., & Nokes, T. (1996). Intrinsic reinforcing properties of putatively neutral stimuli in an instrumental two-lever discrimination task. Animal Learning and Behavior, 24(1), 38-45.
70. Reichel, C., & Bevins, R. (2008). Competition between the conditioned rewarding effects of cocaine and novelty. Behavioral Neuroscience, 122(1), 140-150.
71. Rescorla, R. & Wagner, A. (1972). Variations in the effectiveness of reinforcement and nonreinforcement. In A. H. Black, & W. F. Prokasy (Eds.), Classical conditioning. II: Current research and theory (pp. 64-99). New York: Appleton-Century-Crofts.
72. Ribas-Fernandes, J., Solway, A., Diuk, C., McGuire, J., Barto, A., Niv, Y., & Botvinick, M. (2011). A neural signature of hierarchical reinforcement learning. Neuron, 71(2), 370-379.
73. Robert, C., & Casella, G. (2004). Monte Carlo statistical methods. New York: Springer Verlag.
74. Schönberg, T., Daw, N., Joel, D., & O'Doherty, J. (2007). Reinforcement learning signals in the human striatum distinguish learners from nonlearners during reward-based decision making. The Journal of Neuroscience, 27(47), 12860-12867.
75. Schultz, W. (1998). Predictive reward signal of dopamine neurons. Journal of Neurophysiology, 80(1), 1-27.
76. Schultz, W., Dayan, P., & Montague, P. (1997). A neural substrate of prediction and reward. Science, 275(5306), 1593-1599.
77. Sheldon, A. (1969). Preference for familiar versus novel stimuli as a function of the familiarity of the environment. Journal of Comparative and Physiological Psychology, 67(4), 516-521.
78. Shepard, R. (1987). Toward a universal law of generalization for psychological science. Science, 237(4820), 1317-1323.
79. Shi, L., Griffiths, T., Feldman, N., & Sanborn, A. (2010). Exemplar models as a mechanism for performing bayesian inference. Psychonomic Bulletin & Review, 17(4), 443-464.
80. Steyvers, M., Lee, M., & Wagenmakers, E. (2009). A Bayesian analysis of human decision-making on bandit problems. Journal of Mathematical Psychology, 53(3), 168-179.
81. Suri, R., Schultz, W., et al. (1999). A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task. Neuroscience, 91(3), 871-890.
82. Sutton, R., & Barto, A. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
83. Urcelay, G., & Miller, R. (2010). On the generality and limits of abstraction in rats and humans. Animal Cognition, 13(1), 21-32.
84. Waldmann, M., & Hagmayer, Y. (2006). Categories and causality: The neglected direction. Cognitive Psychology, 53(1), 27-58.
85. Weiskrantz, L., & Cowey, A. (1963). The aetiology of food reward in monkeys. Animal Behaviour, 11(2-3), 225-234.
86. Weizmann, F., Cohen, L., & Pratt, R. (1971). Novelty, familiarity, and the development of infant attention. Developmental Psychology, 4(2), 149-154.
87. Widrow, B., & Hoff, M. (1960). Adaptive switching circuits. In IRE WES CON convention record (Vol. 4, pp. 96-104). New York: IRE-New York.
88. Wittmann, B., Daw, N., Seymour, B., & Dolan, R. (2008). Striatal activity underlies novelty based choice in humans. Neuron, 58(6), 967-973.
89. Zajonc, R. (2001). Mere exposure: A gateway to the subliminal. Current Directions in Psychological Science, 10(6), 224-228.