Novelty or Surprise?

Frontiers in Psychology

Volumn 4, Issue DEC, 2013, Pages

  Barto, Andrew ^a   Mirolli, Marco ^b   Baldassarre, Gianluca ^b  

^a UNIVERSITY OF MASSACHUSETTS   (United States)

^b INSTITUTE OF COGNITIVE SCIENCES AND TECHNOLOGIES   (Italy)

Author keywords

Expectation; Intrinsic motivation; Novelty; Novelty detection; Surprise

Indexed keywords

EID: 84891714978     PISSN: None     EISSN: 16641078     Source Type: Journal    
DOI: 10.3389/fpsyg.2013.00907     Document Type: Article

References (99)

1. Asaad, W. F., Rainer, G., and Miller, E. K. (1998). Neural activity in the primate prefrontal cortex during associative learning. Neuron 21, 1399-1407. doi: 10.1016/S0896-6273(00)80658-3.
2. Baldassarre, G., and Mirolli, M. (eds.). (2013). Intrinsically Motivated Learning in Natural and Artificial Systems. Berlin: Springer-Verlag. doi: 10.1007/978-3-642-32375-1.
3. Barto, A. G. (1990). "Connectionist learning for control: an overview," in Neural Networks for Control, eds T. Miller, R. S. Sutton, and P. J. Werbos (Cambridge, MA: MIT Press), 5-58.
4. Barto, A. G. (1995). "Adaptive critics and the basal ganglia," in Models of Information Processing in the Basal Ganglia, eds J. C. Houk, J. L. Davis, and D. G. Beiser (Cambridge, MA: MIT Press), 215-232.
5. Berlyne, D. E. (1960). Conflict, Arousal, and Curiosity. New York, NY: McGraw-Hill. doi: 10.1037/11164-000.
6. Berridge, K. (2007). The debate over dopamine's role in reward: the case for incentive salience. Psychopharmacology 191, 391-431. doi: 10.1007/s00213-006-0578-x.
7. Bishop, C. M. (2006). Pattern Recognition and Machine Learning. New York, NY: Springer.
8. Brown, M. W., and Aggleton, J. P. (2001). Recognition memory: what are the roles of the perirhinal cortex and hippocampus? Nat. Rev. Neurosci. 2, 51-61. doi: 10.1038/35049064.
9. Bunzeck, N., and Duzel, E. (2006). Absolute coding of stimulus novelty in the human substantia nigra/vta. Neuron 51, 369-379. doi: 10.1016/j.neuron.2006.06.021.
10. Courville, A. C., Daw, N. D., Gordon, G. J., and Touretzky, D. S. (2004). "Model uncertainty in classical conditioning," in Advances in Neural Information Processing Systems 16, eds S. Thrun, L. Saul, and B. Schölkopf (Cambridge, MA: MIT Press), 977-984.
11. Courville, A. C., Daw, N. D., and Touretzky, D. S. (2006). Bayesian theories of conditioning in a changing world. Trends Cogn. Sci. 10, 294-300. doi: 10.1016/j.tics.2006.05.004.
12. Dayan, P., Kakade, S., and Montague, P. R. (2000). Learning and selective attention. Nat. Neurosci. Suppl. 3, 1218-1223. doi: 10.1038/81504.
13. Dayan, P., and Long, T. (1998). "Statistical models of learnng," in Advances in Neural Information Processing Systems 10: Proceedings of the 1997 Conference, eds M. I. Jordan, M. J. Kearns, and S. A. Solla (Cambridge, MA: MIT Press), 117-123.
14. Deci, E., and Ryan, R. (1985). Intrinsic Motivation and Self-Determination in Human Behavior. New York, NY: Plenum Press. doi: 10.1007/978-1-4899-2271-7.
15. Dommett, E., Coizet, V., Blaha, C. D., Martindale, J., Lefebvre, V., Walton, N., et al. (2005). How visual stimuli activate dopaminergic neurons at short latency. Science 307, 1476-1479. doi: 10.1126/science.1107026.
16. Duda, R. O., and Hart, P. E. (1973). Pattern Classification and Scene Analysis. New York, NY: Wiley.
17. Duncan, K., Ketz, N., Inati, S. J., and Davachi, L. (2012). Evidence for area ca1 as a match/mismatch detector: a high-resolution fmri study of the human hippocampus. Hippocampus 22, 389-398. doi: 10.1002/hipo.20933.
18. Duzel, E., Habib, R., Rotte, M., Guderian, S., Tulving, E., and Heinze, H. (2003). Human hippocampal and parahippocampal activity during visual associative recognition memory for spatial and nonspatial stimulus configurations. J. Neurosci. 23, 9439-9444. Available online at: http://www.jneurosci.org/content/23/28/9439.long.
19. Ekman, P., and Davidson, R. J. (eds.). (1994). The Nature of Emotion: Fundamental Questions. Oxford: Oxford University Press.
20. Engle, Y., Mannor, S., and Meir, R. (2003). "Bayes meets Bellman: the Gaussian process approach to temporal difference learning," in Proceedings of the twentieth International Conference on Machine Learning (ICML-2003) (Washington, DC), 154-161.
21. Friedman, D., Cycowicz, Y. M., and Gaeta, H. (2001). The novelty p3: an event-related brain potential (erp) sign of the brain's evaluation of novelty. Neurosci. Biobehav. Rev. 25, 355-373. doi: 10.1016/S0149-7634(01)00019-7.
22. Friston, K. (2009). The free-energy principle: a rough guide to the brain? Trends Cogn. Sci. 13, 293-301. doi: 10.1016/j.tics.2009.04.005.
23. Friston, K. (2010). The free-energy principle: a unified brain theory? Nat. Rev. Neurosci. 11, 127-138. doi: 10.1038/nrn2787.
24. Friston, K., and Kiebel, S. (2009). Predictive coding under the free-energy principle. Philos. Trans. R. Soc. B 364, 1211-1221. doi: 10.1098/rstb.2008.0300.
25. Friston, K. J., Kilner, J., and Harrison, L. (2006). A free-energy principle for the brain. J. Physiol. Paris 100, 70-87. doi: 10.1016/j.jphysparis.2006.10.001.
26. Gershman, S. J., and Blei, D. M. (2012). A tutorial on bayesian nonparametric models. J. Math. Psychol. 56, 1-12. doi: 10.1016/j.jmp.2011.08.004.
27. Glimcher, P. (2011). Understanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis. Proc. Natl. Acad. Sci. U.S.A. 108(Suppl. 3), 15647-15654. doi: 10.1073/pnas.1014269108.
28. Glow, P. (1970). Some acquisition and performance characteristics of response contingent sensory reinforcement in the rat. Aust. J. Psychol. 22, 145-154. doi: 10.1080/00049537008254568.
29. Glow, P., and Winefield, A. (1978). Response-contingent sensory change in a causally structured environment. Learn. Behav. 6, 1-18. doi: 10.3758/BF03211996.
30. Graybiel, A. (2008). Habits, rituals, and the evaluative brain. Annu. Rev. Neurosci. 31, 359-387. doi: 10.1146/annurev.neuro.29.051605.112851.
31. Grossberg, S. (1982). Processing of expected and unexpected events during conditioning and attention: a psychophysiological theory. Psychol. Rev. 89, 529-572. doi: 10.1037/0033-295X.89.5.529.
32. Hasselmo, M. E., and Schnell, E. (1994). Laminar selectivity of the cholinergic suppression of synaptic transmission in rat hippocampal region ca1: computational modeling and brain slice physiology. J. Neurosci. 14, 3898-3914.
33. Henson, R. N. A., and Rugg, M. D. (2003). Neural response suppression, haemodynamic repetition effects, and behavioural priming. Neuropsychologia 41, 263-270. doi: 10.1016/S0028-3932(02)00159-8.
34. Horvitz, J. C. (2000). Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience 96, 651-656. doi: 10.1016/S0306-4522(00)00019-1.
35. Horvitz, J. C., Stewart, T., and Jacobs, B. L. (1997). Burst activity of ventral tegmental dopamine neurons is elicited by sensory stimuli in the awake cat. Brain Res. 759, 251-258. doi: 10.1016/S0006-8993(97)00265-5.
36. Houk, J. C., Adams, J. L., and Barto, A. G. (1995). "A model of how the basal ganglia generates and uses neural signals that predict reinforcement," in Models of Information Processing in the Basal Ganglia, eds J. C. Houk, J. L. Davis, and D. G. Beiser (Cambridge, MA: MIT Press), 249-270.
37. Huron, D. (2004). "Music-engendered laughter: an analysis of humor devices in PDQ Bach," in Proceedings of the 8th International Conference on Music Perception and Cognition, eds S. D. Lipscomb, R. Ashley, R. O. Gjerdingen, and P. Webster (Adelaide, SA: Causal Productions), 700-704.
38. Itti, L., and Baldi, P. F. (2005). "A principled approach to detecting surprising events in video," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (San Diego, CA), 631-637.
39. Itti, L., and Baldi, P. F. (2006). "Bayesian surprise attracts human attention," in Advances in Neural Information Processing Systems 18 (NIPS*2005), eds Y. Weiss, B. Schölkopf, and J. Platt (Cambridge, MA: MIT Press), 547-554.
40. Itti, L., and Baldi, P. F. (2009). Bayesian surprise attracts human attention. Vis. Res. 49, 1295-1306. doi: 10.1016/j.visres.2008.09.007.
41. Kakade, S., and Dayan, P. (2002a). Acquisition and extinction in autoshaping. Psychol. Rev. 109, 533-544. doi: 10.1037/0033-295X.109.3.533.
42. Kakade, S., and Dayan, P. (2002b). Dopamine: generalization and bonuses. Neural Netw. 15, 549-559. doi: 10.1016/S0893-6080(02)00048-5.
43. Kamin, L. J. (1969). "Predictability, surprise, attention, and conditioning," in Punishment and Aversive Behavior, eds B. A. Campbell and R. M. Church (New York, NY: Appleton-Century-Crofts), 279-296.
44. Kish, G. B. (1955). Learning when the onset of illumination is used as reinforcing stimulus. J. Comp. Physiol. Psychol. 48, 261-264. doi: 10.1037/h0040782.
45. Kohler, S., Danckert, S., Gati, J., and Menon, R. (2005). Novelty responses to relational and non-relational information in the hippocampus and the parahippocampal region: a comparison based on event-related fmri. Hippocampus 15, 763-774. doi: 10.1002/hipo.20098.
46. Kohonen, T. (1977). Associative Memory: A System Theoretic Approach. Berlin: Springer-Varlag.
47. Kohonen, T. (1980). Content-Addressable Memories. Berlin: Springer-Verlag. doi: 10.1007/978-3-642-96552-4.
48. Kohonen, T. (1984). Self-Organization and Associative Memory. Berlin: Springer-Verlag.
49. Krebs, R. M., Schott, B. H., Schutze, H., and Duzel, E. (2009). The novelty exploration bonus and its attentional modulation. Neuropsychologia 47, 2272-2281. doi: 10.1016/j.neuropsychologia.2009.01.015.
50. Kumaran, D., and Maguire, E. A. (2007). Which computational mechanisms operate in the hippocampus during novelty detection? Hippocampus 17, 735-748. doi: 10.1002/hipo.20326.
51. Lepora, N. F., Porrill, J., Yeo, C. H., and Dean, P. (2010). Sensory prediction or motor control? Application of Marr-Albus type models of cerebellar function to classical conditioning. Front. Comput. Neurosci. 4:140. doi: 10.3389/fncom.2010.00140.
52. Li, L., Miller, E. K., and Desimone, R. (1993). The representation of stimulus familiarity in anterior inferior temporal cortex. J. Neurophysiol. 69, 1918-1929.
53. Lisman, J. E., and Grace, A. A. (2005). The hippocampal-vta loop: controlling the entry of information into long-term memory. Neuron 46, 703-713. doi: 10.1016/j.neuron.2005.05.002.
54. Ljungberg, T., Apicella, P., and Schultz, W. (1992). Responses of monkey dopamine neurons during learning of behavioral reactions. J. Neurophysiol. 67, 145-163.
55. Mannella, F., Zappacosta, S., Mirolli, M., and Baldassarre, G. (2010). "A computational model of the amygdala nuclei's role in second order conditioning," in From Animals to Animats 10: Proceedings of the Tenth International Conference on the Simulation of Adaptive behavior (SAB2008), Llecture Notes in Artificial Intelligence 5040, eds M. Asada, J. C. Hallam, J.-A. Meyer, and J. Tani (Berlin: Springer-Verlag).
56. Markou, M., and Singh, S. (2003). Novelty detection: a review-part 1: statistical approaches. Signal Process. 83, 2481-2497. doi: 10.1016/j.sigpro.2003.07.018.
57. Marsland, S. (2003). Novelty detection in learning systems. Neural Comput. Surv. 3, 157-195. Available online at: http://seat.massey.ac.nz/personal/s.r.marsland/PUBS/NCS.pdf.
58. Mathys, C., Daunizeau, J., Friston, K. J., and Stephan, K. E. (2011). A Bayesian foundation for individual learning under uncertainty. Front. Hum. Neurosci. 5:39. doi: 10.3389/fnhum.2011.00039.
59. Mirolli, M., Santucci, V., and Baldassarre, G. (2013). Phasic dopamine as a prediction error signal of intrinsic and extrinsic reinforcements: a computational model. Neural Netw. 39, 40-51. doi: 10.1016/j.neunet.2012.12.012.
60. Monsalve, I. F., Frank, S. L., and Vigliocco, G. (2012). "Lexical surprisal as a general predictor of reading time," in Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics (Stroudsburg, PA: Association for Computational Linguistics), 398-408.
61. Moore, J. W., and Schmajuk, N. A. (2008). Kamin blocking. Scholarpedia 3, 3542. doi: 10.4249/scholarpedia.3542.
62. Nehmzow, U., Gatsoulis, Y., Kerr, E., Condell, J., Siddique, N., and McGinnity, T. (2013). "Novelty detection as an intrinsic motivation for cumulative learning robots," in Intrinsically Motivated Learning in Natural and Artificial Systems, eds G. Baldassarre and M. Mirolli (Berlin: Springer-Verlag), 185-207.
63. O'Keefe, J., and Nadel, L. (1978). The Hippocampus as a Cognitive Map. Oxford: Oxford University Press.
64. Palm, G. (2012). Novelty, Information and Surprise. Berlin: Springer-Verlag. doi: 10.1007/978-3-642-29075-6.
65. Pearce, J. M., and Hall, G. (1980). A model for Pavlovian learning: variation in the effectiveness of conditioning but not unconditioned stimuli. Psychol. Rev. 87, 532-552. doi: 10.1037/0033-295X.87.6.532.
66. Ranganath, C., and Paller, K. A. (1999). Frontal brain activity during episodic and semantic retrieval: insights from event-related potentials. J. Cogn. Neurosci. 11, 598-609. doi: 10.1162/089892999563661.
67. Ranganath, C., and Rainer, G. (2003). Neural mechanisms for detecting and remembering novel events. Nat. Rev. Neurosci. 4, 193-202. doi: 10.1038/nrn1052.
68. Rao, R. P., and Ballard, D. H. (1999). Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat. Neurosci. 2, 79-87. doi: 10.1038/4580.
69. Redgrave, P., and Gurney, K. (2006). The short-latency dopamine signal: a role in discovering novel actions? Nat. Rev. Neurosci. 7, 967-975. doi: 10.1038/nrn2022.
70. Redgrave, P., Prescott, T. J., and Gurney, K. (1999). Is the short-latency dopamine response too short to signal reward error? Trends Neurosci. 22, 146-151. doi: 10.1016/S0166-2236(98)01373-3.
71. Reed, P., Mitchell, C., and Nokes, T. (1996). Intrinsic reinforcing properties of putatively neutral stimuli in an instrumental two-lever discrimination task. Anim. Learn. Behav. 24, 38-45. doi: 10.3758/BF03198952.
72. Rescorla, R. A. (1971). Variations in the effectiveness of reinforcement and nonreinforcement following prior inhibitory conditioning. Learn. Motiv. 2, 113-123. doi: 10.1016/0023-9690(71)90002-6.
73. Rescorla, R. A., and Wagner, A. R. (1972). "A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement," in Classical Conditioning II, eds A. H. Black and W. F. Prokasy (New York, NY: Appleton-Century-Crofts), 64-99.
74. Ringo, J. L. (1996). Stimulus specific adaptation in inferior temporal and medial temporal cortex of the monkey. Behav. Brain Res. 76, 191-197. doi: 10.1016/0166-4328(95)00197-2.
75. Roark, B. (2011). Expected Surprisal and Entropy. Technical Report CSLU-11-004, Center for Spoken Language Processing, Oregon Health and Science University, (Portland, OR).
76. Robinson, S., Sotak, B., During, M., and Palmiter, R. (2006). Local dopamine production in the dorsal striatum restores goal-directed behavior in dopamine-deficient mice. Behav. Neurosci. 120, 196-200. doi: 10.1037/0735-7044.120.1.000.
77. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). "Learning internal representations by error propagation," in Parallel Distributed Processing:Explorations in the Microstructure of Cognition, Vol. 1: Foundations. eds D. E. Rumelhart and J. L. McClelland (Cambridge, MA: Bradford Books/MIT Press), 318-362.
78. Russell, A., and Glow, P. (1974). Some effects of short-term immediate prior exposure to light change on responding for light change. Learn. Behav. 2, 262-266. doi: 10.3758/BF03199191.
79. Schmajuk, N. A. (2008). Computational models of classical conditioning. Scholarpedia 3, 1664. doi: 10.4249/scholarpedia.1664.
80. Schmidhuber, J., Storck, J., and Hochreiter, S. (1994). Reinforcement driven information acquisition in nondeterministic environments. Munich: Technical report, Fakultät für Informatik, Technische Universität München.
81. Schultz, W. (1998). Predictive reward signal of dopamine neurons. J. Neurophysiol. 80, 1-27.
82. Schultz, W. (2007). Multiple dopamine functions at different time scales. Annu. Rev. Neurosci. 30, 259-288. doi: 10.1146/annurev.neuro.28.061604.135722.
83. Schultz, W., Dayan, P., and Montague, P. R. (1997). A neural substrate of prediction and reward. Science 275, 1593-1598. doi: 10.1126/science.275.5306.1593.
84. Sobotka, S., and Ringo, J. L. (1994). Stimulus specific adaptation in excited but not in inhibited cells in inferotemporal cortex of macaque. Brain Res. 646, 95-99. doi: 10.1016/0006-8993(94)90061-2.
85. Soltani, M., and Knight, R. T. (2000). Neural origins of the p300. Crit. Rev. Neurobiol. 14, 199-224. doi: 10.1615/CritRevNeurobiol.v14.i3-4.20.
86. Steinfels, G. F., Heym, J., Strecker, R. E., and Jacobs, B. L. (1983). Response of dopaminergic neurons in cat to auditory stimuli presented across the sleep-waking cycle. Brain Res. 277, 150-154. doi: 10.1016/0006-8993(83)90917-4.
87. Storck, J., Hochreiter, S., and Schmidhuber, J. (1995). "Reinforcement-driven information acquisition in non-deterministic environments," in Proceedings of ICANN'95 (Paris), Vol. 2, 159-164.
88. Sutton, R. S. (1988). Learning to predict by the method of temporal differences. Mach. Learn. 3, 9-44. doi: 10.1007/BF00115009.
89. Sutton, R. S., and Barto, A. G. (1990). "Time-derivative models of Pavlovian reinforcement," in Learning and Computational Neuroscience: Foundations of Adaptive Networks, eds M. Gabriel and J. Moore (Cambridge, MA: MIT Press), 497-537.
90. Sutton, R. S., and Tanner, B. (2004). "Temporal-difference networks," in Advances in Neural Information Processing Systems 17, [Neural Information Processing Systems, NIPS 2004], (Vancouver, BC), 1377-1384.
91. Tribus, M. (1961). Thermodynamics and Thermostatics: An Introduction to Energy, Information and States of Matter, with Engineering Applications. New York, NY: D. Van Nostrand Company Inc.
92. Ungless, M. (2004). Dopamine: the salient issue. Trends Neurosci. 27, 702-706. doi: 10.1016/j.tins.2004.10.001.
93. Wan, H., Aggleton, J. P., and Brown, M. W. (1999). Different contributions of the hippocampus and perirhinal cortex to recognition memory. J. Neurosci. 19, 1142-1148.
94. Welch, G., and Bishop, G. (1995). An introduction to the kalman filter. Technical report, Department of Computer Science, University of North Carolina at Chapel Hill, (Chapel Hill, NC).
95. Widrow, B., and Hoff, M. E. (1960). "Adaptive switching circuits," in 1960 WESCON Convention Record Part IV, (NY: Institute of Radio Engineers) 96-104, Reprinted in Anderson, J. A., and Rosenfeld, E. (1988). Neurocomputing: Foundations of Research, (Cambridge, MA: MIT Press), 126-134.
96. Williams, D., and Lowe, G. (1972). Response contingent illumination change as a reinforcer in the rat. Anim. Behav. 20, 259-262. doi: 10.1016/S0003-3472(72)80045-9.
97. Wise, R. (2004). Dopamine, learning and motivation. Nat. Rev. Neurosci. 5, 483-494. doi: 10.1038/nrn1406.
98. Wittmann, B. C., Daw, N. D., Seymour, B., and Dolan, R. J. (2008). Striatal activity underlies novelty-based choice in humans. Neuron 58, 967-973. doi: 10.1016/j.neuron.2008.04.027.
99. Zweifel, L. S., Parker, J. G., Lobb, C. J., Rainwater, A., Wall, V. Z., Fadok, J. P., et al. (2009). Disruption of nmdar-dependent burst firing by dopamine neurons provides selective assessment of phasic dopamine-dependent behavior. Proc. Natl. Acad. Sci. U.S.A. 106, 7281-7288. doi: 10.1073/pnas.0813415106.