Information-seeking, curiosity, and attention: Computational and neural mechanisms

Trends in Cognitive Sciences

Volumn 17, Issue 11, 2013, Pages 585-593

  Gottlieb, Jacqueline ^a   Oudeyer, Pierre Yves ^b,c   Lopes, Manuel ^b,c   Baranes, Adrien ^a  

^a Och Spine at New York Presbyterian Hospitals   (United States)

^b INRIA   (France)

^c ENSTA PARISTECH   (France)

Author keywords

[No Author keywords available]

Indexed keywords

INFORMATION GAIN; INFORMATION SEEKING; LEARNING PROGRESS; MODEL SYSTEM; NEURAL MECHANISMS; OCULOMOTOR SYSTEMS; SYSTEMATIC SEARCHES; VISUAL INFORMATION;

EYE MOVEMENTS;

SEARCH ENGINES;

ATTENTION; AWARENESS; BEHAVIOR CHANGE; COGNITION; COMPUTATION MECHANISM; CURIOSITY; DECISION MAKING; DOPAMINERGIC SYSTEM; EYE MOVEMENT; EYE MOVEMENT CONTROL; HUMAN; INFORMATION SEEKING; MACHINE LEARNING; MENTAL FUNCTION; MOTIVATION; NERVE POTENTIAL; NONHUMAN; PRISM; REINFORCEMENT; REM SLEEP; REVIEW; REWARD; SACCADIC EYE MOVEMENT; UNCERTAINTY; VISUAL INFORMATION; WHITE NOISE;

ANIMALS; ARTIFICIAL INTELLIGENCE; ATTENTION; BRAIN; EXPLORATORY BEHAVIOR; HUMANS; INFORMATION SEEKING BEHAVIOR;

EID: 84886642127     PISSN: 13646613     EISSN: 1879307X     Source Type: Journal    
DOI: 10.1016/j.tics.2013.09.001     Document Type: Review

References (90)

1. Lowenstein G. The psychology of curiosity: a review and reinterpretation. Psychol. Bull. 1994, 116:75-98.
2. Kang M.J., et al. The wick in the candle of learning: epistemic curiosity activates reward circuitry and enhances memory. Psychol. Sci. 2009, 20:963-973.
3. Jepma M., et al. Neural mechanisms underlying the induction and relief of perceptual curiosity. Front. Behav. Neurosci. 2012, 6:5.
4. Tatler B.W., et al. Eye guidance in natural vision: reinterpreting salience. J. Vis. 2011, 11:5-25.
5. Kaelbling L.P., et al. Planning and acting in partially observable stochastic domains. Artif. Intell. 1998, 101:99-134.
6. Dayan P., Daw N.D. Decision theory, reinforcement learning, and the brain. Cogn. Affect. Behav. Neurosci. 2008, 8:429-453.
7. Bialek W., et al. Predictability, complexity, and learning. Neural Comput. 2001, 13:2409-2463.
8. Singh S., et al. Predictive state representations: a new theory for modeling dynamical systems. Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence 2004, 512-519. AUAI Press.
9. Tishby N., Polani D. Information theory of decisions and actions. Perception-Action Cycle 2011, 601-636. Springer. V. Cutsuridis (Ed.).
10. Sutton R.S., Barto A.G. Reinforcement Learning: An Introduction 1998, MIT Press.
11. Kober J., Peters J. Reinforcement learning in robotics: a survey. Adaptation, Learning, and Optimization 2012, Vol. 12:579-610. Springer. W. Wiering, M. Van Otterlo (Eds.).
12. Spall J.C. Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control 2005, Wiley.
13. Goldberg D.E. Genetic Algorithms in Search, Optimization, and Machine Learning 1989, Addison-Wesley Longman.
14. Jones D.R., et al. Efficient global optimization of expensive black-box functions. J. Global Optim. 1998, 13:455-492.
15. Brafman R.I., Tennenholtz M. R-max - a general polynomial time algorithm for near-optimal reinforcement learning. J. Mach. Learn. Res. 2003, 3:213-231.
16. Sutton R.S., et al. Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems 2000, Vol. 12:1057-1063. MIT Press.
17. Sutton R.S. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. Proceedings of the 7th International Conference on Machine Learning 1990, 216-224. ICML.
18. Dayan P., Sejnowski T.J. Exploration bonuses and dual control. Mach. Learn. 1996, 25:5-22.
19. Kearns M., Singh S. Near-optimal reinforcement learning in polynomial time. Mach. Learn. 2002, 49:209-232.
20. Kolter J.Z., Ng A.Y. Near-Bayesian exploration in polynomial time. Proceedings of the 26th International Conference on Machine Learning 2009, 513-520. ICML.
21. Lopes M., et al. Exploration in model-based reinforcement learning by empirically estimating learning progress. Neural Inf. Process. Syst. 2012, (NIPS 2012).
22. Dayan P. Exploration from generalization mediated by multiple controllers. Intrinsically Motivated Learning in Natural and Artificial Systems 2013, 73-91. Springer. G. Baldassarre, M. Mirolli (Eds.).
23. Blake A., Yuille A.A.L. Active Vision 1992, MIT Press.
24. Tsotsos J.K. A Computational Perspective on Visual Attention 2011, MIT Press.
25. O'Regan J.K. Why Red Doesn't Sound Like a Bell: Understanding the Feel of Consciousness 2011, Oxford University Press.
26. Friston K., Ao P. Free energy, value, and attractors. Comput. Math. Methods Med. 2012, 2012:937860.
27. Johansson R.S., et al. Eye-hand coordination in object manipulation. J. Neurosci. 2001, 21:6917-6932.
28. Jovancevic-Misic J., Hayhoe M. Adaptive gaze control in natural environments. J. Neurosci. 2009, 29:6234-6238.
29. Sailer U., et al. Eye-hand coordination during learning of a novel visuomotor task. J. Neurosci. 2005, 25:8833-8842.
30. Land M.F. Eye movements and the control of actions in everyday life. Prog. Retin Eye Res. 2006, 25:296-324.
31. Kable J.W., Glimcher P.W. The neurobiology of decision: consensus and controversy. Neuron 2009, 63:733-745.
32. Gottlieb J. Attention, learning, and the value of information. Neuron 2012, 76:281-295.
33. Berlyne D. Conflict, Arousal and Curiosity 1960, McGraw-Hill.
34. Redgrave P., et al. What is reinforced by phasic dopamine signals?. Brain Res. Rev. 2008, 58:322-339.
35. Bromberg-Martin E.S., Hikosaka O. Midbrain dopamine neurons signal preference for advance information about upcoming rewards. Neuron 2009, 63:119-126.
36. Marr D. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information 2010, MIT Press.
37. Singh S., et al. Intrinsically motivated reinforcement learning: an evolutionary perspective. IEEE Trans. Auton. Ment. Dev. 2010, 2:70-82.
38. Sorg J., et al. Reward design via online gradient ascent. Advances in Neural Information Processing Systems 2010, Vol. 23:2190-2198. MIT Press.
39. Lehman J., Stanley K.O. Abandoning objectives: evolution through the search for novelty alone. Evol. Comput. 2011, 19:189-223.
40. Sequeira P., et al. Emerging social awareness: exploring intrinsic motivation in multiagent learning. IEEE International Conference on Development and Learning (ICDL) 2011.
41. Friston K. The free-energy principle: a unified brain theory?. Nat. Rev. Neurosci. 2010, 11:127-138.
42. Friston K., et al. Free-energy minimization and the dark-room problem. Front. Psychol. 2012, 3:130.
43. Weng J., et al. Autonomous mental development by robots and animals. Science 2001, 291:599-600.
44. Asada M., et al. Cognitive developmental robotics: a survey. IEEE Trans. Autonom. Ment. Dev. 2009, 1:12-34.
45. Oudeyer P.Y. On the impact of robotics in behavioral and cognitive sciences: from insect navigation to human cognitive development. IEEE Trans. Auton. Ment. Dev. 2010, 2:2-16.
46. Lopes M., Oudeyer P.Y. The strategic student approach for life-long exploration and learning. IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL) 2012.
47. Baldassarre G., Mirolli M. Intrinsically Motivated Learning in Natural and Artificial Systems 2013, Springer-Verlag.
48. Oudeyer P.Y., et al. Intrinsically motivated learning of real-world sensorimotor skills with developmental constraints. Intrinsically Motivated Learning in Natural and Artificial Systems 2013, 303-365. Springer. G. Baldassarre, M. Mirolli (Eds.).
49. Ranganath C., Rainer G. Neural mechanisms for detecting and remembering novel events. Nat. Rev. Neurosci. 2003, 4:193-202.
50. Duzel E., et al. Novelty-related motivation of anticipation and exploration by dopamine (NOMAD): implications for healthy aging. Neurosci. Biobehav. Rev. 2010, 34:660-669.
51. Boehnke S.E., et al. Visual adaptation and novelty responses in the superior colliculus. Eur. J. Neurosci. 2011, 34:766-779.
52. Itti L., Baldi P. Bayesian surprise attracts human attention. Vis. Res. 2009, 49:1295-1306.
53. Oudeyer P.Y., Kaplan F. What is intrinsic motivation? A typology of computational approaches. Front. Neurorobot. 2007, 1:6.
54. Cohn D.A. Active learning with statistical models. J. Artif. Intell. Res. 1996, 4:129-145.
55. Rothkopf C.A., Ballard D. Credit assignment in multiple goal embodied visuomotor behavior. Front. Psychol. 2010, 1:173.
56. Thrun S. Exploration in active learning. Handbook of Brain Science and Neural Networks 1995, 381-384. MIT Press. M.A. Arbib (Ed.).
57. Oudeyer P.Y., et al. Intrinsic motivation systems for autonomous mental development. IEEE Trans. Evol. Comput. 2007, 11:265-286.
58. Schmidhuber J. Maximizing fun by creating data with easily reducible subjective complexity. Intrinsically Motivated Learning in Natural and Artificial Systems 2013, 95-128. Springer. G. Baldassarre, M. Mirolli (Eds.).
59. Schmidhuber J. Curious model-building control systems. Proceedings of the International Joint Conference on Neural Networks 1991, 1458-1463. IEEE.
60. Baranes A., Oudeyer P.Y. Active learning of inverse models with intrinsically motivated goal exploration in robots. Robot. Auton. Syst. 2013, 61:49-73.
61. Srivastava R.K., et al. First experiments with PowerPlay. Neural Netw. 2013, 41:130-136.
62. Pape L., et al. Learning tactile skills through curious exploration. Front. Neurorobot. 2012, 6:6.
63. Ngo H., et al. Learning skills from play: artificial curiosity on a Katana robot arm. Proceedings of the 2012 International Joint Conference on Neural Networks (IJCNN) 2012, 1-8. IEEE.
64. Nguyen S.M., Oudeyer P.Y. Socially guided intrinsic motivation for robot learning of motor skills. Auton Robots 2013, 10.1007/s10514-013-9339-y.
65. Oudeyer P.Y., Kaplan F. Discovering communication. Connect. Sci. 2006, 18:189-206.
66. Kaplan F., Oudeyer P.Y. From hardware and software to kernels and envelopes: a concept shift for robotics, developmental psychology and brain science. Neuromorphic and Brain-based Robots 2011, 217-250. Cambridge University Press. J.L. Krichmar, H. Wagatsuma (Eds.).
67. Moulin-Frier C., Oudeyer P.Y. Curiosity-driven phonetic learning. Proceedings of the 2012 IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL) 2012, 1-8. IEEE.
68. Smith L.B., Thelen E. Development as a dynamic system. Trends Cogn. Sci. 2003, 7:343-348.
69. Kaplan F., Oudeyer P.Y. In search of the neural circuits of intrinsic motivation. Front. Neurosci. 2007, 1:225-236.
70. Fleming S.M., et al. Relating introspective accuracy to individual differences in brain structure. Science 2010, 329:1541-1543.
71. Fleming S.M., et al. Prefrontal contributions to metacognition in perceptual decision making. J. Neurosci. 2012, 32:6117-6125.
72. De Martino B., et al. Confidence in value-based choice. Nat. Neurosci 2013, 16:105-110.
73. Isoda M., Hikosaka O. Switching from automatic to controlled action by monkey medial frontal cortex. Nat Neurosci. 2007, 10:240-248.
74. Quilodran R., et al. Behavioral shifts and action valuation in the anterior cingulate cortex. Neuron 2008, 57:314-325.
75. Wurtz, R.H. and Goldberg, M.E., eds (1989) The Neurobiology of Saccadic Eye Movements. In Reviews of Oculomotor Research (Vol. III), Elsevier.
76. Thompson K.G., Bichot N.P. A visual salience map in the primate frontal eye field. Prog. Brain Res. 2005, 147:251-262.
77. Ding L., Hikosaka O. Comparison of reward modulation in the frontal eye field and caudate of the macaque. J. Neurosci. 2006, 26:6695-6703.
78. Isoda M., Hikosaka O. A neural correlate of motivational conflict in the superior colliculus of the macaque. J. Neurophysiol. 2008, 100:1332-1342.
79. Yasuda M., et al. Robust representation of stable object values in the oculomotor basal ganglia. J. Neurosci. 2012, 32:16917-16932.
80. Gottlieb J., Balan P.F. Attention as a decision in information space. Trends Cogn. Sci. 2010, 14:240-248.
81. Suzuki M., Gottlieb J. Distinct neural mechanisms of distractor suppression in the frontal and parietal lobe. Nat. Neurosci. 2013, 16:98-104.
82. Peck C.J., et al. Reward modulates attention independently of action value in posterior parietal cortex. J Neurosci 2009, 29:11182-11191.
83. Hogarth L., et al. Selective attention to conditioned stimuli in human discrimination learning: untangling the effects of outcome prediction, valence, arousal and uncertainty. Attention and Associative Learning: From Brain to Behaviour 2010, 71-98. Oxford University Press. C.J. Mitchell, M.E. Le Pelley (Eds.).
84. Thompson K.G., et al. Neuronal basis of covert spatial attention in the frontal eye field. J. Neurosci. 2005, 25:9479-9487.
85. Oristaglio J., et al. Integration of visuospatial and effector information during symbolically cued limb movements in monkey lateral intraparietal area. J. Neurosci. 2006, 26:8310-8319.
86. Leathers M.L., Olson C.R. In monkeys making value-based decisions, LIP neurons encode cue salience and not action value. Science 2012, 338:132-135.
87. Holland P.C., Maddux J-M. Brain systems of attention in associative learning. Attention and Associative Learning: From Brain to Behaviour 2010, 305-350. Oxford University Press. C.J. Mitchell, M.E. Le Pelley (Eds.).
88. Pearce J.M., Mackintosh N.J. Two Theories of Attention: A Review and a Possible Integration 2010, Oxford University Press.
89. Bach D.R., Dolan R.J. Knowing how much you don't know: a neural organization of uncertainty estimates. Nat. Rev. Neurosci. 2012, 13:572-586.
90. Bubeck S., Cesa-Bianchi N. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Found. Trends Mach. Learn. 2012, 5:1-122.