From hardware and software to kernels and envelopes: A concept shift for robotics, developmental psychology, and brain sciences

Neuromorphic and Brain-Based Robots

Volumn , Issue , 2011, Pages 217-250

  Kaplan, Frédéric ^a   Oudeyer, Pierre Yves ^b  

^a EPFL   (Switzerland)

^b INRIA   (France)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 80155142299     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1017/CBO9780511994838.011     Document Type: Chapter

References (136)

1. Asada, M., Hosoda, K., Kuniyoshi, Y., et al. (2009). Cognitive developmental robotics: a survey. IEEE Transactions on Autonomous Mental Development, 1 (1), 12-34.
2. Baldassarre, G. (2002). A modular neural-network model of the basal ganglia's role in learning and selection motor behaviors. Journal of Cognitive Systems Research, 3 (1), 5-13.
3. Baldwin, J. M. (1925). Mental Development in the Child and the Race. New York: Macmillan.
4. Barto, A. G. (1995). Adaptive critics and the basal ganglia. In Houk, J. C., Davis, J. L., and Beiser, D. G. (eds.), Models of Information Processing in the Basal Ganglia. Cambridge, MA: MIT Press, pp. 215-232.
5. Barto, A. G., Singh, S., and Chentanez, N. (2004). Intrinsically motivated learning of hierarchical collections of skills. In Proceedings of the 3rd IEEE International Conference on Development and Learning (ICDL 2004).
6. Bell, D. S. (1973). The experimental reproduction of amphetamine psychosis. Archives in General Psychiatry, 29, 35-40.
7. Berlyne, D. E. (1960). Conflict, Arousal and Curiosity. New York: McGraw-Hill.
8. Bernheimer, H., Birkmayer, W., Hornykiewicz, O., Jellinger, K., and Seitelberger, F. (1973). Brain dopamine and the syndromes of Parkinson and Huntington: clinical, morphological and neurochemical correlations. Journal of the Neurological Sciences, 20, 415-455.
9. Berridge, K. C. (2007). The debate over dopamine's role in reward: the case of incentive salience. Psychopharmacology, 191, 391-431.
10. Brooks, R. (1999). Cambrian Intelligence: The Early History of the New AI. Cambridge, MA: MIT Press.
11. Bruner, J. (1962). On Knowing: Essays for the Left Hand. Cambridge, MA: Harvard University Press.
12. Cameron, J. and Pierce, W.D. (2002). Rewards and Intrinsic Motivation: Resolving the Controversy. South Hadley, MA: Bergin and Garvey Press.
13. Carboni, E., Imperato, A., Perezzani, L., and Di Chiara, G. (1989). Amphetamine, cocaine, phencyclidine and nomifensine increases extra cellullar dopamine concentrations preferentially in the nucleus accumbens of freely moving rats. Neuroscience, 28, 653-661.
14. Clark, A. (2004). Natural-born Cyborgs: Minds, Technologies and the Future of Human Intelligence. Oxford, UK: Oxford University Press.
15. Csikszenthmihalyi, M. (1991). Flow: The Psychology of Optimal Experience. New York: Harper Perennial.
16. Daw, N., Kakade, S., and Dayan, P. (2002). Opponent interactions between serotonin and dopamine. Neural Networks, 15, 603-616.
17. Dayan, P. and Sejnowski, T. J. (1996). Exploration bonuses and dual control. Machine Learning, 25, 5-22.
18. Dayan, P., Hinton, G., Giles, C., Hanson, S., and Cowan, J. (1993). Feudal reinforcement learning. In Advances in Neural Information Processing Systems, Vol. 5. San Mateo, CA: Morgan Kaufmann, pp. 271-278.
19. De Charms, R. (1968). Personal Causation: The Internal Affective Determinants of Behavior. New York: Academic Press.
20. Deci, E. L. and Ryan, R. M. (1985). Intrinsic Motivation and Self-Determination in Human Behavior. New York: Plenum Press.
21. Dember, W. N. and Earl, R. W. (1957). Analysis of exploratory, manipulatory and curiosity behaviors. Psychological Review, 64, 91-96.
22. Deneve, S., Duhamel, J.-R., and Pouget, A. (2007). Optimal sensorimotor integration in recurrent cortical networks: a neural implementation of Kalman filters. Journal of Neuroscience, 27 (21), 5744-5756.
23. Depue, R. A. and Iacono, W. G. (1989). Neurobehavioral aspects of affective disorders. Annual Review of Psychology, 40, 457-492.
24. Dommett, E., Coizet, V., Blatha, C. D., et al. (2005). How visual stimuli activate dopaminergic neurons at short latency. Science, 307, 1476-1479.
25. Douglas, R. and Martin, K. (1998). Neocortex. In Shepherd, G. M. (ed.), The Synaptic Organization of the Brain. Oxford, UK: Oxford University Press, pp. 459-509.
26. Doya, K. (1999). What are the computations of cerebellum, basal ganglia, and the cerebral cortex? Neural Networks, 12, 961-974.
27. Doya, K. (2002). Metalearning and neuromodulation. Neural Networks, 15 (4-5), 495-506.
28. Doya, K., Samejima, K., Katagiri, K., and Kawato, M. (2002). Multiple model-based reinforcement learning. Neural Computation, 14, 1347-1369.
29. Fedorov, V. V. (1972). Theory of Optimal Experiment. New York: Academic Press.
30. Festinger, L. (1957). A Theory of Cognitive Dissonance. Evanston, IL: Row, Peterson.
31. Fiorillo, C. D. (2004). The uncertain nature of dopamine. Molecular Psychiatry, 9, 122-123.
32. Gibson, J. (1986). The Ecological Approach to Visual Perception. Hillsdale, NJ: Lawrence Erlbaum Associates.
33. Grace, A. A. (1991). Phasic versus tonic dopamine release and the modulation of dopamine system responsivity: a hypothesis for the etiology of schizophrenia. Neuroscience, 41, 1-24.
34. Gray, J. A. (1982). The Neuropsychology of Anxiety: An Enquiry into the Functions of the Septohippocampal System. Oxford, UK: Clarendon Press.
35. Gray, J. A. (1990). Brain systems that mediate both emotion and cognition. Cognition and Emotion, 4, 269-288.
36. Grey Walter, W. (1953). The Living Brain. 2nd edn., 1967, London: Penguin.
37. Harlow, H. F. (1950). Learning and satiation of response in intrinsically motivated complex puzzle performances by monkeys. Journal of Comparative and Physiological Psychology, 43, 289-294.
38. Head, H. and Holmes, G. (1911). Sensory disturbances from cerebral lesions. Brain, 34, 102-254.
39. Heath, R. G. (1963). Electrical self-stimulation of the brain in Man. American Journal of Psychiatry, 120, 571-577.
40. Hebb, D. O. (1955). Drives and the C.N.S (conceptual nervous system). Psychological Review, 62, 243-254.
41. Hooks, M. S. and Kalivas, P. W. (1994). Involvement of dopamine and excitatory amino acid transmission in novelty-induced motor activity. Journal of Pharmacology and Experimental Therapeutics, 269, 976-988.
42. Horvitz, J.-C. (2000). Mesolimbocortical and nigrostriatal dopamine responses to salient nonreward events. Neuroscience, 96 (4), 651-656.
43. Horvitz, J.-C. (2002). Dopamine gating of glutamatergic sensorimotor and incentive motivational input signals to the striatum. Behavioral and Brain Research, 137, 65-74.
44. Houk, J. C., Adams, J. L., and Barto, A. G. (1995). A model of how the basal ganglia generate and use neural signals that predict reinforcement. In Houk, J. C., Davis, J. L., and Beiser, D. G. (eds.), Models of Information Processing in the Basal Ganglia. Cambridge, MA: MIT Press, pp. 249-270.
45. Huang, X. and Weng, J. (2002). Novelty and reinforcement learning in the value system of developmental robots. In Prince, C., Demiris, Y., Marom, Y., Kozima, H., and Balkenius, C. (eds.), Proceedings of the 2nd International Workshop on Epigenetic Robotics: Modeling Cognitive Development in Robotic Systems. Lund University Cognitive Studies 94. Lund, Sweden: Lund University, pp. 47-55.
46. Hull, C. L. (1943). Principles of Behavior: An Introduction to Behavior Theory. New York: Appleton-Century-Croft.
47. Hunt, J. McV. (1965). Intrinsic motivation and its role in psychological development. In Levine, D. (ed.), Nebraska Symposium on Motivation, Vol. 13. Lincoln, NE: University of Nebraska Press, pp. 189-282.
48. Ikemoto, S. and Panksepp, J. (1999). The role of nucleus accumbens dopamine in motivated behavior: a unifying interpretation with special reference to reward-seeking. Brain Research Reviews, 31, 6-41.
49. Iriki, A., Tanaka, A., and Iwamura, Y. (1996). Coding of modified schema during tool use by macaque postcentral neurones. Cognitive Neuroscience and Neuropsychology, 7 (14), 2325-2330.
50. Jaeger, H. (2001). The Echo State Approach to Analyzing and Training Recurrent Neural Networks. Technical Report, GMD Report 148. Bonn, Germany: German National Research Institute for Computer Science (GMD).
51. Jaeger, H. and Haas, H. (2004). Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science, 304 (5667), 78-80.
52. Joel, D., Niv, Y., and Ruppin, E. (2002). Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Networks, 15, 535-547.
53. Jordan, M. and Jacobs, R. (1994). Hierarchical mixtures of experts and the EM algorithm. Neural Computation, 6 (2), 181-214.
54. Kagan, J. (1972). Motives and development. Journal of Personality and Social Psychology, 22, 51-66.
55. Kakade, S. and Dayan, P. (2002). Dopamine: generalization and bonuses. Neural Networks, 15, 549-559.
56. Kaplan, F. (2004). Who is afraid of the humanoid? Investigating cultural differences in the acceptance of robots. International Journal of Humanoid Robotics, 1 (3), 465-480.
57. Kaplan, F. (2005). Les machines apprivoisees: comprendre les robots de loisir. Paris: Vuibert, Collections Automates Intelligents.
58. Kaplan, F. and Oudeyer, P.-Y. (2003). Motivational principles for visual know-how development. In Prince, C. G., Berthouze, L., Kozima, H., et al. (eds.), Proceedings of the 3rd International Workshop on Epigenetic Robotics: Modeling Cognitive Development in Robotic Systems. Lund University Cognitive Studies 101. Lund, Sweden: Lund University, pp. 73-80.
59. Kaplan, F. and Oudeyer, P.-Y. (2006). Trends in epigenetic robotics: Atlas 2006. In Kaplan, F., Oudeyer, P.-Y., Revel, A., et al. (eds.), Proceedings of the Sixth International Workshop on Epigenetic Robotics: Modeling Cognitive Development in Robotic Systems. Lund University Cognitive Studies 128. Lund, Sweden: Lund University.
60. Kaplan, F. and Oudeyer, P.-Y. (2007a). In search of the neural circuits of intrinsic motivation. Frontiers in Neuroscience, 1 (1), 225-236.
61. Kaplan, F. and Oudeyer, P.-Y. (2007b). The progress-drive hypothesis: an interpretation of early imitation. In Nehaniv, C. and Dautenhahn, K. (eds.), Models and Mechanisms of Imitation and Social Learning: Behavioural, Social and Communication Dimensions. Cambridge, UK: Cambridge University Press, pp. 361-377.
62. Kaplan, F. and Oudeyer, P.-Y. (2007c). Un robot motivé pour apprendre: le rôle des motivations intrinsèques dans le développement sensorimoteur. Enfance, 59 (1), 46-58.
63. Kaplan, F. and Oudeyer, P.-Y. (2008). Le corps comme variable expérimentale. La Revue Philosophique, 2008 (3), 287-298.
64. Kaplan, F. and Oudeyer, P.-Y. (2009). Stable kernels and fluid body envelopes. SICE Journal of Control, Measurement, and System Integration, 48 (1).
65. Kaplan, F., d’Esposito, M., and Oudeyer, P.-Y. (2006) (October). AIBO's Playroom. Online: http://aibo.playroom.fr.
66. Kaplan, F., Oudeyer, P.-Y., and Bergen, B. (2007). Computational models in the debate over language learnability. Infant and Child Development, 17 (1), 55-80.
67. Kawato, M. (1999). Internal models for motor control and trajectory planning. Current Opinion in Neurobiology, 9, 718-727.
68. Khamassi, M., Lachèze, L., Girard, B., Berthoz, A., and Guillot, A. (2005). Actor-critic models of reinforcement learning in the basal ganglia. Adaptive Behavior, 13 (2), 131-148.
69. Lakoff, G. and Johnson, M. (1998). Philosophy in the Flesh: The Embodied Mind and its Challenge to Western Thought. New York: Basic Books.
70. Lakoff, G., and Nunez, R. (2001). Where Mathematics Comes From: How the Embodied Mind Brings Mathematics Into Being. New York: Basic Books.
71. Lungarella, M., Metta, G., Pfeifer, R., and Sandini, G. (2003). Developmental robotics: a survey. Connection Science, 15 (4), 151-190.
72. Maas, W., Natschlager, T., and Markram, H. (2002). Real-time computing without stable states: a new framework for neural computation based on perturbations. Neural Computation, 14 (11), 2531-2560.
73. Marshall, J., Blank, D., and Meeden, L. (2004). An emergent framework for self-motivation in developmental robotics. In Proceedings of the 3rd International Conference on Development and Learning (ICDL 2004).
74. McClure, S., Daw, N. D., and Montague, P. R. (2003). A computational substate for incentive salience. Trends in Neurosciences, 26 (8), 423-428.
75. Meltzoff, A. and Gopnick, A. (1993). The role of imitation in understanding persons and developing a theory of mind. In Baron-Cohen, S., Tager-Flusberg, H., and Cohen, D. (eds.), Understanding Other Minds. Oxford, UK: Oxford University Press, pp. 335-366.
76. Merleau-Ponty, M. (1942). La structure du comportement. Paris: Presses Universitaires de France.
77. Merleau-Ponty, M. (1945). Phénoménologie de la Perception. Paris: Gallimard.
78. Montague, P. R., Dayan, P., and Sejnowski, T. J. (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning. Journal of Neuroscience, 16, 1936-1947.
79. Montgomery, K. C. (1954). The role of exploratory drive in learning. Journal of Comparative and Physiological Psychology, 47, 60-64.
80. Moore, C. and Corkum, V. (1994). Social understanding at the end of the first year of life. Developmental Review, 14, 349-372.
81. Mountcastle, V. (1978). An organizing principle for cerebral function: the unit model and the distributed system. In Edelman, G. and Mountcastle, V. (eds.), The Mindful Brain. Cambridge, MA: MIT Press, pp. 17-49.
82. Niv, Y., Daw, N. D., Joel, D., and Dayan, P. (2006). Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology, 507-520.
83. Oades, R. D. (1985). The role of noradrenaline in tuning and dopamine in switching between signals in the CNS. Neuroscience Biobehavorial Review, 9, 261-282.
84. Oudeyer, P.-Y. and Kaplan, F. (2006). Discovering communication. Connection Science, 18 (2), 189-206.
85. Oudeyer, P.-Y. and Kaplan, F. (2007). What is intrinsic motivation? A typology of computational approaches. Frontiers in Neurorobotics, 1 (1), doi:10.3389/neuro.12.006.2007.
86. Oudeyer, P.-Y., Kaplan, F., Hafner, V. V., and Whyte, A. (2005). The playground experiment: taskindependent development of a curious robot. In Bank, D. and Meeden, L. (eds.), Proceedings of the AAAI Spring Symposium on Developmental Robotics, 2005, pp. 42-47.
87. Oudeyer, P.-Y., Kaplan, F., and Hafner, V. (2007). Intrinsic motivation systems for autonomous mental development. IEEE Transactions on Evolutionary Computation, 11 (1), 265-286.
88. Panksepp, J. (1998). Affective Neuroscience: The Foundations of Human and Animal Emotions. New York: Oxford University Press.
89. Pettit, H. O. and Justice, J. B., Jr. (1989). Dopamine in the nucleus accumbens during cocaine self-administration as studied by in vivo microdialysis. Pharmacology Biochemistry and Behavior, 34, 899-904.
90. Pfeifer, R. and Bongard, J. (2007). How the Body Shapes the Way We Think: A New View of Intelligence. Cambridge, MA: MIT Press.
91. Pfeifer, R. and Scheier, C. (1999). Understanding Intelligence. Boston, MA: MIT Press.
92. Piaget, J. (1952). The Origins of Intelligence in Children. New York: Norton Press.
93. Piaget, J. (1962). Play, Dreams and Imitation in Childhood. New York: Norton Press.
94. Ploghaus, A., Tracey, I., Clare, S., et al. (2000). Learning about pain: the neural substrate of the prediction error of aversive events. PNAS, 97, 9281-9286.
95. Quaade, F., Vaernet, K., and Larsson, S. (1974). Stereotaxic stimulation and electrocoagulation of the lateral hypothalamus in obese humans. Acta Neurochirurgica, 30, 111-117.
96. Redgrave, P., Prescott, T., and Gurney, K. (1999). Is the short latency dopamine response too short to signal reward error? Trends in Neurosciences, 22, 146-151.
97. Ring, M. (1994). Continual learning in reinforcement environments. Ph.D. Thesis, University of Texas at Austin.
98. Rochat, P. (2002). Ego function of early imitation. In Melzoff, A. and Prinz, W. (eds.), The Imitative Mind: Development, Evolution and Brain Bases. Cambridge, UK: Cambridge University Press.
99. Rolls, E. T. (1999). The Brain and Emotion. Oxford, UK: Oxford University Press.
100. Roth, G. and Dicke, U. (2005). Evolution of the brain and intelligence. Trends in Cognitive Sciences, 9 (5), 250-257.
101. Ryan, R. and Deci, E.L. (2000). Intrinsic and extrinsic motivations: classic definitions and new directions. Contemporary Educational Psychology, 25, 54-67.
102. Schaffer, H. (1977). Early interactive development in studies of mother-infant interaction. In Proceedings of Loch Lomonds Symposium. New York: Academic Press, pp. 3-18.
103. Schilder, P. (1935). L’image du corps. 1968 edn., Paris: Gallimard.
104. Schmidhuber, J. (1991). Curious model-building control systems. In Proceedings of the IEEE International Joint Conference on Neural Networks, Vol. 2, pp. 1458-1463.
105. Schmidhuber, J. (1992). Learning complex, extended sequences using the principle of history compression. Neural Computation, 4 (2), 234-242.
106. Schmidhuber, J. (2006). Optimal artificial curiosity, developmental robotics, creativity, music, and the fine arts. Connection Science, 18 (2), 173-187.
107. Schultz, W. (1998). Predictive reward signal of dopamine neurons. Journal of Neurophysiology, 80, 1-27.
108. Schultz, W. (2006). Behavioral theories and the neurophysiology of reward. Annual Review of Psychology, 57, 87-115.
109. Schultz, W., Dayan, P., and Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275, 1593-1599.
110. Shepherd, G. M. (1988). A basic circuit for cortical organization. In Gazzaniga, M. (ed.), Perspectives in Memory Research. Cambridge, MA: MIT Press, pp. 93-134.
111. Smith, A., Li, M., Becker, S., and Kapur, S. (2006). Dopamine, prediction error and associative learning: a model-based account. Network: Computation in Neural Systems, 17, 61-84.
112. Steels, L. (1994). The artificial life roots of artificial intelligence. Artificial Life Journal, 1 (1), 89-125.
113. Steels, L. (2003). Intelligence with representation. Philosophical Transactions of the Royal Society A, 361 (1811), 2381-2395.
114. Steels, L. (2004). The autotelic principle. In Fumiya, I., Pfeifer, R., Steels, L., and Kunyoshi, K. (eds.), Embodied Artificial Intelligence. Lecture Notes in AI 3139. Berlin: Springer Verlag, pp. 231-242.
115. Stellar, J. R. (1985). The Neurobiology of Motivation and Reward. New York: Springer Verlag.
116. Storck, J., Hochreiter, S., and Schmidhuber, J. (1995). Reinforcement driven information acquisition in non-deterministic environments. In Proceedings of ICANN 1995: International Conference on Artificial Neural Networks, Vol. 2. Paris: EC2, pp. 159-164.
117. Suri, R. E. and Schultz, W. (2001). Temporal difference model reproduces anticipatory neural activity. Neural Computation, 13, 841-862.
118. Sutton, R. S. (1988). Learning to predict by the methods of temporal differences. Machine Learning, 3 (1), 9-44.
119. Sutton, R. S. and Barto, A. G. (1998). Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press.
120. Sutton, R. S., Precup, D., and Singh, S. (1999). Between MDPSs and semi-MDPs: a framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112, 181-211.
121. Tani, J. (2007). On the interactions between top-down anticipation and bottom-up regression. Frontiers in Neurorobotics, 1.
122. Tani, J. and Nolfi, S. (1999). Learning to perceive the world as articulated: an approach for hierarchical learning in sensory-motor systems. Neural Network, 12, 1131-1141.
123. Thorndike, E. L. (1911). Animal Intelligence: Experimental Studies. New York: Macmillan.
124. Turing, A. (1950). Computing machinery and intelligence. Mind, 59, 433-460.
125. Varela, F. J., Thompson, E., and Rosch, E. (1991). The Embodied Mind: Cognitive Science And Human Experience. Cambridge, MA: MIT Press.
126. von Uexküll, J. (1909). Umwelt und Innenwelt der Tiere. Berlin: Springer.
127. Vygotsky, L. (1978). Mind in Society: The Development of Higher Psychological Processes. Cambridge, MA: Harvard University Press.
128. Walkins, C. J. C. H. and Dayan, P. (1992). Q-learning. Machine Learning, 8, 279-292.
129. Warnier, J-P. (1999). Construire la culture materielle: l’homme qui pensait avec les doigts. Paris: Presses Universitaires de France.
130. Weinberger, D. R. (1987). Implications of normal brain development for the pathogenesis of schizophrenia. Archives in General Psychiatry, 44, 660-669.
131. Weiner, I. and Joel, D. (2002). Dopamine in schizophrenia: dysfunctional information processing in basal ganglia-thalamocortical split circuits. In Chiara, G. D. (ed.), Handbook of Experimental Pharmacology. Vol 154/II: Dopamine in the CNS II. Berlin: Springer, pp. 417-472.
132. White, N. M. (1989). Reward or reinforcement: what's the difference? Neuroscience Biobehavorial Review, 13, 181-186.
133. White, R. N. (1959). Motivation reconsidered: the concept of competence. Psychological Review, 66, 297-333.
134. Wiering, M. and Schmidhuber, J. (1997). HQ-learning. Adaptive Behavior, 6 (2), 219-246.
135. Wise, R. A. (1989). The brain and reward. In Liebman, J. M. and Cooper, S. J. (eds.), The Neuropharmacological Basis of Reward. Oxford, UK: Clarendon Press, pp. 377-424.
136. Yoshimoto, K., McBride, W.J., Lumeng, L., and Li, T.-K. (1991). Alcohol stimulates the release of dopamine and serotonin in the nucleus accumbens. Alcohol, 9, 17-22.