Self-organizing sensorimotor maps plus internal motivations yield animal-like behavior

Adaptive Behavior

Volumn 18, Issue 3, 2010, Pages 315-337

  Butz, Martin V ^a   Shirinov, Elshad ^a   Reif, Kevin L ^a  

^a UNIVERSITY OF WÜRZBURG   (Germany)

Author keywords

cognitive maps; latent learning; motivations; self organizing neural networks; sensorimotor learning

Indexed keywords

ANIMALIA; RATTUS;

EID: 77955720910     PISSN: 10597123     EISSN: 17412633     Source Type: Journal    
DOI: 10.1177/1059712310376842     Document Type: Article

References (85)

1. Arbib, M., Bonaiuto, J., Jacobs, S., & Frey, S. (2009). Tool use and the distalization of the end-effector. Psychological Research, 73(4), 441-462.
2. Arleo, A. & Gerstner, W. (2000). Spatial cognition and neuro-mimetic navigation: A model of hippocampal place cell activity. Biological Cybernetics, 83, 287-299.
3. Ay, N., Bertschinger, N., Der, R., Guttler, F., & Olbrich, E. (2008). Predictive information and explorative behavior of autonomous robots. The European Physical Journal B - Condensed Matter, 63, 329-339.
4. Bertsekas, D. (1989). Parallel and distributed computation: Numerical methods. Englewood Cliffs, NJ: Prentice-Hall International.
5. Bishop, C.M., Hinton, G.E., & Strachan, I.G.D. (1997). GTM through time.Proceedings of the IEEE 5th International Conference on Artificial Neural Networks (pp. 111-116).
6. Bonarini, A., Lazaric, A., & Restelli, M. (2006). Incremental skill acquisition for self-motivated learning animats. From Animals to Animats, 9 (pp. 357-368).
7. Brooks, R.A. (1991). Intelligence without representation. Artificial Intelligence, 47, 139-159.
8. Burgess, N., Jackson, A., Hartley, T., & O'Keefe, J. (2000). Predictions derived from modelling the hippocampal role in navigation. Biological Cybernetics, 83, 301-312.
9. Burgess, N., Recce, M., & O'Keefe, J. (1994). A model of hippocampal function. Neural Networks, 7, 1065-1081.
10. Butz, M.V. (2002). Biasing exploration in an anticipatory learning classifier system. In P. L. Lanzi, W. Stolzmann, & S. W. Wilson (Eds.), Advances in Learning Classifier Systems: 4th International Workshop, IWLCS 2001, LNAI 2321 (pp. 3-22). Berlin Heidelberg: Springer-Verlag.
11. Butz, M.V. & Reif, K. (2010). Motivated TGNG: Algorithm and performance evaluations. Technical Report No. CoboslabY2010N001. Würzburg, Germany: COBOSLAB, Department of Psychology, University of Würzburg. (http://www.coboslab.psychologie.uni-wuerzburg.de).
12. Butz, M.V., Reif, K., & Herbort, O. (2008). Bridging the gap: Learning sensorimotor-linked population codes for planning and motor control. International Conference on Cognitive Systems, CogSys 2008.
13. Butz, M.V., Swarup, S., & Goldberg, D.E. (2004). Effective online detection of task-independent landmarks. IlliGAL Report No. 2004002. University of Illinois at Urbana-Champaign: Illinois Genetic Algorithms Laboratory.
14. Cañamero, L.D. (1997). Modeling motivations and emotions as a basis for intelligent behavior. In W. L. Johnson (Ed.), Proceedings of the 1st International Symposium on Autonomous Agents (Agents'97) (pp. 148-155). New York: The ACM Press.
15. Capdepuy, P., Polani, D., & Nehaniv, C.L. (2007). Construction of an internal predictive model by event anticipation. In M. V. Butz, O. Sigaud, G. Pezzulo, & G. Baldassarre (Eds.), Anticipatory behavior in adaptive learning systems: From brains to individual and social behavior (pp. 218-232). Berlin Heidelberg: Springer-Verlag.
16. Carpenter, G.A., Grossberg, S., Markuzon, N., Reynolds, J.H., & Rosen, D.B. (1992). Fuzzy ARTMAP: A neural network architecture for incremental supervised learning of analog multidimensional maps. IEEE Transactions on Neural Networks, 3(5), 698-713.
17. Crabbe, F.L. (2007). Compromise strategies for action selection. Philosophical Transactions of the Royal Society B: Biological Sciences, 362(1485), 1559-1571.
18. Cuperlier, N., Quoy, M., Giovannangeli, C., Gaussier, P., & Laroque, P. (2006). Transition cells for navigation and planning in an unknown environment. From Animals to Animats, 9 (pp. 286-297).
19. Der, R. & Martius, G. (2006). From motor babbling to purposive actions: Emerging self-exploration in a dynamical systems approach to early robot development. From Animals to Animats, 9 (pp. 406-421).
20. Duckett, T., Marsland, S., & Shapiro, J. (2002). Fast, on-line learning of globally consistent maps. Autonomous Robots, 12(3), 287-300.
21. Dudek, G., Jenkin, M., Milios, E., & Wilkes, D. (1991). Robotic exploration as graph construction. IEEE Transactions on Robotics and Automation, 7, 859-865.
22. Etienne, A.S. & Jeffery, K.J. (2004). Path integration in mammals. Hippocampus, 14(2), 180-192.
23. Fellous, J.-M. & Arbib, M.A. (2005). Who needs emotions: The brain meets the robot. Oxford, GB: Oxford University Press.
24. Fleischer, J.G. (2007). Neural correlates of anticipation in cerebellum, basal ganglia, and hippocampus. In M. V. Butz, O. Sigaud, G. Pezzulo, & G. Baldassarre (Eds.), Anticipatory behavior in adaptive learning systems: From brains to individual and social behavior (pp. 19-34). Berlin Heidelberg: Springer-Verlag.
25. Foster, D.J., Morris, R.G.M., & Dayan, P. (2000). A model of hippocampally dependent navigation, using the temporal difference learning rule. Hippocampus, 10(1), 1-16.
26. Foster, D.J. & Wilson, M.A. (2006). Reverse replay of behavioural sequences in hippocampal place cells during the awake state. Nature, 440, 680-683.
27. Franzius, M., Vollgraf, R., & Wiskott, L. (2007). From grids to places.Journal of Computational Neuroscience, 22(3), 297-299.
28. Fritzke, B. (1995). A growing neural gas network learns topologies. In G. Tesauro, D. S. Touretzky, & T. K. Leen (Eds.), Advances in neural information processing systems 7 (pp. 625-632). Cambridge MA: MIT Press.
29. Gadanho, S.C. & Hallam, J. (2001). Robot learning driven by emotions. Adaptive Behavior, 9, 42-64.
30. Gaussier, P., Revel, A., Banquet, J.P., & Babeau, V. (2002). From view cells and place cells to cognitive map learning: Processing stages of the hippocampal system. Biological Cybernetics, 86(1), 15-28.
31. Greenwald, A. (1970). Sensory feedback mechanisms in performance control: With special reference to the ideo-motor mechanism. Psychological Review, 77, 73-99.
32. Herbart, J.F. (1825). Psychologie als Wissenschaft neu gegründet auf Erfahrung, Metaphysik und Mathematik. Zweiter, analytischer Teil. Königsberg, Germany: August Wilhem Unzer.
33. Hoffmann, J., Berner, M., Butz, M.V., Herbort, O., Kiesel, A., Kunde, W.et al.(2007). Explorations of anticipatory behavioral control (ABC): A report from the cognitive psychology unit of the University of Würzburg. Cognitive Processing, 8, 133-142.
34. Hok, V., Save, E., Lenck-Santini, P.P., & Poucet, B. (2005). Coding for spatial goals in the prelimbic/infralimbic area of the rat frontal cortex.Proceedings of the National Academy of Sciences of the USA, 102(12), 4602-4607.
35. Hommel, B., Müsseler, J., Aschersleben, G., & Prinz, W. (2001). The theory of event coding (TEC): A framework for perception and action planning. Behavioral and Brain Sciences, 24, 849-878.
36. Hsiao, K.Y. & Roy, D. (2005). A habit system for an interactive robot. In C. Castelfranchi, C. Balkenius, M. V. Butz, & A. Ortony (Eds.), From reactive to anticipatory cognitive embodied systems: Papers from the AAAI fall symposium (pp. 83-90). Menlo Park, CA: AAAI Press.
37. Humphries, M.D., Gurney, K., & Prescott, T.J. (2007). Is there a brainstem substrate for action selection?Philosophical Transactions of the Royal Society B, 362, 1627-1639.
38. Humphrys, M. (1996). Action selection methods using reinforcement learning.From Animals to Animats 4: 4th International Conference on Simulation of Adaptive Behavior, SAB-96 (pp. 135-144).
39. James, W. (1890). The principles of psychology. New York: Dover Publications.
40. Johnson, A. & Redish, A.D. (2007). Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. Journal of Neuroscience, 27(45), 12176-12189.
41. Konczak, J. & Dichgans, J. (1997). The development toward stereotypic arm kinematics during reaching in the first 3 years of life. Experimental Brain Research, 117, 346-354.
42. Konidaris, G.D. & Barto, A. (2006). An adaptive robot motivational system. From Animals to Animats, 9 (pp. 346-356).
43. Konidaris, G.D. & Hayes, G.M. (2005). An architecture for behavior-based reinforcement learning. Adaptive Behavior, 13(1), 5-32.
44. Kortenkamp, D. & Weymouth, T. (1994). Topological mapping for mobile robots using a combination of sonar and vision sensing. In AAAI'94: Proceedings of the 12th National Conference on Artificial Intelligence (Vol. 2, pp. 979-984). Menlo Park, CA, USA: American Association for Artificial Intelligence.
45. Kuipers, B. (1978). Modeling spatial knowledge. Cognitive Science, 2, 129-153.
46. Kuipers, B. & Byun, Y. (1991). A robot exploration and mapping strategy based on a semantic hierarchy of spatial representations. Robotics and Autonomous Systems, 8, 47-63.
47. LaValle, S.M. (2006). Planning algorithms. Cambridge, UK: Cambridge University Press. (Available at http://planning.cs.uiuc.edu/).
48. Lisman, J. & Redish, A. (2009). Prediction, sequences and the hippocampus. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1521), 1193-1201.
49. Maes, P. (1991). A bottom-up mechanism for behavior selection in an artificial creature. Proceedings of the 1st International Conference on Simulation of Adaptive Behavior on From Animals to Animats (pp. 238-246).
50. Maes, P. & Brooks, R.A. (1990). Learning to coordinate behaviors. Proceedings of the AAAI-90 (pp. 796-802).
51. Mataric, M.J. & Brooks, R.A. (1990). Learning a distributed map representation based on navigation behaviors. In Proceedings of the USA-Japan Symposium on Flexible Automata (pp. 499-506). New York: ASME.
52. McNaughton, B.L., Battaglia, F.P., Jensen, O., Moser, E.I., & Moser, M.-B. (2006). Path integration and the neural basis of the cognitive map. Nature Reviews Neuroscience, 7, 663-678.
53. Merrick, K. & Maher, M.L. (2009). Motivated learning from interesting events: Adaptive, multitask learning agents for complex environments. Adaptive Behavior, 17, 7-27.
54. Möller, R. & Schenck, W. (2008). Bootstrapping cognition from behavior-a computerized thought experiment. Cognitive Science, 32(3), 504-542.
55. Moser, E.I., Kropff, E., & Moser, M.-B. (2008). Place cells, grid cells, and the brain's spatial representation system. Annual Review of Neuroscience, 31(1), 69-89.
56. O'Keefe, J. & Nadel, L. (1978). The hippocampus as a cognitive map. Oxford: Clarendon Press.
57. Olsson, L.A. (2006). Information self-structuring for developmental robotics: Organization, adaptation, and integration. Unpublished doctoral dissertation, School of Computer Science, Faculty of Engineering and Information Sciences, University of Hertfordshire.
58. Oudeyer, P.-Y., Kaplan, F., & Hafner, V.V. (2007). Intrinsic motivation systems for autonomous mental development. IEEE Transactions on Evolutionary Computation, 11, 265-286.
59. Pierce, D. & Kuipers, B.J. (1997). Map learning with uninterpreted sensors and effectors. Artificial Intelligence, 92, 169-227.
60. Posner, M.I. & Cohen, Y. (1984). Components of visual orienting. Attention and Performance X: Control of Language Processes (pp. 551-556).
61. Poucet, B., Lenck-Santini, P.P., Hok, V., Save, E., Banquet, J.P., Gaussier, P.et al.(2004). Spatial navigation and hippocampal place cell firing: The problem of goal encoding. Reviews in Neurosciences, 15, 89-107.
62. Prescott, T.J., Redgrave, P., & Gurney, K. (1999). Layered control architectures in robots and vertebrates. Adaptive Behavior, 7(1), 99-127.
63. Remolina, E. & Kuipers, B. (2004). Towards a general theory of topological maps. Artificial Intelligence, 152(1), 47- 104.
64. Renaudineau, S., Poucet, B., & Save, E. (2007). Flexible use of proximal objects and distal cues by hippocampal place cells. Hippocampus, 17(5), 381-395.
65. Rolls, E.T. & Kesner, R.P. (2006). A computational theory of hippocampal function, and empirical tests of the theory. Progress in Neurobiology, 79, 1-48.
66. Saigusa, T., Tero, A., Nakagaki, T., & Kuramoto, Y. (2008). Amoebae anticipate periodic events. Physical Review Letters, 100, 018101.
67. Schill, K., Zetzsche, C., & Hois, J. (2009). A belief-based architecture for scene analysis: From sensorimotor features to knowledge and ontology.Fuzzy Sets and Systems, 160(10), 1507-1516.
68. Schmidhuber, J. (1991). A possibility for implementing curiosity and boredom in model-building neural controllers. Proceedings of the 1st International Conference on Simulation of Adaptive Behavior: From Animals to Animats (pp. 222-227).
69. Seward, J.P. (1949). An experimental analysis of latent learning. Journal of Experimental Psychology, 39, 177-186.
70. Shirinov, E. & Butz, M.V. (2009). Distinction between types of motivations: Emergent behavior with a neural, model-based reinforcement learning system. Proceedings of 2009 IEEE Symposium Series on Artificial Life (ALIFE 2009) (pp. 69-76).
71. Simsek, Ö. & Barto, A.G. (2004). Using relative novelty to identify useful temporal abstractions in reinforcement learning. Proceedings of the 21st International Conference on Machine Learning (ICML 2004) (pp. 751-758).
72. Solstad, T., Boccara, C.N., Kropff, E., Moser, M.-B., & Moser, E.I. (2008). Representation of geometric borders in the entorhinal cortex. Science, 322, 1865-1868.
73. Sprague, N. & Ballard, D.H. (2003). Multiple-goal reinforcement learning with modular sarsa(0). Proceedings of the 18th International Joint Conference on Artificial Intelligence, IJCAI 03 (pp. 1445-1447).
74. Stringer, S.M., Rolls, E.T., Trappenberg, T.P., & de Araujo, I.E.T. (2002). Self-organizing continuous attractor networks and path integration: Two-dimensional models of place cells. Network: Computation in Neural Systems, 13, 429-446.
75. Sutton, R.S. (1990). Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. Proceedings of the 7th International Conference on Machine Learning (pp. 216-224).
76. Sutton, R.S. & Barto, A.G. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
77. Sutton, R.S., Precup, D., & Singh, S. (1999). Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112, 181-211.
78. Thrun, S., Burgard, W., & Fox, D. (2005). Probabilistic robotics. Cambridge, MA: MIT Press.
79. Tolman, E.C. (1932). Purposive behavior in animals and men. New York: Appleton.
80. Toussaint, M. (2004). Learning a world model and planning with a self-organizing, dynamic neural system. In B. S. S. Thrun & L. K. Saul (Eds.), Advances in neural information processing systems (Vol. 16, pp. 929-936). Cambridge, MA: MIT Press.
81. Toussaint, M. (2006). A sensorimotor map: Modulating lateral interactions for anticipation and planning. Neural Computation, 18, 1132-1155.
82. Venturini, G. (1994). Adaptation in dynamic environments through a minimal probability of exploration. From Animals to Animats 3: Proceedings of the 3rd International Conference on Simulation of Adaptive Behavior (pp. 371-381).
83. Wiemer, J.C. (2003). The time-organized map algorithm: Extending the self-organizing map to spatiotemporal signals. Neural Computation, 15(5), 1143-1171.
84. Wiener, S.I., Berthoz, A., & Zugaro, M.B. (2002). Multisensory processing in the elaboration of place and head direction responses by limbic system neurons. Cognitive Brain Research, 14, 75-90.
85. Wolpert, D.M. & Kawato, M. (1998). Multiple paired forward and inverse models for motor control. Neural Networks, 11, 1317-1329.