An action-selection calculus

Adaptive Behavior

Volumn 15, Issue 1, 2007, Pages 73-97

  Witkowski, Mark ^a  

^a IMPERIAL COLLEGE LONDON   (United Kingdom)

Author keywords

Action selection; Anticipation; Behaviorism; Conditioning; Expectancy model; Learning theory

Indexed keywords

EID: 33847319176     PISSN: 10597123     EISSN: 17412633     Source Type: Journal    
DOI: 10.1177/1059712306076254     Document Type: Review

References (88)

1. Agre, P. E. (1995). Computational research on interaction and agency. Artificial Intelligence, 72, 1-52.
2. Baerends, G. P. (1976). The functional organization of behaviour. Animal Behaviour, 24, 726-738.
3. Balkenius, C., & Morén, J. (1998). Computational models of classical conditioning: A comparative study. In Proceedings of the 5th International Conference on Simulation of Adaptive Behavior (SAB-5) (pp. 348-353). Zurich, Switzerland.
4. Barnett, S. A. (1975). The rat: A study in behavior. Chicago: University of Chicago Press.
5. Barto, A. G. (1995). Adaptive critics and the basal ganglia. In J. C. Houk, J. Davis, & D. Beiser (Eds.), Models of information processing in the basal ganglia (pp. 215-232). Cambridge, MA: MIT Press.
6. Barto, A. G., & Sutton, R. S. (1982). Simulation of anticipatory responses in classical conditioning by a neuron-like adaptive element. Behavioral Brain Research, 4, 221-235.
7. Bellman, R. (1957). Dynamic programming. Princeton, NJ: Princeton University Press.
8. Bi, G.-Q., & Poo, M.-M. (2001). Synaptic modification by correlated activity: Hebb's postulate revisited. Annual Review of Neuroscience, 24, 139-166.
9. Bindra, D. (1972). A unified account of classical conditioning and operant training. In A. H. Black & W. F. Prokasy (Eds.), Classical conditioning II: Current theory and research (pp. 453-481). New York: Appleton-Century Crofts.
10. Bindra, D., & Stewart, J. (Eds.). (1966). Motivation. Harmondsworth: Penguin Books.
11. Bishop, C. M. (1995). Neural networks for pattern recognition. Oxford: Clarendon Press.
12. Bjork, D. W. (1993). B. F. Skinner: A life. New York: Basic Books.
13. Blackman, D. (1974). Operant conditioning: An experimental analysis of behaviour. London: Methuen & Co.
14. Boden, M. (2006). Mind as machine: A history of cognitive science. Oxford: Oxford University Press.
15. Bower, G. H., & Hilgard, E. R. (1981). Theories of learning. Englewood Cliffs, NJ: Prentice Hall (citations used refer to the 5th edition).
16. Brooks, R. A. (1986). A robust layered control system for a mobile robot. IEEE Journal of Robotics and Automation, 2, 14-23.
17. Brooks, R. A. (1991a). Intelligence without reason. MIT AI Laboratory, AI Memo No 1293 (prepared for Computers and Thought, International Joint Conference on Artificial Intelligence (IJCAI), preprint, April 1991 ).
18. Brooks, R. A. (1991b). Intelligence without representation, Artificial Intelligence, 47, 139-159.
19. Bryson, J. J. (2000). Hierarchy and sequence vs. full parallelism in action selection. In Proceedings of the 6th International Conference on Simulation of Adaptive Behavior (SAB-6) (pp. 147-156). Paris, France.
20. Bryson, J. J., Prescott, T. J., & Seth, A. K. (Eds.). (2005). Modelling of natural action selection: Proceedings of an International Workshop. Edinburgh, UK. Brighton, UK: Society for the Study of Artificial Intelligence and the Simulation of Behavior (SSAISB).
21. Butz, M. V., Sigaud, O., & Gérard, P. (2003). Internal models and anticipations in adaptive learning systems. In M. V. Butz, O. Sigaud, & P. Gérard (Eds.), Anticipatory behavior in adaptive learning systems, Lecture Notes in Artificial Intelligence (Vol. 2684, pp. 86-109). Berlin: Springer.
22. Catania, A. C. (1988). The operant behaviorism of B. F. Skinner. In A. C. Catania & S. Harnad (Eds.), The selection of behavior (pp. 3-8). Cambridge: Cambridge University Press.
23. Courville, A. C., Daw, N. D., & Touretzky, D. S. (2006). Bayesian theories of conditioning in a changing world. Trends in Cognitive Sciences, 10, 294-300.
24. Dayan, P., & Balleine, B. W. (2002). Reward, motivation, and reinforcement learning. Neuron, 36, 285-298.
25. Hafting, T., Fyhn, M., Molden, S., Moser, M.-B., & Moser, E. I. (2005). Microstructure of a spatial map in the entorhinal cortex. Nature, 436, 801-806.
26. Hallam, B. E., Hallam, J. C. T., & Halperin, J. R. P. (1994). An ethological model for implementation in mobile robots. Adaptive Behavior, 3, 51-79.
27. Hart, P. E., Nilsson, N. J., & Raphael, B. (1968). A formal basis for the heuristic determination of minimum cost paths. IEEE Transactions on Systems Science and Cybernetics, 4, 100-107.
28. Hebb, D. O. (1949). The organization of behavior. New York: Wiley.
29. Hergenhahn, B. R., & Olson, M. H. (2005). An introduction to theories of learning (7th edition). Englewood Cliffs, NJ: Prentice-Hall.
30. Howard, R. A. (1960). Dynamic programming and markov processes. Cambridge, MA: MIT Press.
31. Hubel, D. H., & Wiesel, T. N. (1962). Receptive fields, binocular interaction, and functional architecture in the cat's visual cortex. Journal of Physiology, 160, 106-154.
32. Hull, C. (1943). Principles of behavior. New York: Apple-Century Crofts.
33. Jones, G., & Ritter, F. E. (2003). Production systems and rule-based inference. In L. Nadel (Ed.), Encyclopedia of cognitive science (pp. 741-747). London: Nature Publishing Group.
34. Kaas, J. H. (1997). Topographic maps are fundamental to sensory processing. Brain Research Bulletin, 44, 107-112.
35. Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996). Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4, 237-285.
36. Khamassi, M., Lachèze, L., Girard, B., Berthoz, A., & Guillot, A. (2005). Actor-critic models of reinforcement learning in the basal ganglia: From natural to artificial rats. Adaptive Behavior, 13, 131-148.
37. Kirsh, D. (1991). Today the earwig, tomorrow man ? Artificial Intelligence, 47, 161-184.
38. Klahr, D., Langley, P., & Neches, R. (1987). Production system models of learning and development. Cambridge, MA: MIT Press.
39. Langley, P. (1996). Elements of machine learning. San Francisco, CA: Morgan Kaufmann.
40. Leergaard, T. B., Alloway, K. D., Mutic, J. J., & Bjaalie, J. G. (2000). Three-dimensional topography of corticopontine projections from rat barrel cortex: Correlations with corticostriatal organisation. Journal of Neuroscience, 20, 8474-8484.
41. Lorenz, K. Z. (1950). The comparative method in studying innate behaviour patterns. In Physiological mechanisms in animal behaviour, Symposium of the Society of Experimental Biology (Vol. 4, pp. 221-268). London: Academic Press.
42. MacCorquodale, K., & Meehl, P. E. (1953). Preliminary suggestions as to a formalization of expectancy theory. Psychological Review, 60, 55-63.
43. MacCorquodale, K., & Meehl, P. E. (1954). Edward C. Tolman. In W. K. Estes et al. (Eds.), Modern learning theory: A critical analysis of five examples (pp. 177-266). New York: Appleton-Century Crofts.
44. Maes, P. (1991). A bottom-up mechanism for behavior selection in an artificial creature. In Proceedings of the 1st International Conference on Simulation of Adaptive Behavior (SAB) (pp. 238-246). London, UK.
45. Maslow, A. H. (1987). R. Frager et al. (Eds.), Motivation and personality (3rd edition). New York: Harper and Row (first published 1954).
46. Minai, A. A. (1997). Covariance learning of correlated patterns in competitive networks. Neural Computation, 9, 667-681.
47. Moore, A. W., & Atkeson, C. G. (1993). Prioritized sweeping: Reinforcement learning with less data and less time. Machine Learning, 13, 103-130.
48. Mowrer, O. H. (1956). Two-factor learning theory reconsidered, with special reference to secondary reinforcement and the concept of habit. Psychological Review, 63, 114-128.
49. Nilsson, N. J. (1971). Problem-solving methods in artificial intelligence. New York: McGraw-Hill.
50. Nishikawa, K. C. (2002). Evolutionary convergence in nervous systems: Insights from comparative phylogenetic studies. Brain, Behavior and Evolution, 59, 240-249.
51. Oja, E. (1992). Principal components, minor components, and linear neural networks. Neural Networks, 5, 927-935.
52. O'Keefe, J., & Nadel, L. (1978). The hippocampus as a cognitive map. Oxford: Clarendon Press.
53. Pavlov, I. P. (1927). Conditioned reflexes. Oxford: Oxford University Press (available as Dover Publications facsimile reprint, 1960).
54. Penfield, W., & Rasmussen, T. B. (1950). The cerebral cortex of man. New York: Macmillan.
55. Razran, G. (1971). Mind in evolution: An east-west synthesis of learned behavior and cognition. Boston: Houghton Mifflin.
56. Rescorla, R. A., & Wagner, A. R. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In A. H. Black & W. F. Prokasy (Eds.), Classical conditioning II: Current theory and research (pp. 64-99). New York: Appleton-Century Crofts.
57. Russell, S., & Norvig, P. (2003). Artificial intelligence: A modern approach. Englewood Cliffs, NJ: Prentice Hall (citations used refer to the 2nd edition ).
58. Saksida, L. M., Raymond, S. M., & Touretzky, D. S. (1997). Shaping robot behavior using principles from instrumental conditioning. Robotics and Autonomous Systems, 22, 231-249.
59. Schmajuk, N. A. (1994). Behavioral dynamics of escape and avoidance: A neural network approach. In From animals to animats 3: Proceedings of the 3rd International Conference on Simulation of Adaptive Behavior (SAB-3) (pp. 118-127). Brighton, UK.
60. Schoppers, M. J. (1987). Universal plans for reactive robots in unpredictable environments. In Proceedings of the 10th International Joint Conference on Artificial Intelligence (IJCAI) (pp. 1039-1046). Milan, Italy.
61. Schultz, W. (1998). Predictive reward signal of dopamine neurons. Journal of Neurophysiology, 80, 1-27.
62. Sejnowski, T. J. (1977). Storing covariance with nonlinearly interacting neurons. Journal of Mathematical Biology, 4, 303-321.
63. Sejnowski, T. J. (1999). The book of Hebb. Neuron, 24, 773-776.
64. Sejnowski, T. J., Dayan, P., & Montague, P. R. (1995). Predictive Hebbian learning. In Proceedings of the 8th Annual Workshop on Computational Learning Theory (COLT-95) (pp. 15-18). Santa Cruz, CA.
65. Sewards, T. V., & Sewards, M. A. (2003). Representations of motivational drives in mesial cortex, medial thalamus, hypothalamus and midbrain. Brain Research Bulletin, 61, 25-49.
66. Shettleworth, S. J. (1975). Reinforcement and the organisation of behavior in golden hamsters: Hunger, environment, and food reinforcement. Journal of Experimental Psychology: Animal Behavior Processes, 104, 56-87.
67. Stolzmann, W., Butz, M. V., Hoffmann, J., & Goldberg, D. E. (2000). First cognitive capabilities in the anticipatory classifier system. In Proceedings of the 6th International Conference on Simulation of Adaptive Behavior (SAB-6) (pp. 287-296). Paris, France.
68. Sutton, R. S. (1988). Learning to predict by the methods of temporal differences. Machine Learning, 3, 9-44.
69. Sutton, R. S. (1990). Reinforcement learning architectures for animats. In Proceedings of the First International Conference on Simulation of Adaptive Behavior (SAB'90) (pp. 288-296). London, UK. Cambridge, MA: MIT Press.
70. Sutton, R. S., & Barto, A. G. (1987). A temporal-difference model of classical conditioning. In Proceedings of the 9th Annual Conference of the Cognitive Science Society (pp. 355-378). Seattle, WA. Hillsdale, NJ: Lawrence Erlbaum.
71. Sutton, R. S., & Barto, A. G. (1990). Time-derivative models of Pavlovian reinforcement. In M. Gabriel & J. Moore (Eds.), Learning and computational neuroscience: Foundations of adaptive networks (pp. 497-537). Cambridge, MA: MIT Press.
72. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
73. Thistlethwaite, D. A. (1951). Critical review of latent learning and related experiments. Psychological Bulletin, 48, 97-129.
74. Thorndike, E. L. (1898). Animal intelligence: An experimental study of the associative processes in animals. Psychological Review, Monograph Supplements, 2 ( 8 ).
75. Tinbergen, N. (1951). The study of instinct. Oxford: Clarendon Press.
76. Tolman, E. C. (1932). Purposive behavior in animals and men. New York: Century Co.
77. Tolman, E. C., & Honzik, C. H. (1930). Introduction and removal of reward, and maze performance in rats. University of California Publications in Psychology, 4, 257-275 (cited Bower & Hilgard, 1981, pp. 335-338).
78. Travers, M. (1989). Animal construction kits. In C. Langton (Ed.), Artificial life: SFI studies in sciences of complexity (pp. 421-442). Reading, MA: Addison-Wesley.
79. Tyrrell, T. (1993). Computational mechanisms for action selection. Ph.D. thesis. University of Edinburgh.
80. Tyrrell, T. (1994). An evaluation of Maes's bottom-up mechanism for behavior selection. Adaptive Behavior, 1, 387-420.
81. Velmans, M. (2000). Understanding consciousness. London: Routledge.
82. Vogel, E. H., Castro, M. E., & Saavedra, M. A. (2004). Quantitative models of Pavlovian conditioning. Brain Research Bulletin, 63, 173-202.
83. Watkins, C. J. C. H., & Dayan, P. (1992). Technical Note: Q-learning. Machine Learning, 8, 279-292.
84. Welker, C. (1976). Receptive fields of barrels in the somato-sensory neocortex of the rat. Journal of Comparative Neurology, 166, 173-189.
85. Witkowski, M. (1997). Schemes for learning and behaviour: A new expectancy model. Ph.D. Thesis, University of London.
86. Witkowski, M. (1998). Dynamic expectancy: An approach to behaviour shaping using a new method of reinforcement learning. In Proceedings of the 6th International Symposium on Intelligent Robotic Systems (pp. 73-81). Edinburgh, UK.
87. Witkowski, M. (2000). The role of behavioral extinction in animat action selection. In Proceedings of the 6th International Conference on Simulation of Adaptive Behavior (SAB-6) (pp. 177-186). Paris, France.
88. Witkowski, M. (2003). Towards a four factor theory of anticipatory learning. In M. V. Butz, O. Sigaud, & P. Gérard (Eds.), Anticipatory behavior in adaptive learning systems, Lecture Notes in Artificial Intelligence (Vol. 2684, pp. 66-85). Berlin: Springer.