A biologically inspired hierarchical reinforcement learning system

Cybernetics and Systems

Volumn 36, Issue 1, 2005, Pages 1-44

  Zhou, Weidong ^a   Coggins, Richard ^a  

^a UNIVERSITY OF SYDNEY   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

FUNCTIONS; HIERARCHICAL SYSTEMS; LEARNING ALGORITHMS; NEUROLOGY; PROBLEM SOLVING; PUBLIC POLICY;

ARTIFICIAL EMOTION INDICATION (AEI); HIERARCHICAL REINFORCEMENT LEARNING (HRL); LEARNING PROCESSES; SUBTASKS;

LEARNING SYSTEMS;

EID: 13844298400     PISSN: 01969722     EISSN: None     Source Type: Journal    
DOI: 10.1080/01969720590887270     Document Type: Article

References (52)

1. Balkenius, C. and J. Moren. 2001. Emotional learning: A computational model of the amygdala. Cybernetics and Systems, 32:611-636.
2. Barto, A. G. 1995. Adaptive critics and the basal ganglia. In Models of information processing in the basal ganglia, pp. 215-232, edited by J. L. D. J. C. H. a. D. G. Beiser. Cambridge, MA: MIT Press.
3. Baxter, M. G., A. Parker, C. C. Lindner, A. D. Izquierdo, and E. A. Murray. 2000. Control of response selection by reinforcer value requires interaction of amygdala and orbital prefrontal cortex. Journal of Neuroscience, 20:4311-4319.
4. Bechara, A., H. Damasio, and A. R. Damasio. 2000. Emotion, decision making and the orbitofrontal cortex. Cerebral Cortex, 10:295-307.
5. Bozinovski, S. 2001. Self-learning agents: A connectionist theory of emotion based on crossbar value judgment. Cybernetics and Systems, 32:637-669.
6. Bradtke, S. J. and M. O. Duff. 1995. Reinforcement learning methods for continuous-time Markov decision problems. Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press.
7. Cavada, C., T. Compañy, J. Tejedor, R. J. Cruz-Rizzolo, and F. Reinoso-Suárez. 2000. The anatomical connections of the macaque monkey orbitofrontal cortex, A review. Cerebral Cortex, 10:220-242.
8. Damasio, A. 1999. The feeling of what happens: Body & emotion in the making of consciousness. London: Vintage.
9. Dayan, P. and G. Hinton. 1993. Feudal reinforcement learning. Advances in Neural Information Processing Systems. San Mateo, CA: Morgan Kaufmann.
10. Dickinson, A. 1980. Contemporary animal learning theory. New York: Cambridge University Press.
11. Dietterich, T. G. 2000a. Hierarchical reinforcement learning with the MAXQ value function decomposition. Journal of Artificial Intelligence Research, 13:227-303.
12. Dietterich, T. G. 2000b. State abstraction in MAXQ hierarchical reinforcement learning. Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press.
13. Everitt, B. J., R. N. Cardinal, J. Hall, J. A. Parkinson, and T. W. Robbind. 2000. Differential involvement of amygdala subsystems in appetitive conditioning and drug addiction. In The amygdala: A functional analysis, pp. 253-390, edited by J. P. Aggleton. NewYork: Oxford University Press.
14. Fudge, J. L. and S. N. Haber. 2000. The central nucleus of the amygdala projection to dopamine subpopulations in primates. Neuroscience, 97(3):479-494.
15. Gadanho, S. C. and J. Hallam. 2001. Emotion-triggered learning in autonomous robot control. Cybernetics and Systems, 32(5):531-559.
16. Guarraci, F. A., R. J. Frohardt, and B. S. Kapp. 1999. Amygdaloid Dl dopamine receptor involvement in Pavlovian fear conditioning. Brain Research, 827:28-40.
17. Hatfield, T., J.-S. Han, M. Conley, M. Gallagher, and P. Holland. 1996. Neurotoxic lesions of basolateral, but not central, amygdala interfere with pavlovian second-order conditioning and reinforcer devaluation effects. Journal of Neuroscience, 16:5256-5265.
18. Hengst, B. 2002. Discovering hierarchy in reinforcement learning with HEXQ. Nineteenth International Conference on Machine Learning, Sydney, Australia, July, pp. 8-12.
19. Houk, J. C., J. L. Adams, and A. G. Barto. 1995. A model of how the basal ganglia generate and use neural signals that predict reinforcement. In Models of information processing in the basal ganglia, pp. 249-270, edited by J. L. D. J. C. H. a. D. G. Beiser. Cambridge, MA: MIT Press.
20. Kaelbling, L. P. 1993. Hierarchical learning in stochastic domains.Proceedings of the Tenth International Conference on Machine Learning, Amherst, MA, USA, 27-29 June, pp. 167-173.
21. Killcross, S., T. W. Robbins, and B. J. Everitt. 1997. Different types of fear-conditioned behavior mediated by separate nuclei within amygdala. Nature Neuroscience, 388:377-380.
22. LeDoux, J. E. 1992. Brain mechanisms of emotion and emotional learning. Current Opinion in Neurobiology, 2:191-197.
23. LeDoux, J. E. 2000. Orbitofrontal cortex and basolateral amygdala encode expected outcomes during learning. Annual Review of Neuroscience, 23:155-184.
24. Lin, L.-J. 1992. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning, 8:293-321,
25. Lin, L.-J. 1993. Scaling up reinforcement learning for robot control. Tenth International conference on Machine Learning, Amherst, MA, USA, 27-29 June, pp. 182-189.
26. Malkova, L., D. Gaffan, and E. A. Murray. 1997. Excitotoxic lesions of the amygdala fail to produce impairment in visual learning for auditory secondary reinforcement but interfere with reinforcer devaluation effects in rhesus monkeys. Journal of Neuroscience, 17:6011-6020.
27. Menache, I., S. Mannor, and N. Shimkin. 2002. Q-Cut - Dynamic discovery of sub-goals in reinforcement learning. 13th European Conference on Machine Learning, Helsinki, Finland. Berlin, Heidelberg: Springer-Verlag.
28. Minsky, M. L. 1954. Theory of neural-analog reinforcement systems and its application to the brain-model problem. Ph.D. dissertation, Princeton University, Department of Mathematics.
29. O'Doherty, J. K., M. L., Kringelbach, E. T. Rolls, J. Hornak, and C. Andrews. 2001. Abstract reward and punishment representations in the human orbitofrontal cortex. Nature Neuroscience, 4(1):95-102.
30. Paré, D. and D. R. Collins. 2000. Neuronal correlates of fear in the lateral amygdala: Multiple extracellular recordings in conscious cats. Journal of Neuroscience, 20:2701-2710.
31. Parr, R. and S. Russell. 1998. Reinforcement learning with hierarchical machines. Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press.
32. Pitkanen, A. 2000. Connectivity of the rat amygdaloid complex. In The amygdala: A functional analysis, pp. 31-115, edited by J. P. Aggleton. New York: Oxford University Press.
33. Rogan, M. T. and J. E. LeDoux. 1995. LTP is accompanied by commensurate enhancement of auditory-evoked responses in a fear conditioning circuit. Neuron, 15:127-136.
34. Rogers, R. D., A. M. Owen, H. C. Middleton, E. J. Williams, J. D. Pickard, B. J. Sahakian, and T. W. Robbins. 1999. Choosing between small, likely rewards and large, unlikely rewards activates inferior and orbital prefrontal cortex. Journal of Neuroscience, 20(19):9029-9038.
35. Rolls, E. T. 1999. The brain and emotion. New York: Oxford University Press.
36. Rolls, E. T. 2000. Orbitofrontal cortex and reward. Cerebral Cortex, 10:284-294.
37. Sadikot, A. F. and A. Parent. 1990. The monaminergic innervation of the amygdala in squirrel monkey brain: An immunohischemical study. Experimental Brain Research 81:443-446.
38. Schoenbaum, G., A. A. Chiba, and M. Gallagher. 1998. Orbitofrontal cortex and basolateral amygdala encode expected outcome during learning. Nature Neuroscience, 1:155-159.
39. Schoenbaum, G., A. A. Chiba, and M. Gallagher. 1999. Neural encoding in orbitofrontal cortex and basolateral amygdala during olfactory discrimination learning. Journal of Neuroscience, 19(5):1876-1884.
40. Schultz, W. 1998. Predective reward signal of dopamine neurons. Journal of Neurophysiology, 80:1-27.
41. Schultz, W., P. Dayan, and P. R. Montague. 1997. A neural substrate of prediction and reward. Science, 275:1593-1599.
42. Schultz, W. L., L. Tremblay, and J. R. Hollerman. 2000. Reward processing in primate orbitofrontal cortex and basla ganglia. Cerebral Cortex, 10:272-283.
43. Shelton, C. R. 2001. Balancing multiple sources of reward in reinforcement learning. Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press.
44. Simon, H. A. 1967. Motivational and emotional control of cognition. Psychological Review, 74:29-39.
45. Singh, S. P. 1992. Transfer of learning by composing solutions of elemental sequential tasks. Machine Learning, 8:323-339.
46. Sutton, R. S. 1988. Learning to predict by the methods of temporal differences. Machine Learning, 3:9-44.
47. Sutton, R. S. and A. G. Barto. 1981. Toward a modern theory of adaptive networks: Expectation and prediction. Psychological Review, 88:135-140.
48. Sutton, R. S., S. Singh, D. Precup, and B. Ravindran. 1998. Improved switching among temporally abstract actions. Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press.
49. Sutton, R. S., D. Precup, and S. Singh. 1999. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1):181-211.
50. Tremblay, L. and W. Schultz. 1999. Relative reward preference in primate orbitofrontal cortex. Nature Neuroscience, 398:704-708.
51. Tremblay, L. and W. Schultz. 2000. Reward-related neuronal activity during gonogo task performance in primate orbitofrontal cortex. Journal of Neurophysiology, 83:1864-1876.
52. Valesquez, J. D. 1998. When robots weep: Emotional memories and decision-making. AAAI'98. Madison, WI: MIT Press.