Combining correlation-based and reward-based learning in neural control for policy improvement

Advances in Complex Systems

Volumn 16, Issue 2-3, 2013, Pages

  Manoonpong, Poramate ^a,b   Kolodziejski, Christoph ^a   Wörgötter, Florentin ^a   Morimoto, Jun ^a,b  

^a UNIVERSITY OF GÖTTINGEN   (Germany)

^b ADVANCED TELECOMMUNICATIONS RESEARCH INSTITUTE INTERNATIONAL   (Japan)

Author keywords

associative learning; Classical conditioning; goal directed behavior; operant conditioning; pole balancing; reinforcement learning

Indexed keywords

EID: 84880376650     PISSN: 02195259     EISSN: None     Source Type: Journal    
DOI: 10.1142/S021952591350015X     Document Type: Article

References (70)

1. Alonso, E., Mondragon, E. and Kjäll-Ohlsson, N., Pavlovian and instrumental Q-learning: A Rescorla and Wagner-based approach to generalization in Q-learning, in Proc. Adaptation in Artificial and Biological Systems (2006), pp. 23-29.
2. Asmuth, J., Littman, M. L. and Zinkov, R., Potential-based shaping in modelbased reinforcement learning, in Proc. 23rd National Conf. Artificial Intelligence (AAAI'08) (2008), pp. 604-609.
3. Bakker, B. and Schmidhuber, J., Hierarchical reinforcement learning based on subgoal discovery and subpolicy specialization, in Proc. 8th Conf. Intelligent Autonomous Systems (2004), pp. 438-445.
4. Banerjee, B. and Kraemer, L., Action discovery for single and multi-agent reinforcement learning, Adv. Complex Syst. 14 (2011) 279-305.
5. Barnard, C., Animal Behavior: Mechanism, Development, Function, and Evolution (Pearson Education, 2004).
6. Barto, A. G. and Mahadevan, S., Recent advances in hierarchical reinforcement learning, Discrete Event Dyn. Syst. 13 (2003) 41-77.
7. Barto, A. G., Sutton, R. S. and Anderson, C. W., Neuron-like adaptive elements that can solve difficult learning control problems, IEEE Trans. Syst. Man, Cybern. 13 (1983) 834-846. (Pubitemid 14138646)
8. Bekey, G., Autonomous Robots From Biological Inspiration to Implementation and Control (MIT Press, Cambridge, 2005).
9. Botvinick, M. M., Niv, Y. and Barto, A. C., Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective, Cognition 113 (2009) 262-280.
10. Bovet, S., Robots with self-developing brains, Ph.D. thesis, University of Zurich (2007).
11. Brembs, B. and Heisenberg, M., The operant and the classical in conditioned orientation in drosophila melanogaster at the flight simulator, Learn. Memory 7 (2000) 104-115. (Pubitemid 30213024)
12. Cetina, V. U., Supervised reinforcement learning using behavior models, in Proc. Sixth Int. Conf. Machine Learning and Applications (ICMLA 2007) (2007), pp. 336-341.
13. Chang, C. and Gaudiano, P., Application of biological learning theories to mobile robot avoidance and approach behaviors, Adv. Complex Syst. 1 (1998) 79-114.
14. Dasgupta, S., Wörgötter, F. and Manoonpong, P., Information theoretic selforganised adaptation in reservoirs for temporal memory tasks, in Proc. 13th Int. Conf. Engineering Applications of Neural Networks (EANN 2012) (2012), pp. 31-40.
15. Dayan, P. and Balleine, B., Reward, motivation, and reinforcement learning, Neuron 36 (2002) 285-298.
16. Devlin, S., Kudenko, D. and Grzes, M., An empirical study of potential-based reward shaping and advice in complex, multi-agent systems, Adv. Complex Syst. 14 (2011) 251-278.
17. Dietterich, T. G., Hierarchical reinforcement learning with the MAXQ value function decomposition, J. Artif. Intell. Res. 13 (2000) 227-303. (Pubitemid 33682087)
18. Doya, K., Efficient nonlinear control with actor-tutor architecture, in Advances in Neural Information Processing Systems (1997), pp. 1012-1018.
19. Doya, K., Reinforcement learning in continuous time and space, Neural Comput. 12 (2000) 219-245.
20. Endo, G., Morimoto, J., Matsubara, T., Nakanish, J. and Cheng, G., Learning CPGbased biped locomotion with a policy gradient method: Application to a humanoid robot, Int. J. Robot. Res. 27 (2008) 213-228. (Pubitemid 351169715)
21. Fischer, J., A Modulatory Learning Rule for Neural Learning and Metalearning in Real World Robots with Many Degees of Freedom (Shaker Verlag GmbH, 2003).
22. Gomez, F., Schmidhuber, J. and Miikkulainen., R., Accelerated neural evolution through cooperatively coevolved synapses, J. Mach. Lear. Res. 9 (2008) 937-965.
23. Gomi, H. and Kawato, M., Neural network control for a closed-loop system using feedback-error-learning, Neural Netw. 6 (1993) 933-946.
24. Gullapalli, V., A stochastic reinforcement learning algorithm for learning real-valued functions, Neural Netw. 3 (1990) 671-692.
25. Howery, L. D., Why do animals behave the way they dö, Backyards and Beyond: Rural Living in Arizona 3 (2007) 17-18.
26. Hull, C. L., A Behavior System: An Introduction to Behavior Theory Concerning the Individual Organism (Yale University Press, New Haven, CT, 1952).
27. Humeau, Y., Shaban, H., Bissiere, S. and Lüthi, A., Presynaptic induction of heterosynaptic associative plasticity in the mammalian brain, Nature 426 (2003) 841-845. (Pubitemid 38056869)
28. Iida, S., Kuwayama, K., Kanoh, M., Kato, S. and Itoh, H., A dynamic allocation method of basis functions in reinforcement learning, in Proc. 17th Australian Joint Conf. Artificial Intelligence (2004), pp. 71-73.
29. Kassahun, Y., de Gea, J., Edgington, M., Metzen, J. H. and Kirchner, F., Accelerating neuroevolutionary methods using a kalman filter, in Proc. 10th Genetic and Evolutionary Computation Conf. (GECCO-2008) (2008), pp. 1397-1404.
30. Kassahun, Y., Wöhrle, H., Fabisch, A. and Tabie, M., Learning parameters of linear models in compressed parameter space, in Proc. Artificial Neural Networks and Machine Learning (ICANN2012) (2012), pp. 108-115.
31. Kawarai, N. and Kobayashi, Y., Learning of whole arm manipulation with constraint of contact mode maintaining, J. Robot. Mechatron. 22 (2010) 542-550.
32. Klopf, A. H., A neuronal model of classical conditioning, Psychobiology 16 (1988) 85-123.
33. Kolter, J. and Ng, A., Policy search via the signed derivative, in Proc. Robotics: Science and Systems (RSS) (2009), pp. 27, Online.
34. Kondo, T. and Ito, K., A reinforcement learning with adaptive state space recruitment strategy for real autonomous mobile robots, in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems (2002), pp. 897-902. (Pubitemid 35437923)
35. Konorski, J., Integrative Activity of the Brain (University of Chicago Press, Chicago, 1967).
36. Koppejan, R. and Whiteson, S., Neuroevolutionary reinforcement learning for generalized helicopter control, in GECCO 2009: Proc. Genetic and Evolutionary Computation Conf. (2009), pp. 145-152.
37. Kosco, B., Differential hebbian learning, in Proc. Neural Networks for Computing: AIP, Vol. 151 (1986), pp. 277-282.
38. Lee, H., Shen, Y., Yu, C., Singh, G. and Ng, A., Quadruped robot obstacle negotiation via reinforcement learning, in Proc. IEEE Int. Conf. Robotics and Automation (2006), pp. 3003-3010. (Pubitemid 44940521)
39. Lovibond, P. F., Facilitation of instrumental behavior by a pavlovian appetitive conditioned stimulus, J. Exp. Psychol. Anim. B 9 (1983) 225-247. (Pubitemid 13022508)
40. Manoonpong, P., Geng, T., Kulvicius, T., Porr, B. and Wörgötter, F., Adaptive, fast walking in a biped robot under neuronal control and learning, PLoS Comput. Biol. 3 (2007) e134.
41. Manoonpong, P., Parlitz, U. and Wörgötter, F., Neural control and adaptive neural forward models for insect-like, energy-efficient, and adaptable locomotion of walking machines, Front. Neural Circuits 7 (2013). Doi: 10.3389/fncir.2013.00012.
42. Manoonpong, P. and Wörgötter, F., Adaptive sensor-driven neural control for learning in walking machines, in Neural Information Processing, LNCS (2009), pp. 47-55.
43. Manoonpong, P., Wörgötter, F. and Morimoto, J., Extraction of reward-related feature space using correlation-based and reward-based learning methods, in Neural Information Processing, LNCS (2010), pp. 414-421.
44. Morihiro, K., Matsui, N., and Nishimura, H., Effects of chaotic exploration on reinforcement maze learning, in Knowledge Based Intelligent Information and Engineering Systems (2004), pp. 833-839. (Pubitemid 39751885)
45. Morimoto, J. and Doya, K., Reinforcement learning of dynamic motor sequence: Learning to stand up, in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems (1998), pp. 1721-1726.
46. Morimoto, J. and Doya, K., Acquisition of stand-up behavior by a real robot using hierarchical reinforcement learning, Robot. Auton. Syst. 36 (2001) 37-51. (Pubitemid 32610235)
47. Mowrer, O., Learning Theory and Behavior (New York, Wiley, 1960).
48. Ng, A. Y., Harada, D. and Russell, S. J., Policy invariance under reward transformations: Theory and application to reward shaping, in Proc. 16th Int. Conf. Machine Learning (1999), pp. 278-287.
49. Pasemann, F., Evolving neurocontrollers for balancing an inverted pendulum, Network-Comp. Neural 9 (1998) 495-511. (Pubitemid 128692543)
50. Pavlov, I., Conditioned Reflexes (Oxford University Press, Oxford, UK, 1927).
51. Phon-Amnuaisuk, S., Learning cooperative behaviours in multiagent reinforcement learning, in Neural Information Processing, LNCS (2009), pp. 570-579.
52. Porr, B. and Wörgötter, F., Strongly improved stability and faster convergence of temporal sequence learning by using input correlations only, Neural Comput. 18 (2006) 1380-1412. (Pubitemid 43765448)
53. Porr, B. and Wörgötter, F., Fast heterosynaptic learning in a robot food retrieval task inspired by the limbic system, Biosystems 89 (2007) 294-299. (Pubitemid 46695861)
54. Price, B. and Boutilier, C., Accelerating reinforcement learning through implicit imitation, J. Artif. Intell. Res. 19 (2003) 569-629. (Pubitemid 41525929)
55. Rescorla, R. and Solomon, R., Two process learning theory: Relationship between pavlovian conditioning and instrumental learning, Psychol. Rev. 88 (1967) 151-182.
56. Rescorla, R. and Wagner, A., A theory of pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement, in Classical Conditioning II : Current Research and Theory (1972) 64-99.
57. Sherstov, A. and Stone, P., Function approximation via tile coding: Automating parameter choice, in Proc. Symp. Abstraction, Reformulation, and Approximation (SARA-05) (2005), pp. 194-205.
58. Shibata, K., Nishino, T. and Okabe, Y., Active perception and recoginition learning system based on Actor-Q architecture, Syst. Comput. Jpn. 33 (2002) 12-22. (Pubitemid 35372556)
59. Skinner, B., The Behavior of Organisms: An Experimental Analysis (Appleton Century Croft, New York, 1938).
60. Smith, S. C. and Herrmann, J. M., Homeokinetic reinforcement learning, in First IAPR TC3 Workshop, PSL 2011, LNCS (2012), pp. 82-91.
61. Suri, R. E., Bargas, J. and Arbib, M. A., Modeling functions of striatal dopamine modulation in learning and planning, Neuroscience 103 (2001) 65-85. (Pubitemid 32304347)
62. Sutton, R. and Barto, A., Reinforcement Learning: An Introduction (MIT Press, Cambridge, 1998).
63. Thorndike, E., Animal intelligence: An experimental study of the associative process in animals, Psychol. Rev. Monogr. Suppl. 8 (1898) 68-72.
64. Tokic, M., Adaptive -greedy exploration in reinforcement learning based on value differences, in Proc. KI 2010: Advances in Artificial Intelligence (2010), pp. 203-210.
65. Touretzky, D. and Saksida, L., Operant conditioning in skinnerbots, Adapt. Behav. 5 (1997) 219-247.
66. van Dijk, S. G. and Polani, D., Grounding subgoals in information transitions, in Proc. IEEE Symp. Adaptive Dynamic Programming and Reinforcement Learning (Paris, France, 2011), pp. 105-111.
67. Watkins, C. J. C. H., Learning from delayed rewards, Ph.D. thesis, University of Cambridge (1989).
68. Williams, D. andWilliams, H., Auto - maintenance in the pigeon: Sustained pecking despite contingent non-reinforcement, J. Exp. Anal. Behav. 12 (1969) 511-520.
69. Woolley, B. G. and Stanley, K. O., Evolving a single scalable controller for an octopus arm with a variable number of segments, in Parallel Problem Solving from Nature, Lecture Notes in Computer Science (2010), pp. 270-279.
70. Wörgötter, F. and Porr, B., Temporal sequence learning, prediction and control - A review of different models and their relation to biological mechanisms, Neural Comp. 17 (2005) 245-319. (Pubitemid 40190477)