Reinforcement learning in continuous time and space: Interference and not Ill conditioning is the main problem when using distributed function approximators

IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics

Volumn 38, Issue 4, 2008, Pages 950-956

  Baddeley, Bart ^a  

^a UNIVERSITY OF SUSSEX   (United Kingdom)

Author keywords

Continuous time systems; Distributed memory systems; Feedforward neural networks; Ill conditioning; Interference

Indexed keywords

EDUCATION; GRADIENT METHODS; LEARNING SYSTEMS; LIGHTNING; MATHEMATICAL MODELS; MATHEMATICAL PROGRAMMING; NEURAL NETWORKS; POLYNOMIAL APPROXIMATION; REINFORCEMENT; REINFORCEMENT LEARNING; SYSTEMS ENGINEERING;

CONTINUOUS TIME SYSTEMS; CONTINUOUS-TIME; DISTRIBUTED ARCHITECTURES; DISTRIBUTED FUNCTIONS; DISTRIBUTED MEMORY SYSTEMS; FEEDFORWARD NEURAL NETWORKS; FUNCTION APPROXIMATIONS; FUNCTION APPROXIMATORS; GRADIENT DESCENT; HIGH-DIMENSIONAL; ILL-CONDITIONING; INTERFERENCE; LEARNING PROCESSES; LOCAL LINEAR MODELS; MLP NETWORKS; MULTI-LAYER PERCEPTRON; NON-LINEAR MODELING; PENDULUM SWING-UP; RL TECHNIQUES; TEMPORAL DIFFERENCE LEARNING; VALUE FUNCTIONS; VALUE-FUNCTION APPROXIMATION;

LEARNING ALGORITHMS;

ALGORITHM; ARTICLE; ARTIFICIAL NEURAL NETWORK; COMPUTER SIMULATION; FEEDBACK SYSTEM; REINFORCEMENT; SYSTEM ANALYSIS; SYSTEMS THEORY; THEORETICAL MODEL;

ALGORITHMS; COMPUTER SIMULATION; FEEDBACK; MODELS, THEORETICAL; NEURAL NETWORKS (COMPUTER); PROGRAMMING, LINEAR; REINFORCEMENT (PSYCHOLOGY); SYSTEMS THEORY;

EID: 49049087720     PISSN: 10834419     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSMCB.2008.921000     Document Type: Article

References (31)

1. M. Abu-Khalaf and F. L. Lewis, "Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach," Automatica, vol. 41, no. 5, pp. 779-791, 2005.
2. S. Amari, "Natural gradient works efficiently in learning," Neural Comput., vol. 10, no. 2, pp. 251-276, Feb. 1998.
3. C. W. Anderson, "Strategy learning with multilayer connectionist representations," in Proc. 4th Int. Workshop Mach. Learn., 1987, pp. 103-114.
4. B. Ans and S. Rousset, "Avoiding catastrophic forgetting by coupling two reverberating neural networks," C. R. Acad. sci., Sér., Sci. vie, vol. 320, no. 12, pp. 989-997, 1997.
5. L. C. Baird, "Advantage updating," Wright Lab., Wright Patterson Air Force Base, Dayton, OH, Tech. Rep. WL-TR-93-1146, 1993.
6. A. R. Barron, "Universal approximation bounds for superpositions of a sigmoidal function," IEEE Trans. Inf. Theory, vol. 39, no. 3, pp. 930-945, May 1993.
7. R. Bellman, I. Glicksberg, and O. Gross, "On the bang-bang control problem," Q. Appl. Math., vol. 14, no. 1, pp. 11-18, 1956.
8. D. P. Bertsekas, Dynamic Programming and Optimal Control. Belmont, MA: Athena Scientific, 1995.
9. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming. Belmont, MA: Athena Scientific, 1996.
10. C. Bishop, Neural Networks for Pattern Recognition. Oxford, U.K.: Oxford Univ. Press, 1995.
11. R. Coulom, "Reinforcement learning using neural networks, with applications to motor control," M.S. thesis, Inst. Nat. Polytech. Grenoble, Grenoble, France, 2002.
12. R. Coulom, "High-accuracy value-function approximation with neural networks applied to the acrobat," in Proc. Eur. Symp. Artif. Neural Netw., M. Verleysen, Ed., Bruges, Belgium, 2004.
13. K. Doya, "Temporal difference learning in continuous time and space," in Advances in Neural Information Processing Systems, vol. 8, D. S Touretzky, M. C Mozer, and M. E Hasselmo, Eds. Cambridge, MA: MIT Press, 1996, pp. 1073-1079.
14. K. Doya, "Reinforcement learning in continuous time and space," Neural Comput., vol. 12, no. 1, pp. 219-245, Jan. 2000.
15. R. M. French, "Pseudo-recurrent connectionist networks: An approach to the 'sensitivity-stability' dilemma," Connect. Sci., vol. 9, no. 4, pp. 353-380, Dec. 1997.
16. G. J. Gordon, "Stable function approximation in dynamic programming," in Proc. 12th Int. Conf. Mach. Learn., A. Prieditis and S. Russel, Eds., San Francisco, CA, 1995.
17. L. P. Kaelbling, M. L. Littman, and A. W. Moore, "Reinforcement learning: A survey," J. Artif. Intell. Res., vol. 4, pp. 237-285, 1996.
18. K. Levenberg, "A method for the solution of certain non-linear problems in least squares," Q. J. Appl. Math., vol. II, no. 2, pp. 164-168, 1944.
19. D. W. Marquardt, "An algorithm for least-squares estimation of nonlinear parameters," J. Soc. Ind. Appl. Math., vol. 11, no. 2, pp. 431-441, Jun. 1963.
20. M. F. Moller, "A scaled conjugate gradient algorithm for fast supervised learning," Neural Netw., vol. 6, no. 4, pp. 525-533, 1993.
21. R. Munos, L. C. Baird, and A. W. Moore, "Gradient descent approaches to neural-net-based solutions of the Hamilton-Jacobi-Bellman equation," in Proc. Int. Joint Conf. Neural Netw., 1999, pp. 2152-2157.
22. R. Neuneier and H.-G. Zimmermann, "How to train neural networks," in Neural Networks: Tricks of the Trade, G. B. Orr and K.-R. Müller, Eds. New York: Springer-Verlag, 1998.
23. J. K. Peterson, "On-line estimation of the optimal value function: HJB-estimates," in Advances in Neural Information Processing Systems vol. 5, C. L. Giles, S. J. Hanson, and J. D. Cowan, Eds. San Mateo, CA: Morgan Kaufmann, 1993, pp. 319-326.
24. M. Riedmiller and H. Braun, "A direct adaptive method for faster backpropagation learning: The RPROP algorithm," in Proc. IEEE Int. Conf. Neural Netw., 1993, pp. 586-591.
25. A. Robins, "Catastrophic forgetting, rehearsal, and pseudorehearsal," Connect. Sci., vol. 7, no. 2, pp. 123-146, Jun. 1995.
26. W. Shultz, P. Dayan, and P. R. Montague, "A neural substrate of prediction and reward," Science, vol. 275, no. 5306, pp. 1593-1599, Mar. 1997.
27. R. S. Sutton, "Learning to predict by the methods of temporal differences," Mach. Learn., vol. 3, no. 1, pp. 9-44, Aug. 1988.
28. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction Cambridge, MA: MIT Press, 1998.
29. R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour, "Policy gradient methods for reinforcement learning with function approximation," in Proc. Advances Neural Inf. Process. Syst., 2000, vol. 12, pp. 1057-1063.
30. G. Tesauro, "Temporal difference learning and TD-Gammon," Commun. ACM, vol. 38, no. 3, pp. 58-68, Mar. 1995.
31. E. Todorov and M. Jordan, "Optimal feedback control as a theory of motor coordination," Nat. Neurosci., vol. 5, no. 11, pp. 1226-1235, Nov. 2002.