Feature search in the grassmanian in online reinforcement learning

IEEE Journal on Selected Topics in Signal Processing

Volumn 7, Issue 5, 2013, Pages 746-758

  Bhatnagar, Shalabh ^a   Borkar, Vivek S ^b   Prabuchandran, K J ^a  

^a INDIAN INSTITUTE OF SCIENCE   (India)

^b INDIAN INSTITUTE OF TECHNOLOGY BOMBAY   (India)

Author keywords

Feature adaptation; Grassman manifold; online learning; residual gradient scheme; stochastic approximation; temporal difference learning

Indexed keywords

FEATURE ADAPTATION; GRASSMAN MANIFOLD; ONLINE LEARNING; RESIDUAL GRADIENT; STOCHASTIC APPROXIMATIONS; TEMPORAL DIFFERENCE LEARNING;

APPROXIMATION ALGORITHMS; REINFORCEMENT LEARNING;

E-LEARNING;

EID: 84884515550     PISSN: 19324553     EISSN: None     Source Type: Journal    
DOI: 10.1109/JSTSP.2013.2255022     Document Type: Article

References (35)

1. P.-A. Absil, R. Mahony, and R. Sepulchre, Optimization Algorithms on Matrix Manifolds. Princeton, NJ, USA: Princeton Univ. Press, 2008.
2. L. C. Baird, "Residual algorithms: Reinforcement learning with function approximation," in Proc. 12th Int. Conf. Mach. Learn., 1995, pp. 30-37, Morgan Kaufmann.
3. L. Balzano, R. Nowak, and B. Recht, "Online identification and tracking of subspaces from highly incomplete information," in Proc. 48th Annu. Allerton Conf., 2010, pp. 704-711.
4. J. S. Baras andV. S. Borkar, "Alearning algorithmfor Markov decision processes with adaptive state aggregation," in Proc. 39th IEEE Conf. Decision Control, Dec. 12-15, 2000, Sydney, Australia, 2000, vol. 4, pp. 3351-3356. (Pubitemid 32528175)
5. K. Barman and V. S. Borkar, "A note on linear function approximation using random projections," Syst. Control Lett., vol. 57, no. 9, pp. 784-786, 2008.
6. A. Benveniste, M. Metivier, and P. Priouret, Adaptive Algorithms and Stochastic Approximations. Berlin, Germany: Springer-Verlag, 1990.
7. D. P. Bertsekas, Dynamic Programming and Optimal Control Vol. I (3rd ed.). Belmont, MA, USA: Athena Scientific, 2005.
8. D. P. Bertsekas, Dynamic Programming and Optimal Control Vol. II (4th ed.). Belmont, MA, USA: Athena Scientific, 2011.
9. D. P. Bertsekas and J. N. Tsitsiklis, Neuro-Dynamic Programming. Belmont, MA, USA: Athena Scientific, 1996.
10. D. P. Bertsekas and H. Yu, "Projected equation methods for approximate solution of large linear systems," J. Comput. Appl. Math., vol. 227, pp. 27-50, 2009.
11. S. Bhatnagar, M. C. Fu, S. I. Marcus, and I.-J. Wang, "Two time scale simultaneous perturbation stochastic approximation using deterministic perturbation sequences," ACM Trans. Model. Comput. Simulat., vol. 13, no. 2, pp. 180-209, 2003.
12. S. Bhatnagar, H. L. Prasad, and L. A. Prashanth, Stochastic Recursive Algorithms for Optimization: Simultaneous PerturbationMethods, ser. Lecture Notes in Control and Information Sciences. London, U.K.: Springer, 2013.
13. S. Bhatnagar, V. S. Borkar, and L. A. Prashanth, "Adaptive feature pursuit: Online adaptation of features in reinforcement learning," in Chapter 23 of Reinforcement Learning and Approximate Dynamic Programming for Feedback Control, F. Lewis and D. Liu, Eds. Piscataway, NJ, USA: IEEE Press Computational Intelligence Series, 2012.
14. S. Bhatnagar, V. S. Borkar, and K. J. Prabuchandran, "Adaptive Feature Tuning in the Grassmanian in Online Reinforcement Learning," Dept. of Comput. Sci., Indian Inst. of Science, IISc-CSA-SSL-TR-2013-2,March-2013 [Online]. Available: http://stochastic. csa.iisc.ernet.in/www/research/files/ IISc-CSA-SSLTR-2013-2.pdf
15. V. S. Borkar, "Stochastic approximation with two timescales," Syst. Control Lett., vol. 29, pp. 291-294, 1997.
16. V. S. Borkar, "Asynchronous stochastic approximations," SIAM J. Control Optimiz., vol. 36, no. 3, pp. 840-851, 1998. (Pubitemid 128493576)
17. V. S. Borkar, Stochastic Approximation: A Dynamical Systems Viewpoint. Cambridge, U. K.: (Jointly published by) Cambridge Univ. Press, 2008, Hindustan Book Agency, New Delhi, India.
18. J. A. Boyan, "Least-squares temporal difference learning," in Proc. 16th Int. Conf. Mach. Learn. San Francisco, CA, USA: Morgan Kaufmann, 1999, pp. 49-56.
19. D. Di Castro and S.Mannor, J. L. Balcar, F. Bonchi, A. Gionis, and M. Sebag, Eds., "Adaptive bases for reinforcement learning," Mach. Learn. Knowl. Discovery in Databases, Proc. Eur. Conf. Mach. Learn., Barcelona, Spain, ser. Lecture Notes in Computer Science, vol. 6321, pp. 312-327, 2010.
20. A. Edelman, T. A. Arias, and S. T. Smith, "The geometry of algorithms with orthogonality constraints," SIAM J. Matrix Anal. Applicat., vol. 20, no. 2, pp. 303-353, 1998. (Pubitemid 129333771)
21. P.W.Keller, S. Mannor, and D. Precup, "Automatic basis function construction for approximate dynamic programming and reinforcement learning," in Proc. 23rd Int. Conf. Mach. Learn., Pittsburgh, PA, USA, 2006.
22. H. J. Kushner and D. S. Clark, Stochastic Approximation Methods for Constrained and Unconstrained Systems. NewYork,NY,USA: Springer-Verlag, 1978.
23. H. J. Kushner and G. G. Yin, Stochastic Approximation Algorithms and Applications. New York, NY, USA: Springer Verlag, 1997.
24. H. R. Maei, C. Szepesvari, S. Bhatnagar, D. Precup, D. Silver, and R. S. Sutton, "Convergent temporal-difference learning with arbitrary smooth function approximation," in Proc. Adv. Neural Inf. Process. Syst., Vancouver, BC, Canada, 2009.
25. S. Mahadevan and B. Liu, "Basis construction from power series expansions of value functions," in Proc. Adv. Neural Inf. Process. Syst., Vancouver, BC, Canada, 2010.
26. I. Menache, S. Mannor, and N. Shimkin, "Basis function adaptation in temporal difference reinforcement learning," Ann. Operat. Res., vol. 134, pp. 215-238, 2005. (Pubitemid 40550047)
27. R. Parr, C. Painter-Wakefield, L. Li, and M. Littman, "Analyzing feature generation for value-function approximation," in Proc. 24th Int. Conf. Mach. Learn., Corvallis, OR, USA, 2007.
28. M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming. New York, NY, USA: Wiley, 1994.
29. Y. Sun, F. Gomez, M. Ring, and J. Schmidhuber, "Incremental basis construction from temporal difference error," in Proc. 28th Int. Conf. Mach. Learn., Bellevue,WA, USA, 2011.
30. R. S. Sutton, "Learning to predict by the method of temporal differences," Mach. Learn., vol. 3, pp. 9-44, 1988.
31. R. S. Sutton and A. Barto, Reinforcement Learning: An Introduction. Cambridge, MA, USA: MIT Press, 1998.
32. R. S. Sutton, H. R.Maei, D. Precup, S. Bhatnagar, D. Silver, C. Szepesvari, and E. Wiewiora, "Fast gradient-descent methods for temporaldifference learning with linear function approximation," in Proc. 26th Int. Conf. Mach. Learn., 2009, pp. 993-1000.
33. J. N. Tsitsiklis and B. Van Roy, "An analysis of temporal difference learning with function approximation," IEEE Trans. Autom. Control, vol. 42, no. 5, pp. 674-690, 1997. (Pubitemid 127760263)
34. J. Tsitsikis and B. Van Roy, "Average cost temporal-difference learning," Automatica, vol. 35, pp. 1799-1808, 1999. (Pubitemid 32078092)
35. H. Yu and D. P. Bertsekas, "Basis function adaptation methods for cost approximation in MDP," in Proc. IEEE Int. Symp. Adaptive Dynamic Program. Reinforce. Learn., Nashville, TN, 2009.