Proto-value functions: A Laplacian framework for learning representation and control in Markov decision processes

Journal of Machine Learning Research

Volumn 8, Issue , 2007, Pages 2169-2231

  Mahadevan, Sridhar ^a   Maggioni, Mauro ^b  

^a Biologically Inspired Neural and Dynamical Systems Laboratory   (United States)

^b DUKE UNIVERSITY   (United States)

Author keywords

Manifold learning; Markov decision processes; Reinforcement learning; Spectral graph theory; Value function approximation

Indexed keywords

LEAST SQUARES POLICY ITERATION (LSPI); MANIFOLD LEARNING; MARKOV DECISION PROCESSES; PROTO VALUE FUNCTIONS (PVFS); SPECTRAL GRAPH THEORY; VALUE FUNCTION APPROXIMATION;

EIGENVALUES AND EIGENFUNCTIONS; ITERATIVE METHODS; LAPLACE EQUATION; MARKOV PROCESSES; VALUE ENGINEERING;

REINFORCEMENT LEARNING;

EID: 35748957806     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (111)

1. th International Conference on Neural Information Processing Systems (NIPS), pages 335-342. MIT Press, 2002.
2. S. Amarel. On representations of problems of reasoning about actions. In Donald Michie, editor, Machine Intelligence 3, volume 3, pages 131-171. Elsevier/North-Holland, 1968.
3. J. Bagnell and J. Schneider. Covariant policy search. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pages 1019-1024, 2003.
4. A. Baker. Matrix Groups: An Introduction to Lie Group Theory. Springer, 2001.
5. C. Baker. The Numerical Treatment of Integral Equations. Oxford: Clarendon Press, 1977.
6. A. Barto and S. Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete Event Systems Journal, 13:41-77, 2003.
7. M. Belkin and P. Niyogi. Towards a theoretical foundation for Laplacian-based manifold methods. In Proceedings of the International Conference on Computational Learning Theory (COLT), pages 486-500, 2005.
8. M. Belkin and P. Niyogi. Semi-supervised learning on Riemannian manifolds. Machine Learning, 56:209-239, 2004.
9. M. Belkin, P. Niyogi, and V. Sindhwani. Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. Journal of Machine Learning Research, 7:23992434, 2006.
10. th European Conference on Computer vision, pages 531-542, 2002.
11. M. Ben-Chen and C. Gotsman. On the optimality of spectral compression of mesh data. ACM Transactions on Graphics, 24(1), 2005.
12. A. Bernasconi. Mathematical Techniques for Analysis of Boolean Functions. PhD thesis, University of Pisa, 1998.
13. D. Bertsekas and J. Tsitsiklis. Neuro-Dynamic Programming. Athena Scientific, Belmont, Massachusetts, 1996.
14. L. Billera and P. Diaconis. A geometric interpretation of the Metropolis-Hasting algorithm. Statistical Science, 16:335-339, 2001.
15. C. Boutilier, T. Dean, and S. Hanks. Decision-theoretic planning: Structural assumptions and computational leverage. Journal of Artificial Intelligence Research, 11:1-94, 1999.
16. J. A. Boyan. Least-squares temporal difference learning. In Proceedings of the 16th International Conference on Machine Learning, pages 49-56. Morgan Kaufmann, San Francisco, CA, 1999.
17. S. Bradtke and A. Barto. Linear least-squares algorithms for temporal difference learning. Machine Learning, 22:33-57, 1996.
18. J. Bremer, R. Coifman, M.Maggioni, and A. Szlam. Diffusion wavelet packets. Applied and Computational Harmonic Analysis, 21(1):95-112, July 2006.
19. G. Chirikjian and A. Kyatkin. Engineering Applications of Noncommutative Harmonic Analysis. CRC Press, 2001.
20. T. Chow. The Q-spectrum and spanning trees of tensor products of bipartite graphs. Proceedings of the American Mathematical Society, 125(11):3155-3161, 1997.
21. F. Chung. Spectral Graph Theory. Number 92 in CBMS Regional Conference Series in Mathematics. American Mathematical Society, 1997.
22. F Chung. Laplacians and the Cheeger inequality for directed graphs. Annals of Combinatorics, 9 (1):1-19, April 2005.
23. F. Chung and S. Sternberg. Laplacian and vibrational spectra for homogeneous graphs. Journal of Graph Theory, 16(6):605-627, 1992.
24. R. Coifman and M. Maggioni. Diffusion wavelets. Applied and Computational Harmonic Analysis, 21(1):53-94, July 2006.
25. R. Coifman, S. Lafon, A. Lee, M. Maggioni, B. Nadler, F. Warner, and S. Zucker. Geometric diffusions as a tool for harmonic analysis and structure definition of data, part i: Diffusion maps. Proceedings of National Academy of Science., 102(21):7426-7431, May 2005a.
26. R. Coifman, S. Lafon, A. Lee, M. Maggioni, B. Nadler, Frederick Warner, and Steven Zucker. Geometric diffusions as a tool for harmonic analysis and structure definition of data, part ii: Multiscale methods. Proceedings of the National Academy of Science, 102(21):7432-7437, May 2005b.
27. R. Coifman, M. Maggioni, S. Zucker, and I. Kevrekidis. Geometric diffusions for the analysis of data from sensor networks. Curr Opin Neurobiol, 15(5):576-84, October 2005c.
28. D. Cvetkovic, M. Doob, and H. Sachs. Spectra of Graphs: Theory and Application. Academic Press, 1980.
29. D. Cvetkovic, P. Rowlinson, and S. Simic. Eigenspaces of Graphs. Cambridge University Press, 1997.
30. P. Dayan. Improving generalisation for temporal difference learning: The successor representation. Neural Computation, 5:613-624, 1993.
31. D. de Farias. The linear programming approach to approximate dynamic programming. In Learning and Approximate Dynamic Programming: Scaling up to the Real World. John Wiley and Sons, 2003.
32. F. Deutsch. Best Approximation In Inner Product Spaces. Canadian Mathematical Society, 2001.
33. T. Dietterich and X. Wang. Batch value function approximation using support vectors. In Proceedings of Neural Information Processing Systems. MIT Press, 2002.
34. P Drineas and M W Mahoney. On the Nyström method for approximating a Gram matrix for improved kernel-based learning. J. Machine Learning Research, 6:2153-2175, 2005.
35. C. Drummond. Accelerating reinforcement learning by composing solutions of automatically identified subtasks. Journal of AIResearch, 16:59-104, 2002.
36. Y. Engel, S. Mannor, and R. Meir. Bayes meets Bellman: The Gaussian process approach to temporal difference learning. In Proceedings of the 20th International Conference on Machine Learning, pages 154-161. AAAI Press, 2003.
37. K. Ferguson and S. Mahadevan. Proto-transfer learning in Markov decision processes using spectral methods. In International Conference on Machine Learning (ICML) Workshop on Transfer Learning, 2006.
38. D. Foster and P. Dayan. Structure in the space of value functions. Machine Learning, 49:325-346, 2002.
39. A Frieze, R Kannan, and S Vempala. Fast Monte Carlo algorithms for finding low-rank approximations. In Proceedings of the 39th annual IEEE symposium on foundations of computer science, pages 370-378, 1998.
40. G. Gordon. Stable function approximation in dynamic programming. Technical Report CMU-CS95-103, Department of Computer Science, Carnegie Mellon University, 1995.
41. A. Graham. Kronecker Products and Matrix Calculations: With Applications. Ellis Horwood, 1981.
42. C. Guestrin, D. Koller, and R. Parr. Max-norm projections for factored Markov decision processes. In Proceedings of the 15th IJCAI, 2001.
43. C. Guestrin, D. Koller, R. Parr, and S. Venkataraman. Efficient solution algorithms for factored MDPs. Journal of AI Research, 19:399-468, 2003.
44. D. Gurarie. Symmetries and Laplacians: Introduction to Harmonic Analysis, Group Representations and Laplacians. North-Holland, 1992.
45. M. Hein, J. Audibert, and U. von Luxburg. Graph Laplacians and their convergence on random neighborhood graphs. Journal of Machine Learning Research, 8:1325-1368, 2007.
46. J. Jackson. The Harmonic Sieve: A Novel Application of Fourier Analysis to Machine Learning Theory and Practice. PhD thesis, Carnegie-Mellon University, 1995.
47. J. Johns and S. Mahadevan. Constructing basis functions from directed graphs for value function approximation. In Proceedings of the International Conference on Machine Learning (ICML), pages 385-392. ACM Press, 2007.
48. J. Johns, S. Mahadevan, and C. Wang. Compact spectral bases for value function approximation using Kronecker factorization. In Proceedings of the National Conference on Artificial Intelligence (AAAI), 2007.
49. P. Jones, M. Maggioni, and R. Schul. Universal parametrizations via eigenfunctions of the Laplacian and heat kernels. Submitted, 2007.
50. S. Kakade. A Natural Policy Gradient. In Proceedings of Neural Information Processing Systems. MIT Press, 2002.
51. G. Karypis and V. Kumar. A fast and high quality multilevel scheme for partitioning irregular graphs. SIAM Journal of Scientific Computing, 20(1):359-392, 1999.
52. nd International Conference on Machine Learning (ICML), pages 449-456. MIT Press, 2006.
53. D. Koller and R. Parr. Policy iteration for factored MDPs. In Proceedings of the 16th Conference on Uncertainty in AI, pages 326-334, 2000.
54. R. Kondor and J. Lafferty. Diffusion kernels on graphs and other discrete input spaces. In Proceedings of the 19th International Conference on Machine Learning, pages 315-322, 2002.
55. R. Kondor and R. Vert. Diffusion kernels. In Kernel Methods in Computational Biology. MIT Press, 2004.
56. R. Kretchmar and C. Anderson. Using temporal neighborhoods to adapt function approximators in reinforcement learning. In International Work Conference on Artificial and Natural Neural Networks, pages 488-496, 1999.
57. S. Kveton and M. Hauskrecht. Learning basis functions in hybrid domains. In Proceedings of the Twentieth National Conference on Artificial Intelligence, 2006.
58. J. Lafferty and G. Lebanon. Diffusion kernels on statistical manifolds. Journal of Machine Learning Research, 6:129-163, 2005.
59. S. Lafon. Diffusion Maps and Geometric Harmonics. PhD thesis, Yale University, Dept of Mathematics & Applied Mathematics, 2004.
60. M. Lagoudakis and R. Parr. Least-squares policy iteration. Journal of Machine Learning Research, 4:1107-1149, 2003.
61. J. C. Latombe. Robot Motion Planning. Kluwer Academic Press, 1991.
62. S. Lavalle. Planning Algorithms. Cambridge University Press, 2006.
63. J. M. Lee. Introduction to Smooth Manifolds. Springer, 2003.
64. M. Maggioni and S. Mahadevan. Fast direct policy evaluation using multiscale analysis of Markov Diffusion Processes. In Proceedings of the 23rd international conference on Machine learning, pages 601-608, New York, NY, USA, 2006. ACM Press.
65. S. Mahadevan. Proto-Value Functions: Developmental Reinforcement Learning. In Proceedings of the International Conference on Machine Learning, pages 553-560, 2005a.
66. S. Mahadevan. Enhancing transfer in reinforcement learning by building stochastic models of robot actions. In Proceedings of the Ninth International Conference on Machine Learning, Aberdeen, Scotland, pages 290-299, 1992.
67. S. Mahadevan. Representation policy iteration. In Proceedings of the 21th Annual Conference on Uncertainty in Artificial Intelligence (UAI-05), pages 372-37. AUAI Press, 2005b.
68. S. Mahadevan and M. Maggioni. Value function approximation with Diffusion Wavelets and Laplacian Eigenfunctions. In Proceedings of the Neural Information Processing Systems (NIPS). MIT Press, 2006.
69. S. Mahadevan, M. Maggioni, K. Ferguson, and S. Osentoski. Learning representation and control in continuous markov decision processes. In Proceedings of the National Conference on Artificial Intelligence (AAAI), 2006.
70. S. Mallat. A theory for multiresolution signal decomposition: The wavelet representation. IEEE Trans. Pattern Anal. Mach. Intell., 11(7):674-693, 1989. ISSN 0162-8828.
71. S. Mannor, I. Menache, A. Hoze, and U. Klein. Dynamic abstraction in reinforcement learning via clustering. In International Conference on Machine Learning, 2004.
72. A. McGovern. Autonomous Discovery of Temporal Abstractions from Interactions with an Environment. PhD thesis, University of Massachusetts, Amherst, 2002.
73. N. Menache, N. Shimkin, and S. Mannor. Basis function adaptation in temporal difference reinforcement learning. Annals of Operations Research, 134:215-238, 2005.
74. R. Munos. Error bounds for approximate value iteration. In Proceedings of the National Conference on Artificial Intelligence (AAAI), pages 1006-1011, 2005.
75. R. Munos. Error bounds for approximate policy iteration. In Proceedings of the International Conference on Machine Learning (ICML), pages 560-567, 2003.
76. A. Nedic and D. Bertsekas. Least-squares policy evaluation algorithms with linear function approximation. Discrete Event Systems Journal, 13, 2003.
77. A. Ng, M. Jordan, and Y Weiss. On spectral clustering: Analysis and an algorithm. In NIPS, 2002.
78. P. Niyogi, I. Matveeva, and M. Belkin. Regression and regularization on large graphs. Technical report, University of Chicago, Nov. 2003.
79. D. Ormoneit and S. Sen. Kernel-based reinforcement learning. Machine Learning, 49(2-3): 161178, 2002.
80. S. Osentoski and S. Mahadevan. Learning State Action Basis Functions for Hierarchical Markov Decison Processes. In Proceedings of the International Conference on Machine Learning (ICML), pages 705-712, 2007.
81. R. Parr, C. Painter-Wakefiled, L. Li, and M. Littman. Analyzing feature generation for value function approximation. In Proceedings of the International Conference on Machine Learning (ICML), pages 737-744, 2007.
82. R. Patrascu, P. Poupart, D. Schuurmans, C. Boutilier, and C. Guestrin. Greedy Linear Value Function Approximation for Factored Markov Decision Processes. In Proceedings of the National Conference on Artificial Intelligence (AAAI), pages 285-291, 2002.
83. J. Peters, S. Vijaykumar, and S. Schaal. Reinforcement learning for humanoid robots. In Proceedings of the Third IEEE-RAS International Conference on Humanoid Robots, 2003.
84. M. Petrik. An analysis of Laplacian methods for value function approximation in MDPs. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pages 25742579, 2007.
85. P. Poupart and C. Boutilier. Value directed compression of POMDPs. In Proceedings of the International Conference on Neural Information Processing Systems (NIPS), 2003.
86. M. L. Puterman. Markov Decision Processes. Wiley Interscience, New York, USA, 1994.
87. C. Rasmussen and M. Kuss. Gaussian Processes in Reinforcement Learning. In Proceedings of the International Conference on Neural Information Processing Systems, pages 751-759. MIT Press, 2004.
88. C. Rasmussen and C. Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
89. B. Ravindran and A. Barto. SMDP homomorphisms: An algebraic approach to abstraction in Semi-Markov Decision Processes. In Proceedings of the 18th International Joint Conference on Artificial Intelligence, 2003.
90. S Rosenberg. The Laplacian on a Riemannian Manifold. Cambridge University Press, 1997.
91. S. Roweis and L. Saul. Nonlinear dimensionality reduction by local linear embedding. Science, 290:2323-2326, 2000.
92. B. Sallans and G. Hinton. Reinforcement learning with factored states and actions. Journal of Machine Learning Research, 5:1063-1088, 2004.
93. T. Sato. Perturbation Theory for Linear Operators. Springer, 1995.
94. B. Scholkopf and A. Smola. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, 2001.
95. A. Sherstov and P. Stone. Improving action selection in Markov Decision Processes via knowledge transfer. In Proceedings of the Twentieth National Conference on Artificial Intelligence, 2005.
96. J. Shi and J. Malik. Normalized cuts and image segmentation. IEEE PAMI, 22:888-905, 2000.
97. O. ŞimŞek, A. Wolfe, and A. Barto. Identifying useful subgoals in reinforcement learning by local graph partitioning. In Proceedings of the Twenty-Second International Conference on Machine Learning, pages 816-823, 2005.
98. G. Stewart and J. Sun. Matrix Perturbation Theory. Academic Press, 1990.
99. D. Subramanian. A Theory of Justified Reformulations. Ph.D. Thesis, Stanford University, 1989.
100. M. Sugiyama, H. Hachiya, C. Towell, and S. Vijaykumar. Value function approximation on nonlinear manifolds for robot motor control. In Proceedings of the IEEE Conference on Robots and Automation (ICRA), 2007.
101. R. Sutton and A. G. Barto. An Introduction to Reinforcement Learning. MIT Press, 1998.
102. A. Szlam, M. Maggioni, and R. Coifman. A general framework for adaptive regularization based on diffusion processes on graphs. Technical Report YALE/DCS/TR1365, Yale Univ, July 2006.
103. J. Tenenbaum, V. de Silva, and J. Langford. A global geometric framework for nonlinear dimensionality reduction. Science, 290:2319-2323, 2000.
104. G. Tesauro. Practical issues in temporal difference learning. Machine Learning, 8:257-278, 1992.
105. M. Thornton, R. Drechsler, and D. Miller. Spectral Methods for VLSI Design. Kluwer Academic, 2001.
106. J. Tsitsiklis and B. Van Roy. An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control, 42:674-690, 1997.
107. P. Utgoff and D. Stracuzzi. Many-layered learning. Neural Computation, 14:2497-2529, 2002.
108. C. Van Loan and N. Pitsianis. Approximation with Kronecker products. In Linear Algebra for Large Scale and Real Time Applications, pages 293-314. Kluwer Publications, 1993.
109. B. Van Roy. Learning and Value Function Approximation in Complex Decision Processes. PhD thesis, MIT, 1998.
110. C. Watkins. Learning from Delayed Rewards. PhD thesis, King's College, Cambridge, England, 1989.
111. C. Williams and M. Seeger. Using the Nyström method to speed up kernel machines. In Proceedings of the International Conference on Neural Information Processing Systems, pages 682-688, 2000.