Learning representation and control in markov decision processes: New frontiers

Foundations and Trends in Machine Learning

Volumn 1, Issue 4, 2008, Pages 403-565

  Mahadevan, Sridhar ^a  

^a Biologically Inspired Neural and Dynamical Systems Laboratory   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DECISION PROBLEMS; DIAGONALIZATIONS; DIMENSIONALITY REDUCTION; DRAZIN INVERSE; EXACT SOLUTION; GENERIC ALGORITHM; LAPLACIAN OPERATOR; LAPLACIANS; LOW-DIMENSIONAL REPRESENTATION; MACHINE-LEARNING; MARKOV DECISION PROCESSES; MATHEMATICAL FRAMEWORKS; MATRIX REPRESENTATION; MODEL FREE; MODEL-BASED; OFF-DIAGONAL ELEMENTS; OPTIMAL CONTROLS; OPTIMAL POLICIES; POLICY ITERATION; ROW SUMS;

COMPUTER SCIENCE; LAPLACE EQUATION; LAPLACE TRANSFORMS; MARKOV PROCESSES; OPTIMIZATION; PROCESS CONTROL;

MATHEMATICAL OPERATORS;

EID: 70349322784     PISSN: 19358237     EISSN: 19358245     Source Type: Journal    
DOI: 10.1561/2200000003     Document Type: Article

References (144)

1. D. Achlioptas, F. McSherry, and B. Scholkopff, "Sampling techniques for Kernel methods," in Proceedings of the 14th International Conference on Neural Information Processing Systems (NIPS), pp. 335-342, MIT Press, 2002.
2. R. Agaev and P. Cheboratev, "On the spectra of nonsymmetric Laplacian matrices," Linear Algebra and Its Applications, vol.399, pp. 157-168, 2005.
3. S. Amarei, "On representations of problems of reasoning about actions," in Machine Intelligence 3, (D. Michie, ed.), pp. 131-171, Elsevier/North-Holland, 1968.
4. S. Axler, P. Bourdon, and W. Ramey, Harmonic Function Theory. Springer, 2001.
5. J. Bagnell and J. Schneider, "Covariant Policy Search," in Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pp. 1019-1024, 2003.
6. C. T. H. Baker, The Numerical Treatment of Integral Equations. Oxford: Clarendon Press, 1977.
7. A. Barto and S. Mahadevan, "Recent advances in hierarchical reinforcement learning," Discrete Event Systems Journal, vol.13, pp. 41-77, 2003.
8. M. Belkin and P. Niyogi, "Semi-supervised learning on Riemannian manifolds," Machine Learning, vol.56, pp. 209-239, 2004.
9. S. Belongie, C. Fowlkes, F. Chung, and J. Malik, "Spectral partitioning with indefinite Kernels using the Nyström extension," in Proceedings of the 7th European Conference on Computer Vision, pp. 531-542, 2002.
10. A. Berman and R. Plemmons, Nonnegative Matrices in the Mathematical Sciences. SIAM Press, 1994.
11. D. Bertsekas and D. Castanon, "Adaptive Aggregation Methods for infinite horizon dynamic programming," IEEE Transactions on Automatic Control, vol.34, pp. 589-598, 1989.
12. D. Bertsekas and J. Tsitsiklis, Neuro-Dynamic Programming. Belmont, MA: Athena Scientific, 1996.
13. B. Bethke, J. How, and A. Ozdaglar, "Approximate dynamic programming using support vector regression," in Proceedings of the IEEE Conference on Decision and Control, 2008.
14. G. Beylkin, R. R. Coifman, and V. Rokhlin, "Fast wavelet transforms and numerical algorithms," Common Pure and Applied Mathematic, vol.44, pp. 141-183, 1991.
15. L. Billera and P. Diaconis, "A geometric interpretation of the MetropolisHasting algorithm," Statistical Science, vol.16, pp. 335-339, 2001.
16. J. A. Boyan, "Least-squares temporal difference learning," in Proceedings of the 16th International Conference on Machine Learning, pp. 49-56, San Francisco, CA: Morgan Kaufmann, 1999.
17. S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge University Press, 2004.
18. S. Bradtke and A. Barto, "Linear least-squares algorithms for temporal difference learning," Machine Learning, vol.22, pp. 33-57, 1996.
19. J. Bremer, R. Coifman, M. Maggioni, and A. Szlam, "Diffusion wavelet packets," Applied and Computational Harmonic Analysis, vol.21, no.1, pp. 95-112, July 2006.
20. S. Campbell and C. Meyer, Generalized Inverses of Linear Transformations. Pitman, 1979.
21. [21] X. Cao, "The relations among potentials, perturbation analysis, and Markov decision processes," Discrete-Event Dynamic Systems, vol.8, no.1, pp. 71-87, 1998. (Pubitemid 128512397)
22. J. Caughman and J. Veerman, "Kernels of directed graph Laplacians," Electronic Journal of Combinatorics, vol.13, no.1, pp. 253-274, 2006.
23. I. Chavel, Eigenvalues in Riemannian Geometry: Pure and Applied Mathematics. Academic Press, 1984.
24. P. Chebotarev and R. Agaev, "Forest matrices around the Laplacian matrix," Linear Algebra and Its Applications, vol.15, no.1, pp. 253-274, 2002.
25. L. Chen, E. Krishnamurthy, and I. Macleod, "Generalized matrix inversion and rank computation by repeated squaring," Parallel Computing, vol.20, pp. 297-311, 1994.
26. F. Chung, Spectral Graph Theory, Number 92 in CBMS Regional Conference Series in Mathematics. American Mathematical Society, 1997.
27. F. Chung, "Laplacians and the Cheeger inequality for directed graphs," Annals of Combinatorics, vol.9, no.1, pp. 1-19, April 2005.
28. R. Coifman, S. Lafon, A. Lee, M. Maggioni, B. Nadler, F. Warner, and S. Zucker, "Geometric diffusions as a tool for harmonic analysis and structure definition of data. Part i: Diffusion maps," Proceedings of National Academy of Science, vol.102, no.21, pp. 7426-7431, May 2005.
29. R. Coifman, S. Lafon, A. Lee, M. Maggioni, B. Nadler, F. Warner, and S. Zucker, "Geometrie diffusions as a tool for harmonie analysis and structure definition of data. Part ii: Multiscale methods," Proceedings of the National Academy of Science, vol.102, no.21, pp. 7432-7437, May 2005.
30. R. Coifman and M. Maggioni, "Diffusion wavelets," Applied and Computational Harmonic Analysis, vol.21, no.1, pp. 53-94, July 2006.
31. R. Coifman, M. Maggioni, S. Zucker, and I. Kevrekidis, "Geometric diffusions for the analysis of data from sensor networks," Curr Opin Neurobiol, vol.15, no.5, pp. 576-584, October 2005.
32. D. Cvetkovic, M. Doob, and H. Sachs, Spectra of Graphs: Theory and Application. Academic Press, 1980.
33. T. Das, A. Gosavi, S. Mahadevan, and N. Marchalleck, "Solving semi-Markov decision problems using average-reward reinforcement learning," Management Science, vol.45, no.4, pp. 560-574, 1999.
34. I. Daubechies, Ten Lectures on Wavelets, Society for Industrial and Applied Mathematics. 1992.
35. P. Dayan, "Improving generalisation for temporal difference learning: The successor representation," Neural Computation, vol.5, pp. 613-624, 1993.
36. D. de Farias, "The linear programming approach to approximate dynamic programming," in Learning and Approximate Dynamic Programming: Scaling Up to the Real World, John Wiley and Sons, 2003.
37. F. Deutsch, Best Approximation in Inner Product Spaces. Canadian Mathematical Society, 2001.
38. T. Dietterich and X. Wang, "Batch value function approximation using support vectors," in Proceedings of Neural Information Processing Systems, MIT Press, 2002.
39. P. Drineas and M. W. Mahoney, "On the Nyström method for approximating a Gram matrix for improved Kernel-based learning," Journal of Machine Learning Research, vol.6, pp. 2153-2175, 2005.
40. C. Drummond, "Accelerating reinforcement learning by composing solutions of automatically identified subtasks," Journal of AI Research, vol.16, pp. 59-104, 2002.
41. [41] M. Eiermann and O. Ernst, "Geometric aspects of the theory of Krylov subspace methods," Acta Numérica, pp. 251-312, 2001. (Pubitemid 33305812)
42. Y. Engel, S. Mannor, and R. Meir, "Bayes meets Bellman: The Gaussian process approach to temporal difference learning," in Proceedings of the 20th International Conference on Machine Learning, pp. 154-161, AAAI Press, 2003.
43. K. Ferguson and S. Mahadevan, "Proto-transfer learning in Markov decision processes using spectral methods," in International Conference on Machine Learning (ICML) Workshop on Transfer Learning, 2006.
44. M. Fiedler, "Algebraic connectivity of graphs," Czechoslovak Mathematical Journal, vol.23, no.98, pp. 298-305, 1973.
45. D. Foster and P. Dayan, "Structure in the space of value functions," Machine Learning, vol.49, pp. 325-346, 2002.
46. A. Frieze, R. Kannan, and S. Vempala, "Fast Monte Carlo algorithms for finding low-rank approximations," in Proceedings of the 39th Annual IEEE Symposium on Foundations of Computer Science, pp. 370-378, 1998.
47. M. Ghavamzadeh and S. Mahadevan, "Hierarchical average-reward reinforcement learning," Journal of Machine Learning Research, vol.8, pp. 2629-2669, 2007.
48. R. Givan and T. Dean, "Model minimization in Markov decision processes," AAAI, 1997.
49. G. Golub and C. V. Loan, Matrix Computations. Johns Hopkins University Press, 1989.
50. G. Gordon, "Stable function approximation in dynamic programming," Technical Report, CMU-CS-95-103, Department of Computer Science, Carnegie Mellon University, 1995.
51. C. Guestrin, D. Koller, R. Parr, and S. Venkataraman, "Efficient solution algorithms for factored MDPs," Journal of AI Research, vol.19, pp. 399-468, 2003.
52. D. Gurarie, Symmetries and Laplacians: Introduction to Harmonic Analysis, Group Representations and Laplacians. North-Holland, 1992.
53. M. Hein, J. Audibert, and U. von Luxburg, "Graph Laplacians and their convergence on random neighborhood graphs," Journal of Machine Learning Research, vol.8, pp. 1325-1368, 2007.
54. M. Herbster, M. Pontil, and L. Wainer, "Online learning over graphs," in Proceedings of the Twenty-Second International Conference on Machine Learning, 2005.
55. J. Hoey, R. St-aubin, A. Hu, and C. Boutilier, "SPUDD: Stochastic planning using decision diagrams," in Proceedings of the Fifteenth Conference on Uncertainty in Artificial Intelligence, pp. 279-288, Morgan Kaufmann, 1999.
56. R. Howard, Dynamic Programming and Markov Decision Processes. MIT Press, 1960.
57. J. Johns and S. Mahadevan, "Constructing basis functions from directed graphs for value function approximation," in Proceedings of the International Conference on Machine Learning (ICML), pp. 385-392, ACM Press, 2007.
58. J. Johns, S. Mahadevan, and C. Wang, "Compact spectral bases for value function approximation using Kronecker factorization," in Proceedings of the National Conference on Artificial Intelligence (AAAI), 2007.
59. T. Jolliffe, Principal Components Analysis. Springer-Verlag, 1986.
60. P. Jones, M. Maggioni, and R. Schul, "Universal parametrizations via Eigenfunctions of the Laplacian and heat kernels," Forthcoming 2007.
61. S. Kakadě, "A natural policy gradient," in Proceedings of Neural Information Processing Systems, MIT Press, 2002.
62. G. Karypis and V. Kumar, "A fast and high quality multilevel scheme for partitioning irregular graphs," SIAM Journal of Scientific Computing, vol.20, no.1, pp. 359-392, 1999.
63. A. Kaveh and A. Nikbakht, "Block diagonalization of Laplacian matrices of symmetric graphs using group theory," International Journal for Numerical Methods in Engineering, vol.69, pp. 908-947, 2007.
64. R. Kretchmar and C. Anderson, "Using temporal neighborhoods to adapt function approximators in reinforcement learning," in International Work Conference on Artificial and Natural Neural Networks, pp. 488-496, 1999.
65. H. Kushner and G. Yin, Stochastic Approximation and Recursive Algorithms and Applications. Springer, 2003.
66. B. Kveton, "Learning basis functions in hybrid domains," in Proceedings of the 21st National Conference on Artificial Intelligence, pp. 1161-1166, 2006.
67. J. Lafferty and G. Lebanon, "Diffusion Kernels on statistical manifolds," Journal of Machine Learning Research, vol.6, pp. 129-163, 2005.
68. S. Lafon, "Diffusion maps and geometric harmonics," PhD thesis, Yale University, Department of Mathematics and Applied Mathematics, 2004.
69. M. Lagoudakis and R. Parr, "Least-squares policy iteration," Journal of Machine Learning Research, vol.4, pp. 1107-1149, 2003.
70. A. Langville and C. Meyer, "Updating the stationary vector of an irreducible Markov chain with an eye on google's pagerank," SIAM Journal on Matrix Analysis, vol.27, pp. 968-987, 2005.
71. J. C. Latombe, Robot Motion Planning. Kluwer Academic Press, 1991.
72. S. Lavalle, Planning Algorithms. Cambridge University Press, 2006.
73. J. Lee and M. Verleysen, Nonlinear Dimensionality Reduction. Springer, 2007.
74. J. M. Lee, Introduction to Smooth Manifolds. Springer, 2003.
75. L. Li, T. Walsh, and M. Littman, "Towards a unified theory of state abstraction for MDPs," in Proceedings of the Ninth International Symposium on Artificial Intelligence and Mathematics, pp. 531-539, 2006.
76. M. Maggioni and S. Mahadevan, "Fast direct policy evaluation using multiscale analysis of Markov diffusion processes," in Proceedings of the 23rd International Conference on Machine Learning, pp. 601-608, New York, NY, USA: ACM Press, 2006.
77. S. Mahadevan, "Proto-value functions: Developmental reinforcement learning," in Proceedings of the International Conference on Machine Learning, pp. 553-560, 2005.
78. S. Mahadevan, "Representation policy iteration," in Proceedings of the 21st Annual Conference on Uncertainty in Artificial Intelligence (UAI-05), pp. 372-437, AUAI Press, 2005.
79. S. Mahadevan, "Fast spectral learning using lanczos eigenspace projections," in Proceedings of the National Conference on Artificial Intelligence (AAAI), 2008.
80. S. Mahadevan, Representation Discovery Using Harmonic Analysis. Morgan and Claypool Publishers, 2008.
81. [81] S. Mahadevan and J. Connell, "Automatic programming of behaviorbased robots using reinforcement learning," Artificial Intelligence, vol.55, pp. 311-365, 1992. Appeared originally as IBM TR RC16359, December 1990. (Pubitemid 23565211)
82. S. Mahadevan and M. Maggioni, "Value function approximation with diffusion wavelets and Laplacian eigenfunctions," in Proceedings of the Neural Information Processing Systems (NIPS), MIT Press, 2006.
83. S. Mahadevan and M. Maggioni, "Proto-value functions: A Laplacian framework for learning representation and control in Markov decision processes," Journal of Machine Learning Research, vol.8, pp. 2169-2231, 2007.
84. S. Mahadevan, M. Maggioni, K. Ferguson, and S. Osentoski, "Learning representation and control in continuous Markov decision processes," in Proceedings of the National Conference on Artificial Intelligence (AAAI), 2006.
85. S. Mahadevan, N. Marchalleck, T. Das, and A. Gosavi, "Self-improving factory simulation using continuous-time average-reward reinforcement learning," in Proceedings of 14-th International Conference on Machine Learning, pp. 202-210, Morgan Kaufmann, 1997.
86. S. Mallat, "A theory for multiresolution signal decomposition: The wavelet representation," IEEE Transactions on Pattern Analysis of Machanical Intelligence, vol.11, no.7, pp. 674-693, 1989.
87. S. Mallat, A Wavelet Tour in Signal Processing. Academic Press, 1998.
88. D. Malsen, M. Orrison, and D. Rockmore, "Computing isotypic projections with the lanczos iteration," SIAM, vol.2, nos. 60/61, pp. 601-628, 2003.
89. S. Mannor, I. Menache, A. Hoze, and U. Klein, "Dynamic abstraction in reinforcement learning via clustering," International Conference on Machine Learning, 2004.
90. A. McGovern, "Autonomous discovery of temporal abstractions from interactions with an environment," PhD thesis, University of Massachusetts, Amherst, 2002.
91. M. Meila and J. Shi, "Learning segmentation by random walks," NIPS, 2001.
92. C. Meyer, "Sensitivity of the stationary distribution of a Markov chain," SIAM Journal of Matrix Analysis and Applications, vol.15, no.3, pp. 715-728, 1994.
93. A. Moore, "Barycentric interpolators for continuous space and time reinforcement learning," in Advances in Neural Information Processing Systems, MIT Press, 1998.
94. R. Munos, "Error bounds for approximate policy iteration," in Proceedings of the International Conference on Machine Learning (ICML), pp. 560-567, 2003.
95. E. Nelson, Tensor Analysis. Princeton University Press, 1968.
96. A. Ng, M. Jordan, and Y. Weiss, "On spectral clustering: Analysis and an algorithm," NIPS, 2002.
97. A. Ng, H. Kim, M. Jordan, and S. Sastry, "Autonomous helicopter flight via Reinforcement Learning," in Proceedings of Neural Information Processing Systems, 2004.
98. P. Niyogi and M. Belkin, "Semi-supervised learning on Riemannian manifolds," Technical Report TR-2001-2030, University of Chicago, Computer Science Deparment, November 2001.
99. P. Niyogi, I. Matveeva, and M. Belkin, "Regression and regularization on large graphs," Techncial Report, University of Chicago, November 2003.
100. D. Ormoneit and S. Sen, "Kernel-based reinforcement learning," Machine Learning, vol.49, nos. 2-3, pp. 161-178, 2002.
101. S. Osentoski, "Action-based representation discovery in Markov decision processes," PhD thesis, University of Massachusetts, Amherst, 2009.
102. S. Osentoski and S. Mahadevan, "Learning state action basis functions for Hierarchical Markov decison processes," in Proceedings of the International Conference on Machine Learning (ICML), pp. 705-712, 2007.
103. [103] R. Parr, C. Painter-Wakefiled, L. Li, and M. Littman, "Analyzing feature generation for value function approximation," in Proceedings of the International Conference on Machine Learning (ICML), pp. 737-744, 2007. (Pubitemid 350094046)
104. R. Parr, C. Painter-Wakefiled, L. Li, and M. Littman, "An analysis of linear models, linear value-function approximation, and feature selection for reinforcement learning," in Proceedings of the International Conference on Machine Learning (ICML), 2008.
105. J. Peters, S. Vijaykumar, and S. Schaal, "Reinforcement learning for humanoid robots," in Proceedings of the Third IEEE-RAS International Conference on Humanoid Robots, 2003.
106. M. Petrik, "An analysis of Laplacian methods for value function approximation in MDPs," in Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pp. 2574-2579, 2007.
107. P. Poupart and C. Boutilier, "Value Directed Compression of POMDPs," in Proceedings of the International Conference on Neural Information Processing Systems (NIPS), 2003.
108. P. Poupart, C. Boutilier, R. Patrascu, and D. Schuurmans, "Piecewise linear value function approximation for factored Markov decision processes," in Proceedings of the National Conference on Artificial Intelligence (AAAI), pp. 285-291, 2002.
109. W. Powell, Approximate Dynamic Programming: Solving the Curses of Dimensionality. Wiley, 2007.
110. M. L. Puterman, Markov Decision Processes. New York, USA: Wiley Interscience, 1994.
111. C. Rasmussen and M. Kuss, "Gaussian processes in reinforcement learning," in Proceedings of the International Conference on Neural Information Processing Systems, pp. 751-759, MIT Press, 2004.
112. B. Ravindran and A. Barto, "SMDP homomorphisms: An algebraic approach to abstraction in semi-Markov decision processes," in Proceedings of the 18th International Joint Conference on Artificial Intelligence, 2003.
113. C. Robert and G. Casella, Monte-Carlo Methods in Statistics. Springer, 2005.
114. K. Rohanimanesh and S. Mahadevan, "Coarticulation: An approach for generating concurrent plans in Markov decision processes," in Proceedings of the International Conference on Machine Learning, ACM Press, 2005.
115. S. Rosenberg, The Laplacian on a Riemannian Manifold. Cambridge University Press, 1997.
116. [116] S. Roweis and L. Saul, "Nonlinear dimensionality reduction by local linear embedding," Science, vol.290, pp. 2323-2326, 2000. (Pubitemid 32041578)
117. S. Rusell and P. Norvig, Artificial Intelligence: A Modern Approach. PrenticeHall, 2002.
118. Y. Saad, Iterative Methods for Sparse Linear Systems. SIAM Press, 2003.
119. B. Sallans and G. Hinton, "Reinforcement learning with factored states and actions," Journal of Machine Learning Research, vol.5, pp. 1063-1088, 2004.
120. B. Scholkopf and A. Smola, Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, 2001.
121. P. Schweitzer, "Perturbation theory and finite Markov chains," Journal of Applied Probability, vol.5, no.2, pp. 410-413, 1968.
122. P. Schweitzer and A. Seidmann, "Generalized polynomial approximations in Markov decision processes," Journal of Mathematical Analysis and Applications, vol.110, pp. 568-582, 1985.
123. J. Serre, Linear Representations of Finite Groups. Springer, 1977.
124. R. St-Aubin, J. Hoey, and C. Boutilier, "Approximate policy construction using decision diagrams," NIPS, 2000.
125. E. M. Stein and R. Shakarchi, Fourier Analysis: An Introduction. Princeton University Press, 2003.
126. G. Stewart and J. Sun, Matrix Perturbation Theory. Academic Press, 1990.
127. G. Strang, Introduction to Linear Algebra. Wellesley-Cambridge Press, 2003.
128. D. Subramanian, "A theory of justified reformulations," PhD thesis, Stanford University, 1989.
129. R. Sutton and A. G. Barto, An Introduction to Reinforcement Learning. MIT Press, 1998.
130. R. S. Sutton, "Learning to predict by the methods of temporal differences," Machine Learning, vol.3, pp. 9-44, 1988.
131. [131] J. Tenenbaum, V. de Silva, and J. Langford, "A global geometric framework for nonlinear dimensionality reduction," Science, vol.290, pp. 2319-2323, 2000. (Pubitemid 32041577)
132. G. Tesauro, "Td-gammon, a self-teaching backgammon program, achieves master-level play," Neural Computation, vol.6, pp. 215-219, 1994.
133. Y. Tsao, K. Xiao, and V. Soo, "Graph Laplacian based transfer learning in reinforcement learning," in AAMAS '08: Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems, pp. 1349-1352, 2008.
134. B. Turker, J. Leydold, and P. Stadler, Laplacian Eigenvectors of Graphs. Springer, 2007.
135. P. Utgoff and D. Stracuzzi, "Many-layered learning," Neural Computation, vol.14, pp. 2497-2529, 2002.
136. C. Van Loan, Computational Frameworks for the Fast Fourier Transform. SIAM Press, 1987.
137. C. Van Loan and N. Pitsianis, "Approximation with Kronecker products," in Linear Algebra for Large Scale and Real Time Applications, pp. 293-314, Kluwer Publications, 1993.
138. B. Van Roy, "Learning and value function approximation in complex decision processes," PhD thesis, MIT, 1998.
139. G. Wahba, "Spline models for observational data," Society for Industrial and Applied Mathematics, 1990.
140. C. Watkins, "Learning from delayed rewards," PhD thesis, King's College, Cambridge, England, 1989.
141. Y. Wei, "Successive matrix squaring algorithm for computing the Drazin inverse," Applied Mathematics and Computation, vol.108, pp. 67-75, 2000.
142. C. Williams and M. Seeger, "Using the Nyström method to speed up Kernel machines," in Proceedings of the International Conference on Neural Information Processing Systems, pp. 682-688, 2000.
143. W. Zhang and T. Dietterich, "A reinforcement learning approach to job-shop scheduling," in Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence (IJCAI), pp. 1114-1120, 1995.
144. X. Zhou, "Semi-supervised learning with graphs," PhD thesis, Carnegie Mellon University, 2005.