Projected equation methods for approximate solution of large linear systems

Journal of Computational and Applied Mathematics

Volumn 227, Issue 1, 2009, Pages 27-50

  Bertsekas, Dimitri P ^a   Yu, Huizhen ^b  

^a Department of Electrical Engineering and Computer Science   (United States)

^b UNIVERSITY OF HELSINKI   (Finland)

Author keywords

Dynamic programming; Jacobi method; Linear equations; Projected equations; Simulation; Temporal differences; Value iteration

Indexed keywords

APPROXIMATION ALGORITHMS; DIFFERENCE EQUATIONS; DYNAMIC PROGRAMMING; LINEAR SYSTEMS; SYSTEMS ENGINEERING;

JACOBI METHOD; PROJECTED EQUATIONS; SIMULATION; TEMPORAL DIFFERENCES; VALUE ITERATION;

ITERATIVE METHODS;

EID: 61849106433     PISSN: 03770427     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.cam.2008.07.037     Document Type: Article

References (30)

1. A. Basu, Bhattacharyya, V. Borkar, A learning algorithm for risk-sensitive cost, Tech. Report No. 2006/25, Dept. of Math., Indian Institute of Science, Bangalore, India, 2006
2. Bertsekas D.P., Borkar V., and Nedić A. Improved temporal difference methods with linear function approximation. In: Si J., Barto A., and Powell W. (Eds). Learning and Approximate Dynamic Programming (2004), IEEE Press, NY
3. Barto A.G., and Duff M. Monte Carlo matrix inversion and reinforcement learning. Advances in Neural Information Processing Systems 6 (1994) 687-694
4. D.P. Bertsekas, S. Ioffe, Temporal differences-based policy iteration and applications in neuro-dynamic programming, Lab. for Info. and Dec. Sys. Report LIDS-P-2349, MIT, Cambridge, MA, 1996
5. D.P. Bertsekas, H. Yu, Solution of large systems of equations using approximate dynamic programming methods, Lab. for Info. and Dec. Sys. Report LIDS-2754, MIT, Cambridge, MA, 2007
6. Bertsekas D.P. Dynamic Programming and Optimal Control, Vol. II. 3rd ed. (2007), Athena Scientific, Belmont, MA
7. Bertsekas D.P., and Tsitsiklis J.N. Neuro-Dynamic Programming (1996), Athena Scientific, Belmont, MA
8. Boyan J.A. Technical update: Least-squares temporal difference learning. Machine Learning 49 (2002) 1-15
9. Bradtke S.J., and Barto A.G. Linear least-squares algorithms for temporal difference learning. Machine Learning 22 (1996) 33-57
10. Choi D.S., and Van Roy B. A generalized Kalman filter for fixed point approximation and efficient temporal-difference learning. Discrete Event Dynamic Systems: Theory and Applications 16 (2006) 207-239
11. Curtiss J.H. Monte Carlo methods for the iteration of linear operators. Journal of Mathematics and Physics 32 (1953) 209-232
12. Curtiss J.H. A theoretical comparison of the efficiencies of two classical methods and a Monte Carlo method for computing one component of the solution of a set of linear algebraic equations. In: Meyer H.A. (Ed). Symposium on Monte Carlo Methods (1954), Wiley, New York, NY 191-233
13. Forsythe G.E., and Leibler R.A. Matrix inversion by a Monte Carlo method. Mathematical Tables and Other Aids to Computation 4 (1950) 127-129
14. Halton J.H. A retrospective and prospective survey of the Monte Carlo method. SIAM Review 12 (1970) 1-63
15. P. Keller, S. Mannor, D. Precup, Automatic basis function construction for approximate dynamic programming and reinforcement learning, in: Proceedings of the Twenty-third International Conference on Machine Learning, 2006
16. Liu J.S. Monte Carlo Strategies in Scientific Computing (2001), Springer, NY
17. Menache I., Mannor S., and Shimkin N. Basis function adaptation in temporal difference reinforcement learning. Annals of Operations Research 134 (2005)
18. Nedić A., and Bertsekas D.P. Least squares policy evaluation algorithms with linear function approximation. Discrete Event Dynamic Systems: Theory and Applications 13 (2003) 79-110
19. R. Parr, C. Painter-Wakefield, L. Li, M. Littman, Analyzing feature generation for value-function approximation, in: Proc. of the 24th International Conference on Machine Learning, Corvallis, OR. 2007
20. Puterman M.L. Markov Decision Processes: Discrete Stochastic Dynamic Programming (1994), J. Wiley, NY
21. Saad Y. Iterative Methods for Sparse Linear Systems. 2nd ed. (2003), SIAM, Phila., PA
22. Sutton R.S., and Barto A.G. Reinforcement Learning (1998), MIT Press, Cambridge, MA
23. Sutton R.S. Learning to predict by the methods of temporal differences. Machine Learning 3 (1988) 9-44
24. Tsitsiklis J.N., and Van Roy B. An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control 42 (1997) 674-690
25. Tsitsiklis J.N., and Van Roy B. Average cost temporal-difference learning. Automatica 35 (1999) 1799-1808
26. Tsitsiklis J.N., and Van Roy B. Optimal stopping of Markov processes: Hilbert space theory, approximation algorithms, and an application to pricing financial derivatives. IEEE Transactions on Automatic Control 44 (1999) 1840-1851
27. M.J. Valenti, Approximate dynamic programming with applications in multi-agent systems, Ph.D. Thesis, Dept. of Electrical Engineering and Computer Science, MIT, 2007
28. Wasow W.R. A note on the inversion of matrices by random walks. Mathematical Tables and Other Aids to Computation 6 (1952) 78-81
29. H. Yu, D.P. Bertsekas, Convergence results for some temporal difference methods based on least squares, Lab. for Info. and Dec. Sys. Report 2697, MIT, 2006
30. H. Yu, D.P. Bertsekas, A least squares Q-learning algorithm for optimal stopping problems, Lab. for Info. and Dec. Sys. Report 2731, MIT, 2007