Approximate modified policy iteration and its application to the game of tetris

Journal of Machine Learning Research

Volumn 16, Issue , 2015, Pages 1629-1676

  Scherrer, Bruno ^a   Ghavamzadeh, Mohammad ^b   Gabillon, Victor ^c   Lesner, Boris ^a   Geist, Matthieu ^d  

^a INRIA   (France)

^b ADOBE RESEARCH   (United States)

^c INRIA   (France)

^d CNRS   (France)

Author keywords

Approximate dynamic programming; Finite sample analysis; Game of tetris; Markov decision processes; Performance bounds; Reinforcement learning

Indexed keywords

ALGORITHMS; APPROXIMATION ALGORITHMS; DYNAMIC PROGRAMMING; MARKOV PROCESSES; PARAMETER ESTIMATION; REINFORCEMENT LEARNING; SAMPLING;

APPROXIMATE DYNAMIC PROGRAMMING; FINITE SAMPLES; GAME OF TETRIS; MARKOV DECISION PROCESSES; PERFORMANCE BOUNDS;

ITERATIVE METHODS;

EID: 84962317462     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (48)

1. A. Antos, R. Munos, and Cs. Szepesvári. Fitted q-iteration in continuous action-space MDPs. In Proceedings of the Advances in Neural Information Processing Systems 19, pages 9-16, 2007.
2. D. Bertsekas and S. Ioffe. Temporal differences-based policy iteration and applications in neuro-dynamic programming. Technical report, MIT, 1996.
3. D. Bertsekas and J. Tsitsiklis. Neuro-dynamic programming. Athena Scientific, 1996.
4. H. Burgiel. How to lose at tetris. Mathematical Gazette, 81:194-200, 1997.
5. Pelin Canbolat and Uriel Rothblum. (Approximate) iterated successive approximations algorithm for sequential decision processes. Annals of Operations Research, pages 1-12, 2012. ISSN 0254-5330.
6. C. Chang and C. Lin. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2(3):27:1-27:27, May 2011. ISSN 2157-6904. doi: 10.1145/1961189.1961199. URL http://doi.acm.org/10.1145/1961189.1961199.
7. E. Demaine, S. Hohenberger, and D. Liben-Nowell. Tetris is hard, even to approximate. In Proceedings of the Ninth International Computing and Combinatorics Conference, pages 351-363, 2003.
8. C. Dimitrakakis and M. Lagoudakis. Rollout sampling approximate policy iteration. Machine Learning Journal, 72(3):157-171, 2008.
9. D. Ernst, P. Geurts, and L. Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503-556, 2005.
10. C. Fahey. Tetris AI, computer plays tetris, 2003. http://colinfahey.com/tetris/tetris.html.
11. A. Farahmand, R. Munos, and Cs. Szepesvári. Error propagation for approximate policy and value iteration. In Proceedings of the Advances in Neural Information Processing Systems 22, pages 568-576, 2010.
12. V. Farias and B. Van Roy. Tetris: A study of randomized constraint sampling. Springer-Verlag, 2006.
13. A. Fern, S. Yoon, and R. Givan. Approximate policy iteration with a policy language bias: Solving relational Markov decision processes. Journal of Artificial Intelligence Research, 25:75-118, 2006.
14. T. Furmston and D. Barber. A unifying perspective of parametric policy search methods for Markov decision processes. In Proceedings of the Advances in Neural Information Processing Systems 24, pages 2726-2734, 2012.
15. V. Gabillon, A. Lazaric, M. Ghavamzadeh, and B. Scherrer. Classification-based policy iteration with a critic. In Proceedings of the Twenty-Eighth International Conference on Machine Learning, pages 1049-1056, 2011.
16. V. Gabillon, M. Ghavamzadeh, and B. Scherrer. Approximate dynamic programming finally performs well in the game of tetris. In Proceedings of Advances in Neural Information Processing Systems 26, pages 1754-1762, 2013.
17. M. Geist and B. Scherrer. Off-policy learning with eligibility traces: A survey. Journal of Machine Learning Research, 14, April 2014.
18. G. J. Gordon. Stable function approximation in dynamic programming. In ICML, pages 261-268, 1995.
19. L. Györfi, M. Kolher, M. Krzyžak, and H. Walk. A distribution-free theory of nonparametric regression. Springer-Verlag, 2002.
20. N. Hansen and A. Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evolutionary Computation, 9:159-195, 2001.
21. S. Kakade. A natural policy gradient. In Proceedings of the Advances in Neural Information Processing Systems 14, pages 1531-1538, 2002.
22. M. Kearns, Y. Mansour, and A. Ng. Approximate planning in large pomdps via reusable trajectories. In Proceedings of the Advances in Neural Information Processing Systems 12, pages 1001-1007. MIT Press, 2000.
23. M. Lagoudakis and R. Parr. Least-squares policy iteration. Journal of Machine Learning Research, 4:1107-1149, 2003a.
24. M. Lagoudakis and R. Parr. Reinforcement learning as classification: Leveraging modern classifiers. In Proceedings of the Twentieth International Conference on Machine Learning, pages 424-431, 2003b.
25. A. Lazaric, M. Ghavamzadeh, and R. Munos. Analysis of a classification-based policy iteration algorithm. Technical Report inria-00482065, INRIA, 2010a.
26. A. Lazaric, M. Ghavamzadeh, and R. Munos. Finite-sample analysis of LSTD. In Proceedings of the Twenty-Seventh International Conference on Machine Learning, pages 615-622, 2010b.
27. A. Lazaric, M. Ghavamzadeh, and R. Munos. Analysis of a classification-based policy iteration algorithm. In Proceedings of the Twenty-Seventh International Conference on Machine Learning, pages 607-614, 2010c.
28. A. Lazaric, M. Ghavamzadeh, and R. Munos. Finite-sample analysis of least-squares policy iteration. Journal of Machine Learning Research, 13:3041-3074, 2012.
29. Boris Lesner and Bruno Scherrer. Tight performance bounds for approximate modified policy iteration with non-stationary policies. CoRR, abs/1304.5610, 2013.
30. R. Munos. Error bounds for approximate policy iteration. In Proceedings of the Twentieth International Conference on Machine Learning, pages 560-567, 2003.
31. p-norm for approximate value iteration. SIAM J. Control and Optimization, 46(2):541-561, 2007.
32. R. Munos and Cs. Szepesvari. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 9:815-857, 2008.
33. D. Precup, R. Sutton, and S. Singh. Eligibility traces for off-policy policy evaluation. In Proceedings of the Seventeenth International Conference on Machine Learning, pages 759-766, 2000.
34. D. Precup, R. Sutton, and S. Dasgupta. Off-policy temporal difference learning with function approximation. In Proceedings of the Eighteenth International Conference on Machine Learning, pages 417-424, 2001.
35. M. Puterman. Markov decision processes. Wiley, New York, 1994.
36. R. Rubinstein and D. Kroese. The cross-entropy method: A unified approach to combinatorial optimization, Monte-Carlo simulation, and machine learning. Springer-Verlag, 2004.
37. B. Scherrer. Performance bounds for λ-policy iteration and application to the game of tetris. Journal of Machine Learning Research, 14:1175-1221, 2013.
38. B. Scherrer and C. Thiéry. Performance bound for approximate optimistic policy iteration. Technical report, INRIA, 2010.
39. B. Scherrer, M. Ghavamzadeh, V. Gabillon, and M. Geist. Approximate modified policy iteration. In Proceedings of the Twenty Ninth International Conference on Machine Learning, pages 1207-1214, 2012.
40. S. Singh and R. Yee. An upper bound on the loss from approximate optimal-value functions. Machine Learning, 16-3:227-233, 1994.
41. Cs. Szepesvári. Reinforcement learning algorithms for mdps. In Wiley Encyclopedia of Operations Research. Wiley, 2010.
42. I. Szita and A. Lorincz. Learning tetris using the noisy cross-entropy method. Neural Computation, 18(12):2936-2941, 2006.
43. C. Thiery and B. Scherrer. Building controllers for tetris. International Computer Games Association Journal, 32:3-11, 2009a. URL http://hal.inria.fr/inria-00418954.
44. C. Thiery and B. Scherrer. Improvements on learning tetris with cross entropy. International Computer Games Association Journal, 32, 2009b. URL http://hal.inria.fr/inria-00418930.
45. C. Thiery and B. Scherrer. Least-squares λ-policy iteration: bias-variance trade-off in control problems. In Proceedings of the Twenty-Seventh International Conference on Machine Learning, pages 1071-1078, 2010a.
46. C. Thiery and B. Scherrer. MDPTetris features documentation, 2010b. http://mdptetris. gforge.inria.fr/doc/feature-functions-8h.html.
47. J. Tsitsiklis and B. Van Roy. Feature-based methods for large scale dynamic programming. Machine Learning, 22:59-94, 1996.
48. J. Tsitsiklis and B. Van Roy. An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control, 42(5):674-690, 1997.