Automatic complexity reduction in reinforcement learning

Computational Intelligence

Volumn 26, Issue 1, 2010, Pages 1-25

  Chiu, Chung Cheng ^a   Soo, Von Wun ^a,b  

^a NATIONAL TSING HUA UNIVERSITY   (Taiwan)

^b NATIONAL UNIVERSITY OF KAOHSIUNG   (Taiwan)

Author keywords

Complexity reduction; Reinforcement learning; Spectral graph theory

Indexed keywords

ACTION SEQUENCES; COMPLEXITY REDUCTION; DECOMPOSITION ALGORITHM; DIMENSION REDUCTION; DYNAMIC STATE; HIGH DIMENSIONALITY; LEARNING PROCESS; NORMALIZED GRAPH LAPLACIAN; SCALE-UP; SPECTRAL ANALYSIS; SPECTRAL GRAPH THEORY; STATE REPRESENTATION; STATE SPACE; SUB-PROBLEMS; SUBTASKS;

GRAPH THEORY; REINFORCEMENT LEARNING; SPECTRUM ANALYSIS; SPECTRUM ANALYZERS;

REINFORCEMENT;

EID: 76349120621     PISSN: 08247935     EISSN: 14678640     Source Type: Journal    
DOI: 10.1111/j.1467-8640.2009.00350.x     Document Type: Article

References (23)

1. Ahuja, R. K., T. L. Magnanti, and J. B. Orlin. 1993. Network Flows: Theory, Algorithms, and Applications. Prentice-Hall, Englewood Cliffs, Upper Saddle River, NJ.
2. Andre, D., and S. Russell. 2001. Programmable reinforcement learning agents. In Advances in Neural Information Processing Systems 13. MIT Press, Cambridge, MA, pp. 1019 1025.
3. Barto, A. G., and S. Mahadevan. 2003. Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems, 13 : 341 379.
4. Cheeger, J. 1970. A lower bound for the smallest eigenvalue of the Laplacian. In Problems in Analysis (Papers Dedicated to Salomon Bochner, 1969). Princeton University Press, Princeton, NJ, pp. 195 199.
5. Chung, F. 1997, February. Spectral Graph Theory (CBMS Regional Conference Series in Mathematics, No. 92) (CBMS Regional Conference Series in Mathematics). American Mathematical Society.
6. Dietterich, T. 2000. Hierarchical reinforcement learning with the MAXQ value function decomposition. Journal of Artificial Intelligence Research, 13 : 227 303.
7. Digney, B. 1998. Learning hierarchical control structure for multiple tasks and changing environments. In Proceedings of the Fifth Conference on the Simulation of Adaptive Behavior. MIT Press, Cambridge, MA, pp. 321 330.
8. Hengst, B. 2002. Discovering hierarchy in reinforcement learning with HEXQ. In Proceedings of the Nineteenth International Conference on Machine Learning. Morgan Kaufmann, San Francisco, pp. 243 250.
9. Jong, N. K., and P. Stone. 2005. Abstraction discovery from irrelevant state variables. In Proceedings of the Nineteenth International Joint Conference on Artificial Intelligence. Professional Book Center, Denver, CO, pp. 752 757.
10. Jonsson, A., and A. Barto. 2006. Causal graph based decomposition of factored MDPs. Journal of Machine Learning Research, 7 : 2259 2301.
11. Langley, P., and D. Choi. 2006. Learning recursive control programs from problem solving. Journal of Machine Learning Research, 7 : 439 518.
12. Mahadevan, S., and M. Maggioni. 2007. Proto-value functions: A Laplacian framework for learning representation and control in Markov decision processes. Journal of Machine Learning Research, 8 : 2169 2231.
13. Mannor, S., I. Menache, A. Hoze, and U. Klein. 2004. Dynamic abstraction in reinforcement learning via clustering. In Proceedings of the Twenty-First International Conference on Machine Learning. ACM Press, New York, pp. 560 567.
14. McGovern, A., and A. G. Barto. 2001. Automatic discovery of subgoals in reinforcement learning using diverse density. In Proceedings of the Eighteenth International Conference on Machine Learning. Morgan Kaufmann, San Franciso, pp. 361 368.
15. Mehta, N., S. Ray, P. Tadepalli, and T. Dietterich. 2008. Automatic discovery and transfer of MAXQ hierarchies. In Proceedings of the Twenty-Fifth International Conference on Machine Learning. ACM Press, New York, pp. 648 655.
16. Menache, I., S. Mannor, and N. Shimkin. 2002. Q-Cut-dynamic discovery of sub-goals in reinforcement learning. In Proceedings of the Thirteenth European Conference on Machine Learning. LNAI 2430. Springer, Heidelberg, pp. 295 306.
17. Parr, R., and S. Russell. 1998. Reinforcement learning with hierarchies of machines. In Advances in Neural Information Processing Systems 10, MIT Press, Cambridge, MA, pp. 1043 1049.
18. Parr, R., C. Painter-Wakefield, L. Li, and M. Littman. 2007. Analyzing feature generation for value-function approximation. In Proceedings of the Twenty-Fourth International Conference on Machine Learning. ACM Press, New York, pp. 737 744.
19. Shi, J., and J. Malik. 2000. Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22 : 888 905.
20. Şimşek, Ö., A. P. Wolfe, and A.G. Barto. 2005. Identifying useful subgoals in reinforcement learning by local graph partitioning. In Proceedings of the Twenty-Second International Conference on Machine Learning. ACM Press, New York, pp. 816 823.
21. Şimşek, Ö., and A. G. Barto. 2004. Using relative novelty to identify useful temporal abstractions in reinforcement learning. In Proceedings of the Twenty-First International Conference on Machine Learning. ACM Press, New York, pp. 751 758.
22. Sutton, R. S., D. Precup, and S. P. Singh. 1999. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112 : 181 211.
23. Wymann, B., and E. Espie. 2000. TORCS, The Open Racing Car Simulator. Software available at http://torcs.sourceforge.net/.