A Monte-Carlo AIXI approximation

Journal of Artificial Intelligence Research

Volumn 40, Issue , 2011, Pages 95-142

  Veness, Joel ^a   Ng, Kee Siong ^b   Hutter, Marcus ^b   Uther, William ^b   Silver, David ^c  

^a UNIVERSITY OF NEW SOUTH WALES   (Australia)

^b AUSTRALIAN NATIONAL UNIVERSITY   (Australia)

^c MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CONTEXT TREE WEIGHTING; MONTE CARLO; OPTIMALITY; PRACTICAL ALGORITHMS; REINFORCEMENT LEARNING AGENT; TREE SEARCH ALGORITHM;

APPROXIMATION ALGORITHMS; INTELLIGENT AGENTS; PLANT EXTRACTS; REINFORCEMENT LEARNING;

TREES (MATHEMATICS);

EID: 79956344726     PISSN: None     EISSN: 10769757     Source Type: Journal    
DOI: 10.1613/jair.3125     Document Type: Article

References (72)

1. Auer, P. (2002). Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3, 397-422.
2. Begleiter, R., El-Yaniv, R., & Yona, G. (2004). On prediction using variable order Markov models. Journal of Artificial Intelligence Research, 22, 385-421. (Pubitemid 41525891)
3. Bertsekas, D. P., & Castanon, D. A. (1999). Rollout algorithms for stochastic scheduling problems. Journal of Heuristics, 5(1), 89-108.
4. Blockeel, H., & De Raedt, L. (1998). Top-down induction of first-order logical decision trees. Artificial Intelligence, 101(1-2), 285-297. (Pubitemid 128387397)
5. Boots, B., Siddiqi, S. M., & Gordon, G. J. (2010). Closing the learning-planning loop with predictive state representations. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: Volume 1 - Volume 1, AAMAS '10, pp. 1369-1370 Richland, SC. International Foundation for Autonomous Agents and Multiagent Systems.
6. Brafman, R. I., & Tennenholtz, M. (2003). R-max - a general polynomial time algorithm for nearoptimal reinforcement learning. Journal of Machine Learning Research, 3, 213-231.
7. Cassandra, A. R., Kaelbling, L. P., & Littman, M. L. (1994). Acting optimally in partially observable stochastic domains. In AAAI, pp. 1023-1028.
8. Chaslot, G.-B., Winands, M., Uiterwijk, J., van den Herik, H., & Bouzy, B. (2008a). Progressive strategies for Monte-Carlo Tree Search. New Mathematics and Natural Computation, 4(3), 343-357.
9. Chaslot, G. M., Winands, M. H., & Herik, H. J. (2008b). Parallel monte-carlo tree search. In Proceedings of the 6th International Conference on Computers and Games, pp. 60-71 Berlin, Heidelberg. Springer-Verlag.
10. Cover, T. M., & Thomas, J. A. (1991). Elements of information theory. Wiley-Interscience, New York, NY, USA.
11. Farias, V., Moallemi, C., Van Roy, B., & Weissman, T. (2010). Universal reinforcement learning. Information Theory, IEEE Transactions on, 56(5), 2441-2454.
12. Finnsson, H., & Bj̈ornsson, Y. (2008). Simulation-based approach to general game playing. In AAAI, pp. 259-264.
13. Gelly, S., & Silver, D. (2007). Combining online and offline learning in UCT. In Proceedings of the 17th International Conference on Machine Learning, pp. 273-280.
14. Gelly, S., & Wang, Y. (2006). Exploration exploitation in Go: UCT for Monte-Carlo Go. In NIPS Workshop on On-line trading of Exploration and Exploitation.
15. Gelly, S., Wang, Y., Munos, R., & Teytaud, O. (2006). Modification of UCT with patterns in Monte-Carlo Go. Tech. rep. 6062, INRIA, France.
16. Hoehn, B., Southey, F., Holte, R. C., & Bulitko, V. (2005). Effective short-term opponent exploitation in simplified poker. In AAAI, pp. 783-788. (Pubitemid 43006704)
17. Holmes, M. P., & Jr, C. L. I. (2006). Looping suffix tree-based inference of partially observable hidden state. In ICML, pp. 409-416.
18. Hutter, M. (2002a). The fastest and shortest algorithm for all well-defined problems. International Journal of Foundations of Computer Science., 13(3), 431-443.
19. Hutter, M. (2002b). Self-optimizing and Pareto-optimal policies in general environments based on Bayes-mixtures. In Proceedings of the 15th Annual Conference on Computational Learning Theory (COLT 2002), Lecture Notes in Artificial Intelligence. Springer.
20. Hutter, M. (2005). Universal Artificial Intelligence: Sequential Decisions Based on Algorithmic Probability. Springer.
21. Hutter, M. (2007). Universal algorithmic intelligence: A mathematical top?down approach. In Artificial General Intelligence, pp. 227-290. Springer, Berlin.
22. Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1995). Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101, 99-134. (Pubitemid 128387390)
23. Kocsis, L., & Szepesvári, C. (2006). Bandit based Monte-Carlo planning. In ECML, pp. 282-293. (Pubitemid 44618839)
24. Kramer, S., & Widmer, G. (2001). Inducing classification and regression trees in first order logic. In Džeroski, S., & Lavrač, N. (Eds.), Relational Data Mining, chap. 6. Springer.
25. Krichevsky, R., & Trofimov, V. (1981). The performance of universal coding. IEEE Transactions on Information Theory, IT-27, 199-207.
26. Kuhn, H. W. (1950). A simplified two-person poker. In Contributions to the Theory of Games, pp. 97-103.
27. Legg, S., & Hutter, M. (2004). Ergodic MDPs admit self-optimising policies. Tech. rep. IDSIA-21-04, Dalle Molle Institute for Artificial Intelligence (IDSIA).
28. Legg, S. (2008). Machine Super Intelligence. Ph.D. thesis, Department of Informatics, University of Lugano.
29. Levin, L. A. (1973). Universal sequential search problems. Problems of Information Transmission, 9, 265-266.
30. Li, M., & Vitányi, P. (2008). An Introduction to Kolmogorov Complexity and Its Applications (Third edition). Springer.
31. Littman, M., Sutton, R., & Singh, S. (2002). Predictive representations of state. In NIPS, pp. 1555-1561.
32. Lloyd, J.W. (2003). Logic for Learning: Learning Comprehensible Theories from Structured Data. Springer.
33. Lloyd, J.W., & Ng, K. S. (2007). Learning modal theories. In Proceedings of the 16th International Conference on Inductive Logic Programming, LNAI 4455, pp. 320-334.
34. Makino, T. (2009). Proto-predictive representation of states with simple recurrent temporaldifference networks. In ICML, pp. 697-704.
35. McCallum, A. K. (1996). Reinforcement Learning with Selective Perception and Hidden State. Ph.D. thesis, University of Rochester.
36. Ng, K. S. (2005). Learning Comprehensible Theories from Structured Data. Ph.D. thesis, The Australian National University.
37. Pankov, S. (2008). A computational approximation to the AIXI model. In AGI, pp. 256-267.
38. Poland, J., & Hutter, M. (2005). Defensive universal learning with experts. In Proc. 16th International Conf. on Algorithmic Learning Theory, Vol. LNAI 3734, pp. 356-370. Springer.
39. Poland, J., & Hutter, M. (2006). Universal learning of repeated matrix games. Tech. rep. 18-05, IDSIA.
40. Poupart, P., & Vlassis, N. (2008). Model-based bayesian reinforcement learning in partially observable domains. In ISAIM.
41. Poupart, P., Vlassis, N., Hoey, J., & Regan, K. (2006). An analytic solution to discrete bayesian reinforcement learning. In ICML '06: Proceedings of the 23rd international conference on Machine learning, pp. 697-704 New York, NY, USA. ACM.
42. Rissanen, J. (1983). A universal data compression system. IEEE Transactions on Information Theory, 29(5), 656-663.
43. Ron, D., Singer, Y., & Tishby, N. (1996). The power of amnesia: Learning probabilistic automata with variable memory length. Machine Learning, 25(2), 117-150.
44. Rosencrantz, M., Gordon, G., & Thrun, S. (2004). Learning low dimensional predictive representations. In Proceedings of the twenty-first International Conference on Machine Learning, p. 88 New York, NY, USA. ACM.
45. Ross, S., Chaib-draa, B., & Pineau, J. (2008). Bayes-adaptive POMDPs. In Platt, J., Koller, D., Singer, Y., & Roweis, S. (Eds.), Advances in Neural Information Processing Systems 20, pp. 1225-1232. MIT Press, Cambridge, MA.
46. Schmidhuber, J., Zhao, J., & Wiering, M. A. (1997). Shifting inductive bias with success-story algorithm, adaptive Levin search, and incremental self-improvement. Machine Learning, 28, 105-130. (Pubitemid 127507171)
47. Schmidhuber, J. (1997). Discovering neural nets with low Kolmogorov complexity and high generalization capability. Neural Networks, 10(5), 857-873. (Pubitemid 27315721)
48. Schmidhuber, J. (2002). The speed prior: A new simplicity measure yielding near-optimal computable predictions. In Proc. 15th Annual Conf. on Computational Learning Theory, pp. 216-228.
49. Schmidhuber, J. (2003). Bias-optimal incremental problem solving. In Advances in Neural Information Processing Systems 15, pp. 1571-1578. MIT Press.
50. Schmidhuber, J. (2004). Optimal ordered problem solver. Machine Learning, 54, 211-254.
51. Shani, G. (2007). Learning and Solving Partially Observable Markov Decision Processes. Ph.D. thesis, Ben-Gurion University of the Negev.
52. Shani, G., & Brafman, R. (2004). Resolving perceptual aliasing in the presence of noisy sensors. In NIPS.
53. Silver, D., & Tesauro, G. (2009). Monte-carlo simulation balancing. In ICML '09: Proceedings of the 26th Annual International Conference on Machine Learning, pp. 945-952 New York, NY, USA. ACM.
54. Silver, D., & Veness, J. (2010). Monte-Carlo Planning in Large POMDPs. In Advances in Neural Information Processing Systems (NIPS). To appear.
55. Singh, S., James, M., & Rudary, M. (2004). Predictive state representations: A new theory for modeling dynamical systems. In UAI, pp. 512-519.
56. Solomonoff, R. J. (1964). A formal theory of inductive inference: Parts 1 and 2. Information and Control, 7, 1-22 and 224-254.
57. Strehl, A. L., Li, L., & Littman, M. L. (2009). Reinforcement learning in finite MDPs: PAC analysis. Journal of Machine Learning Research, 10, 2413-2444.
58. Strehl, A. L., Li, L., Wiewiora, E., Langford, J., & Littman, M. L. (2006). PAC model-free reinforcement learning. In ICML '06: Proceedings of the 23rd international conference on Machine learning, pp. 881-888 New York, NY, USA. ACM.
59. Strens, M. (2000). A Bayesian framework for reinforcement learning. In ICML, pp. 943-950.
60. Suematsu, N., & Hayashi, A. (1999). A reinforcement learning algorithm in partially observable environments using short-term memory. In NIPS, pp. 1059-1065.
61. Suematsu, N., Hayashi, A., & Li, S. (1997). A Bayesian approach to model learning in non- Markovian environment. In ICML, pp. 349-357.
62. Sutton, R. S., & Barto, A. G. (1998). Reinforcement Learning: An Introduction. MIT Press.
63. Sutton, R. S., & Tanner, B. (2004). Temporal-difference networks. In NIPS.
64. Tjalkens, T. J., Shtarkov, Y. M., &Willems, F. M. J. (1993). Context tree weighting: Multi-alphabet sources. In Proceedings of the 14th Symposium on Information Theory Benelux.
65. Veness, J., Ng, K. S., Hutter, M., & Silver, D. (2010). Reinforcement Learning via AIXI Approximation. In Proceedings of the Conference for the Association for the Advancement of Artificial Intelligence (AAAI).
66. Veness, J., Silver, D., Uther, W., & Blair, A. (2009). Bootstrapping from Game Tree Search. In Neural Information Processing Systems (NIPS).
67. Wang, T., Lizotte, D. J., Bowling, M. H., & Schuurmans, D. (2005). Bayesian sparse sampling for on-line reward optimization. In ICML, pp. 956-963.
68. Watkins, C., & Dayan, P. (1992). Q-learning. Machine Learning, 8, 279-292.
69. Willems, F., Shtarkov, Y., & Tjalkens, T. (1997). Reflections on "The Context Tree Weighting Method: Basic properties". Newsletter of the IEEE Information Theory Society, 47(1).
70. Willems, F. M. J. (1998). The context-tree weighting method: Extensions. IEEE Transactions on Information Theory, 44, 792-798. (Pubitemid 128737641)
71. Willems, F. M. J., Shtarkov, Y. M., & Tjalkens, T. J. (1996). Context weighting for general finitecontext sources. IEEE Trans. Inform. Theory, 42, 42-1514.
72. Willems, F. M., Shtarkov, Y. M., & Tjalkens, T. J. (1995). The context tree weighting method: Basic properties. IEEE Transactions on Information Theory, 41, 653-664.