Automatic induction of Bellman-error features for probabilistic planning

Journal of Artificial Intelligence Research

Volumn 38, Issue , 2010, Pages 687-755

  Wu, Jia Hong ^a   Givan, Robert ^a  

^a Purdue University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AUTOMATIC INDUCTION; BELLMAN EQUATIONS; BELLMAN ERROR; DECISION-THEORETIC; DOMAIN SPECIFIC; ERROR FEATURE; FEATURE LANGUAGE; FEATURE SETS; FIRST DOMAIN; HUMAN EXPERT; HUMAN INTERVENTION; HYPOTHESIS SPACE; LEARNING METHODS; MACHINE-LEARNING; PROBABILISTIC PLANNING; PROBLEM STRUCTURE; REAL WORLD DOMAIN; RELATIONAL FEATURE SPACES; STATE FEATURE; STATE-SPACE; TRAINING SETS; VALUE ITERATION;

EQUATIONS OF STATE; HEURISTIC ALGORITHMS; ITERATIVE METHODS; LEARNING SYSTEMS;

FEATURE EXTRACTION;

EID: 77957872063     PISSN: None     EISSN: 10769757     Source Type: Journal    
DOI: 10.1613/jair.3021     Document Type: Article

References (63)

1. Bacchus, F., & Kabanza, F. (2000). Using temporal logics to express search control knowledge for planning. Artificial Intelligence, 116, 123-191.
2. Bertsekas, D. P. (1995). Dynamic Programming and Optimal Control. Athena Scientific.
3. Bertsekas, D. P., & Tsitsiklis, J. N. (1996). Neuro-Dynamic Programming. Athena Scientific.
4. Blockeel, H., & De Raedt, L. (1998). Top-down induction of first-order logical decision trees. Artificial Intelligence, 101, 285-297.
5. Bonet, B., & Givan, R. (2006). Non-deterministic planning track of the 2006 international planning competition. Website. http://www.ldc.usb.ve/ bonet/ipc5/.
6. Boutilier, C., Reiter, R., & Price, B. (2001). Symbolic dynamic programming for first-order MDPs. In Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence, pp. 690-700.
7. Chandra, A., & Merlin, P. (1977). Optimal implementation of conjunctive queries in relational data bases. In Proceedings of the Ninth Annual ACM Symposium on Theory of Computing, pp. 77-90.
8. Davis, R., & Lenat, D. (1982). Knowledge-Based Systems in Artificial Intelligence. McGraw-Hill, New York.
9. Driessens, K., & Džeroski, S. (2004). Integrating guidance into relational reinforcement learning. Machine Learning, 57, 271-304.
10. Driessens, K., Ramon, J., & Gärtner, T. (2006). Graph kernels and gaussian processes for relational reinforcement learning. Machine Learning, 64, 91-119.
11. Džeroski, S., DeRaedt, L., & Driessens, K. (2001). Relational reinforcement learning. Machine Learning, 43, 7-52. (Pubitemid 32286614)
12. Dzeroski, S., Todorovski, L., & Urbancic, T. (1995). Handling real numbers in ILP: A step towards better behavioural clones. In Proceedings of the Eighth European Conference on Machine Learning, pp. 283-286.
13. Estlin, T. A., & Mooney, R. J. (1997). Learning to improve both efficiency and quality of planning. In Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence, pp. 1227-1232.
14. Fahlman, S., & Lebiere, C. (1990). The cascade-correlation learning architecture. In Advances in Neural Information Processing Systems 2, pp. 524 - 532.
15. Farias, V. F., & Van Roy, B. (2004). Tetris: A study of randomized constraint sampling. In Probabilistic and Randomized Methods for Design Under Uncertainty. Springer-Verlag.
16. Fawcett, T. (1996). Knowledge-based feature discovery for evaluation functions. Computational Intelligence, 12(1), 42-64.
17. Fern, A., Yoon, S., & Givan, R. (2004). Learning domain-specific control knowledge from random walks. In Proceedings of the Fourteenth International Conference on Automated Planning and Scheduling, pp. 191-199.
18. Fern, A., Yoon, S., & Givan, R. (2006). Approximate policy iteration with a policy language bias: Solving relational Markov decision processes. Journal of Artificial Intelligence Research, 25, 75-118.
19. Fox,M., & Long, D. (1998). The automatic inference of state invariants in TIM. Journal of Artificial Intelligence Research, 9, 367-421.
20. Gerevini, A., & Schubert, L. (1998). Inferring state constraints for domain-independent planning. In Proceedings of the Fifteenth National Conference on Artificial Intelligence, pp. 905-912.
21. Gordon, G. (1995). Stable function approximation in dynamic programming. In Proceedings of the Twelfth International Conference on Machine Learning, pp. 261-268.
22. Gretton, C., & Thiébaux, S. (2004). Exploiting first-order regression in inductive policy selection. In Proceedings of the Twentieth Conference on Uncertainty in Artificial Intelligence, pp. 217- 225.
23. Guestrin, C., Koller, D., & Parr, R. (2001). Max-norm projections for factored MDPs. In Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence, pp. 673-680.
24. Holldobler, S., Karabaev, E., & Skvortsova, O. (2006). FluCaP: A heuristic search planner for first-order MDPs. Journal of Artificial Intelligence Research, 27, 419-439.
25. Holldobler, S., & Skvortsova, O. (2004). A logic-based approach to dynamic programming. In Proceedings of the Workshop on "Learning and Planning in Markov Processes-Advances and Challenges" at the Nineteenth National Conference on Artificial Intelligence, pp. 31-36.
26. Kakade, S. (2001). A natural policy gradient. In Advances in Neural Information Processing Systems 14, pp. 1531-1538.
27. Kambhampati, S., Katukam, S., & Qu, Y. (1996). Failure driven dynamic search control for partial order planners: An explanation based approach. Artificial Intelligence, 88(1-2), 253-315.
28. Karalic, A., & Bratko, I. (1997). First order regression. Machine Learning, 26, 147-176.
29. Keller, P., Mannor, S., & Precup, D. (2006). Automatic basis function construction for approximate dynamic programming and reinforcement learning. In Proceedings of the Twenty-Third International Conference on Machine Learning, pp. 449-456.
30. Kersting, K., Van Otterlo, M., & De Raedt, L. (2004). Bellman goes relational. In Proceedings of the Twenty-First International Conference on Machine Learning, pp. 465-472.
31. Kersting, K., & Driessens, K. (2008). Non-parametric policy gradients: A unified treatment of propositional and relational domains. In Proceedings of the Twenty-Fifth International Conference on Machine learning, pp. 456-463.
32. Khardon, R. (1999). Learning action strategies for planning domains. Artificial Intelligence, 113(1-2), 125-148.
33. Lagoudakis, M. G., Parr, R., & Littman, M. L. (2002). Least-squares methods in reinforcement learning for control. In SETN 02: Proceedings of the Second Hellenic Conference on AI, pp. 249-260.
34. Mahadevan, S., & Maggioni, M. (2007). Proto-value functions: A Laplacian framework for learning representation and control in Markov decision processes. Journal of Machine Learning Research, 8, 2169-2231.
35. Martin, M., & Geffner, H. (2004). Learning generalized policies from planning examples using concept languages. Applied Intelligence, 20, 9-19.
36. Mitchell, T. M. (1997). Machine Learning. McGraw-Hill.
37. Muggleton, S. (1991). Inductive logic programming. New Generation Computing, 8(4), 295-318.
38. Parr, R., Li, L., Taylor, G., Painter-Wakefield, C., & Littman, M. (2008). An analysis of linear models, linear value-function approximation, and feature selection for reinforcement learning. In Proceedings of the Twenty-Fifth International Conference on Machine Learning, pp. 752- 759.
39. Parr, R., Painter-Wakefield, C., Li, L., & Littman, M. (2007). Analyzing feature generation for value-function approximation. In Proceedings of the Twenty-Fourth International Conference on Machine Learning, pp. 737-744.
40. Patrascu, R., Poupart, P., Schuurmans, D., Boutilier, C., & Guestrin, C. (2002). Greedy linear valueapproximation for factored Markov decision processes. In Proceedings of the Eighteenth National Conference on Artificial Intelligence, pp. 285-291.
41. Petrik, M. (2007). An analysis of Laplacian methods for value function approximation in MDPs. In Proceedings of the twentith International Joint Conference on Artificial Intelligence, pp. 2574-2579.
42. Quinlan, J. R. (1993). C4.5: Programs for Machine Learning. Morgan Kaufmann.
43. Quinlan, J. R. (1996). Learning first-order definitions of functions. Journal of Artificial Intelligence Research, 5, 139-161.
44. Rivest, F., & Precup, D. (2003). Combining TD-learning with cascade-correlation networks. In Proceedings of the Twentieth International Conference on Machine Learning, pp. 632-639.
45. Sanner, S., & Boutilier, C. (2006). Practical linear value-approximation techniques for first-order MDPs. In Proceedings of the Twenty-Second Conference on Uncertainty in Artificial Intelligence, pp. 409-417.
46. Sanner, S., & Boutilier, C. (2009). Practical solution techniques for first-order MDPs. Artificial Intelligence, 173(5-6), 748-788.
47. Singh, S., Jaakkola, T., Littman, M., & Szepesvari, C. (2000). Convergence results for single-step on-policy reinforcement-learning algorithms. Machine Learning, 38(3), 287-308.
48. Sutton, R. S. (1988). Learning to predict by the methods of temporal differences. Machine Learning, 3, 9-44.
49. Sutton, R. S., & Barto, A. G. (1998). Reinforcement Learning: An Introduction. MIT Press.
50. Szita, I., & Lorincz, A. (2006). Learning tetris using the noisy cross-entropy method. Neural Computation, 18, 2936-2941.
51. Tesauro, G. (1995). Temporal difference learning and TD-Gammon. Communications of the ACM, 38(3), 58-68.
52. Tsitsiklis, J., & Roy, B. V. (1997). An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control, 42(5), 674-690.
53. Utgoff, P. E., & Precup, D. (1997). Relative value function approximation. Tech. rep., University of Massachusetts, Department of Computer Science.
54. Utgoff, P. E., & Precup, D. (1998). Constuctive function approximation. In Motoda, & Liu (Eds.), Feature Extraction, Construction, and Selection: A Data-Mining Perspective, pp. 219-235. Kluwer.
55. Veloso, M., Carbonell, J., Perez, A., Borrajo, D., Fink, E., & Blythe, J. (1995). Integrating planning and learning: The PRODIGY architecture. Journal of Experimental and Theoretical AI, 7(1), 81-120.
56. Widrow, B., & Hoff, Jr, M. E. (1960). Adaptive switching circuits. IRE WESCON Convention Record, 96-104.
57. Williams, R. J., & Baird, L. C. (1993). Tight performance bounds on greedy policies based on imperfect value functions. Tech. rep., Northeastern University.
58. Wu, J., & Givan, R. (2007). Discovering relational domain features for probabilistic planning. In Proceedings of the Seventeenth International Conference on Automated Planning and Scheduling, pp. 344-351.
59. Wu, J., Kalyanam, R., & Givan, R. (2008). Stochastic enforced hill-climbing. In Proceedings of the Eighteenth International Conference on Automated Planning and Scheduling, pp. 396-403.
60. Wu, J., & Givan, R. (2005). Feature-discovering approximate value iteration methods. In Proceedings of the Symposium on Abstraction, Reformulation, and Approximation, pp. 321-331.
61. Yoon, S., Fern, A., & Givan, R. (2002). Inductive policy selection for first-order MDPs. In Proceedings of the Eighteenth Conference on Uncertainty in Artificial Intelligence, pp. 568-576.
62. Yoon, S., Fern, A., & Givan, R. (2007). FF-Replan: A baseline for probabilistic planning. In Proceedings of the Seventeenth International Conference on Automated Planning and Scheduling, pp. 352-358.
63. Younes, H., Littman, M., Weissman, D., & Asmuth, J. (2005). The first probabilistic track of the international planning competition. Journal of Artificial Intelligence Research, 24, 851-887.