A critical assessment of benchmark comparison in planning

Journal of Artificial Intelligence Research

Volumn 17, Issue , 2002, Pages 1-33

  Howe, Adele E ^a   Dahlman, Eric ^a  

^a Colorado State University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; ARTIFICIAL INTELLIGENCE; BENCHMARKING; COMPETITION; RESEARCH; ROBOTICS; SCHEDULING;

ARTIFICIAL INTELLIGENCE PLANNING AND SCHEDULING (AIPS); PLANNERS; RESEARCH RESOURCES; RESEARCHERS;

PLANNING;

EID: 9444238479     PISSN: 10769757     EISSN: None     Source Type: Journal    
DOI: 10.1613/jair.935     Document Type: Article

References (44)

1. Bacchus, F. (2000). AIPS-2000 planning competition. http://www.cs.toronto.edu/aips2000/SelfContainedAIPS-2000.ppt.
2. Barrett, A., Golden, K., Penberthy, S., & Weld, D. (1993). UCPOP User's Manual. Dept. of Computer Science and Engineering, University of Washington, Seattle, WA. TR 93-09-06.
3. Blum, A. L., & Furst, M. L. (1997). Fast planning through planning graph analysis. Artificial Intelligence Journal, 90(1-2), 225-279.
4. Bonet, B., & Geffner, H. (1999). Planning as heuristic search: New results. In Proceedings of the Fifth European Conference on Planning (ECP-99) Durham, UK.
5. Chien, S., Kambhampati, S., & Knoblock, C. A. (Eds.)(2000). Proceedings of the Fifth International Conference on Artificial Intelligence Planning and Scheduling (AIPS 2000). AAAI Press, Breckenridge, CO.
6. Cohen, P. R. (1991). A survey of the eighth national conference on artificial intelligence: Pulling together or pulling apart? AI Magazine, 12(1), 16-41.
7. Cohen, P. R. (1995). Empirical Methods for Artificial Intelligence. MIT Press.
8. Drummond, M. E., Kaelbling, L. P., & Rosenschein, S. J. (1990). Collected notes from the benchmarks and metrics workshop. Artificial intelligence branch FIA-91-06, NASA Ames Research Center.
9. Engelhardt, B., Chien, S., Barrett, T., Willis, J., & Wilklow, C. (2001). The data-chaser and citizen explorer benchmark problem sets. In Proceedings of the Sixth European Conference on Planning (ECP 01) Toledo, Spain.
10. Estlin, T. A., & Mooney, R. J. (1997). Learning to improve both efficicency and quality of planning. In Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence, pp. 1227-1233, Nagoya, Japan.
11. Fox, M., & Long, D. (1999). The efficient implementation of the plan-graph in STAN. Journal of Artificial Intelligence Research, 10, 87-115.
12. Fox, M., & Long, D. (2002). PDDL2.1: An extension to PDDL for expressing temporal planning domains. Available at http://www.dur.ac.uk/d.p.long/pddl2.ps.gz.
13. Genevini, A., & Schubert, L. (1996). Accelerating partial-order planners: Some techniques for effective search control and pruning. Journal of Artificial Intelligence Research, 5, 95-137.
14. Hanks, S., Nguyen, D., & Thomas, C. (1993). A beginner's guide to the truckworld simulator. Dept. of Computer Science and Engineering UW-CSE-TR 93-06-09, University of Washington.
15. Hanks, S., Pollack, M. E., & Cohen, P. R. (1994). Benchmarks, test beds, controlled experimentation and the design of agent architectures. AI Magazine, 17-42.
16. Haslum, P., & Geffner, H. (2000). Admissible heuristics for optimal planning. In Proceedings of the Fifth International Conference on Artificial Intelligence Planning and Scheduling (AIPS 2000), pp. 140-149, Breckenridge, CO. AAAI Press.
17. Helmert, M. (2001). On the complexity of planning in transportation domains. In 6th European Conference on Planning (ECP'01), Lecture Notes in Artificial Intelligence, New York, Springer-Verlag.
18. Hoffmann, J. (2001). Local search topology in planning benchmarks: An empirical analysis. In Proceedings of the 17th International Joint Conference on Artificial Intelligence Seattle, WA, USA.
19. Hoffmann, J., & Nebel, B. (2001). The FF planning system: Fast plan generation through heuristic search. Journal of Artificial Intelligence Research, 14, 253-302.
20. Howe, A. E., Dahlman, E., Hansen, C., Scheetz, M., & von Mayrhauser, A. (1999). Exploiting competitive planner performance. In Proceedings of the Fifth European Conference on Planning, Durham, UK.
21. Howe, A. E., von Mayrhauser, A., & Mraz, R. T. (1997). Test case generation as an AI planning problem. Automated Software Engineering, 4(1), 77-106.
22. Joslin, D., & Pollack, M. (1994). Least-cost flaw repair: A plan refinement strategy for partial-order planning. In Proceedings of the Twelfth National Conference on Artificial Intelligence, pp. 1004-1009, Seattle, WA.
23. Kautz, H., & Selman, B. (1998). BLACKBOX: A new approach to the application of theorem proving to problem solving. In Working notes of the AIPS98 Workshop on Planning as Combinatorial Search, Pittsburgh, PA.
24. Kautz, H. blackbox: a SAT technology planning system. http://www.cs.washington.edu/homes/kautz/blackbox/index.html.
25. Kautz, H., & Selman, B. (1999). Unifying SAT-based and graph-based planning. In Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence, Stockholm, Sweden.
26. Koehler, J. (1999). Handling of conditional effects and negative goals in IPP. Tech. rep. 128, Institute for Computer Science, Albert Ludwigs University, Freiburg, Germany.
27. Koehler, J., Nebel, B., Hoffmann, J., & Dimopoulos, Y. (1997). Extending planning graphs to an ADL subset. In Proceedings of the Fourth European Conference in Planning.
28. McDermott, D., Ghallab, M., Howe, A., Knoblock, C., Ram, A., Veloso, M., Weld, D., & Wilkins, D. (1998). The Planning Domain Definition Language.
29. McDermott, D. (2000). The 1998 AI planning systems competition. AI Magazine, 21(2), 35-56.
30. Penberthy, J. S., & Weld, D. S. (1992). UCPOP: a sound, complete, partial order planner for adl. In Proceedings of the Third International Conference on Knowledge Representation and Reasoning, pp. 103-114.
31. Perez, M. A. (1995). Learning Search Control Knowledge to Improve Plan Quality. Ph.D. thesis, Carnegie-Mellon University.
32. Piccinocchi, S., Ceccarelli, M., Piloni, F., & Bicchi, A. (1997). Interactive benchmark for planning algorithms on the web. In Proceedings of IEEE International Conference on Robotics and Automation.
33. Pollack, M. E., & Ringuette, M. (1990). Introducing the Tileworld: Experimentally evaluating agent architectures. In Proceedings of the Eight National Conference on Artificial Intelligence, pp. 183-189, Boston, MA.
34. Pollack, M., Joslin, D., & Paolucci, M. (1997). Flaw selection strategies for partial-order planning. Journal of Artificial Intelligence Research, 6, 223-262.
35. Rabideau, G., Englehardt, B., & Chien, S. (2000). Using generic prferences to incrementally improve plan quality. In Proceedings of the Fifth International Conference on Artificial Intelligence Planning and Scheduling (AIPS 2000), Breckenridge, CO.
36. Slaney, J., & Thiebaux, S. (2001). Blocks world revisited. Artificial Intelligence Journal, 125 (1-2), 119-153.
37. Srinivasan, R., & Howe, A. E. (1995). Comparison of methods for improving search efficiency in a partial-order planner. In Proceedings of the 14th International Joint Conference on Artificial Intelligence, pp. 1620-1626, Montreal, Canada.
38. Sussman, G. A. (1973). A computational model of skill acquisition. Tech. rep. Memo no. AI-TR-297, MIT AI Lab.
39. The Prodigy Research Group (1992). PRODIGY 4.0; the manual and tutorial. School of Computer Science 92-150, Carnegie Mellon University.
40. Watson, J., Barbulescu, L., Howe, A., & Whitley, L. D. (1999). Algorithm performance and problem structure for flow-shop scheduling. In Proceedings of the Sixteenth National Conference on Artificial Intelligence (AAAI-99), Orlando, FL.
41. Watson, J., Beck, J., Barbulescu, L., Whitley, L. D., & Howe, A. (2001). Toward a descriptive model of local search cost in job-shop scheduling. In Proceedings of Sixth European Conference on Planning (ECP'01), Toledo, Spain.
42. Weld, D., Andersen, C., & Smith, D. (1998). Extending graphplan to handle uncertainty and sensing actions. In Proceedings of the Fifteenth National Conference on Artificial Intelligence Madison, WI.
43. Weld, D. S. (1999). Recent advances in AI planning. AI Magazine, 20(2), 93-122.
44. Wilkins, D. E., & desJardins, M. (2001). A call for knowledge-based planning. AI Magazine, 22(1), 99-115.