Empirical hardness models: Methodology and a case study on combinatorial auctions

Journal of the ACM

Volumn 56, Issue 4, 2009, Pages

  Leyton Brown, Kevin ^a   Nudelman, Eugene ^b   Shoham, Yoav ^b  

^a UNIVERSITY OF BRITISH COLUMBIA   (Canada)

^b Stanford University   (United States)

Author keywords

Algorithm portfolios; Combinatorial auctions; Empirical analysis of algorithms; Runtime prediction

Indexed keywords

BENCHMARK SUITES; COMBINATORIAL AUCTION; COMBINATORIAL AUCTIONS; EMPIRICAL ANALYSIS OF ALGORITHMS; HARD PROBLEMS; NP COMPLETE PROBLEMS; PROBLEM INSTANCES; RUNNING TIME; RUNTIME PREDICTION; RUNTIMES; SUPERVISED MACHINE LEARNING; WINNER DETERMINATION PROBLEM;

COMBINATORIAL MATHEMATICS; COMMERCE; LEARNING ALGORITHMS; RESEARCH; STANDARDIZATION;

MATHEMATICAL MODELS;

EID: 68549128640     PISSN: 00045411     EISSN: 1557735X     Source Type: Journal    
DOI: 10.1145/1538902.1538906     Document Type: Article

References (84)

1. ACHLIOPTAS, D. 2001. Lower bounds for random 3-SAT via differential equations. Theoret. Comput. Sci. 265, 1-2, 159-185.
2. ACHLIOPTAS, D., BEAME, P., AND MOLLOY, M. 2004. A sharp threshold in proof complexity yields lower bounds for satisfiability search. J. Comput. Syst. Sci. 68, 2, 238-268.
3. ACHLIOPTAS, D., GOMES, C. P., KAUTZ, H. A., AND SELMAN, B. 2000. Generating satisfiable problem instances. In AAAI: Proceedings of the AAAI Conference on Artificial Intelligence. 256-261.
4. ACHLIOPTAS, D., JIA, H., AND MOORE, C. 2005. Hiding satisfying assignments: Two are better than one. J. Artif. Intell. Res. 24, 623-639.
5. ANDERSON, A., TENHUNEN, M., AND YGGE, F. 2000. Integer programming for combinatorial auction winner determination. In ICMAS: Proceedings of the International Conference on Multi-agent Systems. 39-46.
6. BEAME, P., KARP, R., PITASSI, T., AND SAKS, M. 1998. On the complexity of unsatisfiability proofs for random k-CNF formulas. In STOC: Proceedings of the AnnualACM Symposium on Theory of Computing. ACM, New York, 561-571.
7. BREWER, E. 1994. Portable high-performance supercomputing: High-level platform-dependent optimization. Ph.D. dissertation, Massachusetts Institute of Technology, Cambridge, MA.
8. BREWER, E. 1995. High-level optimization via automated statistical modeling. In SIGPLAN: Proceedings of the ACM Symposium on Principles and Practice of Parallel Programming. ACM, New York, 80- 91.
9. CARCHRAE, T., AND BECK, J. C. 2005. Applying machine learning to low knowledge control of optimization algorithms. Computat. Intell. 21, 4, 372-387.
10. CHALONER, K., AND VERDINELLI, I. 1995. Bayesian experimental design: A review. Statist. Sci. 10, 273-304.
11. CHEESEMAN, P., KANEFSKY, B., AND TAYLOR, W. M. 1991. Where the really hard problems are. In IJCAI: Proceedings of the International Joint Conference on Artificial Intelligence. 331-337.
12. COCCO, S., AND MONASSON, R. 2004. Heuristic average-case analysis of the backtrack resolution of random 3-satisfiability instances. Theoret. Comput. Sci. 320, 2-3, 345-372.
13. CRAMTON, P., SHOHAM, Y., AND STEINBERG, R. (EDS), 2006. Combinatorial Auctions. MIT Press, Cambridge.
14. DE FARIAS, D. P., AND MEGIDDO, N. 2004. Howto combine expert (or novice) advice when actions impact the environment. In NIPS: Proceedings of the Neural Information Processing Systems Conference.
15. DE VRIES, S., AND VOHRA, R. 2003. Combinatorial auctions: A survey. INFORMS J. Comput. 15, 3, 284-309.
16. DIAO, Y., ESKESEN, F., PROEHLICH, S., HELLERSTEIN, J., SPAINHOWER, L., AND SURENDRA, M. 2003. Generic online optimization of multiple configuration parameters with application to a database server. Proceedings of the IFIP/IEEE International Workshop on Distributed Systems: Operations and Management.
17. DOUCET, A., DE FREITAS, N., AND GORDEN, N. (EDS.) 2001. Sequential Monte Carlo Methods in Practice. Springer-Verlag.
18. DUBOIS, O., AND BOUFKHAD, Y. 1997. A general upper bound for the satisfiability threshold of random r-SAT formulae. Journal of Algorithms 24, 2, 395-420.
19. DUBOIS, O., BOUFKHAD, Y., AND MANDLER, J. 2000. Typical random 3-SAT formulae and the satisfiability threshold. In SODA: Proceedings of the ACM-SIAM Symposium on Discrete Algorithms. ACM, New York, 126-127.
20. EFRON, B., HASTIE, T., JOHNSTONE, I., AND TIBSHIRANI, R. 2004. Least angle regression. Ann. Stat. 32, 407-451.
21. EPSTEIN, S., FREUDER, E., WALLACE, R., MOROZOV, A., AND SAMUELS, B. 2002. The adaptive constraint engine. In CP: Proceedings of the International Conference on Principles and Practice of Constraint Programming. 525-540.
22. FRANCO, J. 2001. Results related to threshold phenomena research in satisfiability: Lower bounds. Theoret. Comput. Sci. 265, 1-2, 147-157.
23. FRIEDMAN, J. 1991. Multivariate adaptive regression splines. Ann. Stat. 19, 1, 1-141.
24. FRIEZE, A., AND SUEN, S. 1996. Analysis of two simple heuristics on a random instance of k-SAT. J. Algor. 20, 2, 312-355.
25. FUJISHIMA, Y., LEYTON-BROWN, K., AND SHOHAM, Y. 1999. Taming the computational complexity of combinatorial auctions: Optimal and approximate approaches. In IJCAI: Proceedings of the International Joint Conference on Artificial Intelligence. 548-553.
26. GAGLIOLO, M., AND SCHMIDHUBER, J. 2006. Learning dynamic algorithm portfolios. Ann. Mathematics and Artificial Intelligence 47, 3-4, 295-328.
27. GEBRUERS, C., AND GUERRI, A. 2004. Machine learning for portfolio selection using structure at the instance level. In Principles and Practice of Constraint Programming, Doctoral Consortium.
28. GEBRUERS, C., GUERRI, A., HNICH, B., AND MILANO, M. 2004. Making choices using structure at the instance level within a case based reasoning framework. In CPAIOR: Proceedings of the International Conference on Integration of AI and OR Techniques in Constraint Programming for Combinatorial Optimization Problems. 380-386.
29. GEBRUERS, C., HNICH, B., BRIDGE, D., AND FREUDER, E. 2005. Using CBR to select solution strategies in constraint programming. In ICCBR: Proceedings of the International Conference on Case-Based Reasoning. 222-236.
30. GOLDSMITH, S., AIKEN, A., AND WILKERSON, D. 2007. Measuring empirical computational complexity. In Proceedings of the the 6th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering. ACM, New York, 395-404.
31. GOLDSZMIDT, M. 2007. Making life better one large system at a time: Challenges for UAI research. In UAI: Proceedings of the Conference on Uncertainty in Artificial Intelligence.
32. GOMES, C., FERNÁNDEZ, C., SELMAN, B., AND BESSIÈRE, C. 2004. Statistical regimes across constrained-ness regions. In CP: Proceedings of the International Conference on Principles and Practice of Constraint Programming.
33. GOMES, C., AND SELMAN, B. 2001. Algorithm portfolios. Artif. Intell. 126, 1-2, 43-62.
34. GOMES, C., SELMAN, B., CRATO, N., AND KAUTZ, H. 2000. Heavy-tailed phenomena in satisfiability and constraint satisfaction problems. J. Automat. Reason. 24, 1-2, 67-100.
35. GOMES, C. P., AND SELMAN, B. 1997. Problem structure in the presence of perturbations. In AAAI: Proceedings of the AAAI Conference on Artif. Intell. 221-226.
36. GONEN, R., AND LEHMANN, D. 2000. Optimal solutions for multi-unit combinatorial auctions: Branch and bound heuristics. In EC: Proceedings of the ACM Conference on Electronic Commerce. ACM, New York, 13-20.
37. GONEN, R., AND LEHMANN, D. 2001. Linear programming helps solving large multi-unit combinatorial auctions. Tech. Rep. TR-2001-8, Leibniz Center for Research in Computer Science. April.
38. GUO, H., AND HSU, W. 2007. A machine learning approach to algorithm selection for NP-hard optimization problems. Ann. Oper. Res. 156, 1 (Dec.), 61-82.
39. HASTIE, T., TIBSHIRANI, R., AND FRIEDMAN, J. 2001. Elements of Statistical Learning. Springer-Verlang, Berlin, Germany.
40. HOOS, H., AND BOUTILIER, C. 2000. Solving combinatorial auctions using stochastic local search. In AAAI: Proceedings of the AAAI Conference on Artif. Intell. 22-29.
41. HOOS, H. H., AND STÜTZLE, T. 1999. Towards a characterisation of the behaviour of stochastic local search algorithms for SAT. Artif. Intell. 112, 1-2, 213-232.
42. HOOS, H. H., AND STÜTZLE, T. 2004. Stochastic Local Search: Foundations and Applications. Morgan-Kaufmann, Reading.
43. HORVITZ, E., RUAN, Y., GOMES, C., KAUTZ, H., SELMAN, B., AND CHICKERING, M. 2001. A Bayesian approach totackling hard computational problems. In UAI: Proceedings of the Conference on Uncertainty in Artif. Intell. 235-244.
44. HUBERMAN, B., LUKOSE, R., AND HOGG, T. 1997. An economics approach to hard computational problems. Science 265, 51-54.
45. HUTTER, F., HAMADI, Y., HOOS, H. H., AND LEYTON-BROWN, K. 2006. Performance prediction and automated tuning of randomized and parametric algorithms. In CP: Proceedings of the International Conference on Principles and Practice of Constraint Programming. Lecture Notes in Computer Science, vol. 4204. Springer-Verlag, Berlin, Germany, 213-228.
46. JIA, H., MOORE, C., AND SELMAN, B. 2004. From spin glasses to hard satisfiable formulas. In SAT: Proceedings of the International Conference on Theory and Applications of Satisfiability Testing. 199- 210.
47. JIA, H., MOORE, C., AND STRAIN, D. 2007. Generating hard satisfiable formulas by hiding solutions deceptively. J. Artif. Intell. Res. 28, 107-118.
48. KNUTH, D. 1975. Estimating the efficiency of backtrack programs. Math. Comput. 29, 129, 121-136.
49. KOHAVI, R., AND JOHN, G. 1997. Wrappers for feature subset selection. Artif. Intell. 97, 1-2, 273-324.
50. KOLAITIS, P. 2003. Constraint satisfaction, databases and logic. In IJCAI: Proceedings of the International Joint Conference on Artificial Intelligence. 1587-1595.
51. KORF, R., AND REID, M. 1998. Complexity analysis of admissible heuristic search. AAAI: Proceedings of the AAAI Conference on Artificial Intelligence, 305-310.
52. LAGOUDAKIS, M., AND LITTMAN, M. 2000. Algorithm selection using reinforcement learning. In ICML: Proceedings of the International Conference on Machine Learning. 511-518.
53. LAGOUDAKIS, M., AND LITTMAN, M. 2001. Learning to select branching rules in the DPLL procedure for satisfiability. In SAT: Proceedings of the International Conference on Theory and Applications of Satisfiability Testing. 344-359.
54. LEYTON-BROWN, K., NUDELMAN, E., ANDREW, G., MCFADDEN, J., AND SHOHAM, Y. 2003a. Boosting as a metaphor for algorithm design. In CP: Proceedings of the International Conference on Principles and Practice of Constraint Programming. 899-903.
55. LEYTON-BROWN, K., NUDELMAN, E., ANDREW, G., MCFADDEN, J., AND SHOHAM, Y. 2003b. A portfolio approach to algorithm selection. In IJCAI: Proceedings of the International Joint Conference on Artificial Intelligence. 1542-1543.
56. LEYTON-BROWN, K., NUDELMAN, E., AND SHOHAM, Y. 2002. Learning the empirical hardness of optimization problems: The case of combinatorial auctions. In CP: Proceedings of the International Conference on Principles and Practice of Constraint Programming. 556-572.
57. LEYTON-BROWN, K., NUDELMAN, E., AND SHOHAM, Y. 2006. Empirical hardness models for combinatorial auctions. In Combinatorial Auctions. MIT Press, Cambridge, Chap. 19, 479-504.
58. LEYTON-BROWN, K., PEARSON, M., AND SHOHAM, Y. 2000a. Towards a universal test suite for combinatorial auction algorithms. In EC: Proceedings of the ACM Conference on Electronic Commerce. ACM, New York, 66-76.
59. LEYTON-BROWN, K., SHOHAM, Y., AND TENNENHOLTZ, M. 2000b. An algorithm for multi-unit combinatorial auctions. In AAAI: Proceedings of the AAAI Conference on Artificial Intelligence. 56-61.
60. LOBJOIS, L., AND LEMACIRCITRE, M. 1998. Branch and bound algorithm selection byperformance prediction. In AAAI: Proceedings of the AAAI Conference on Artificial Intelligence. 353-358.
61. MASON, R. L., GUNST, R. F., AND HESS, J. L. 2003. Statistical Design and Analysis of Experiments. Wiley-Interscience, New York.
62. MONASSON, R., ZECCHINA, R., KIRKPATRICK, S., SELMAN, B., AND TROYANSKY, L. 1998. Determining computational complexity for characteristic 'phase transitions'. Nature 400, 133-137.
63. NISAN, N. 2000. Bidding and allocation in combinatorial auctions. In EC: Proceedings of the ACM Conference on Electronic Commerce. ACM, New York, 1-12.
64. NUDELMAN, E., LEYTON-BROWN, K., DEVKAR, A., SHOHAM, Y., AND HOOS, H. H. 2004a. SATzilla: An algorithm portfolio for SAT. In Proceedings of the International Conference on Theory and Applications of Satisfiability Testing, SAT 2004 Competition: Solver Descriptions. 13-14.
65. NUDELMAN, E., LEYTON-BROWN, K., DEVKAR, A., SHOHAM, Y., AND HOOS, H.H. 2004b. Understanding random SAT: Beyond the clauses-to-variables ratio. In CP: Proceedings of the International Conference on Principles and Practice of Constraint Programming. 438-452.
66. RICE, J. R. 1976. The algorithm selection problem. Adv. Comput. 15, 65-118.
67. ROTHKOPF, M., PEKEČ, A., AND HARSTAD, R. 1998. Computationally manageable combinatorial auctions. Manage. Sci. 44, 8, 1131-1147.
68. RUAN, Y., HORVITZ, E., AND KAUTZ, H. 2002. Restart policies with dependence among runs: A dynamic programming approach. In CP: Proceedings of the International Conference on Principles and Practice of Constraint Programming. 573-586.
69. SANDHOLM, T. 1999. Analgorithm for optimal winner determinationin combinatorial auctions. In IJCAI: Proceedings of the International Joint Conference on Artificial Intelligence. 542-547.
70. SANDHOLM, T. 2002. Algorithm for optimal winner determination in combinatorial auctions. Artif. In-tell. 135, 1, 1-54.
71. SANDHOLM, T., SURI, S., GILPIN, A., AND LEVINE, D. 2001. CABOB: A fast optimal algorithm for combinatorial auctions. In IJCAI: Proceedings of the International Joint Conference on Artificial Intelligence. 1102-1108.
72. SANDHOLM, T., SURI, S., GILPIN, A., AND LEVINE, D. 2005. CABOB: A fast optimal algorithm for winner determination in combinatorial auctions. Manage. Sci. 51, 3, 374-390.
73. SCHAPIRE, R. 1990. The strength of weak learnability. Mach. Learn. 5, 197-227.
74. SCHMEE, J., AND HAHN, G. J. 1979. A simple method for regression analysis with censored data. Technometrics 21, 4, 417-432.
75. SELMAN, B., MITCHELL, D. G., AND LEVESQUE, H. J. 1996. Generating hard satisfiability problems. Artif. Intell. 81, 1-2, 17-29.
76. SLANEY, J., AND WALSH, T. 2001. Backbones in optimization and approximation. In IJCAI: Proceedings of the International Joint Conference on Artificial Intelligence. 254-259.
77. STREETER, M., GOLOVIN, D., AND SMITH, S. F. 2007. Combining multiple heuristics online. In AAAI: Proceedings of the AAAI Conference on Artificial Intelligence. 1197-1203.
78. WILLIAMS, R., GOMES, C., AND SELMAN, B. 2003. Backdoors to typical case complexity. In IJCAI: Proceedings of the International Joint Conference on Artificial Intelligence. 1173-1178.
79. XU, L., HOOS, H. H., AND LEYTON- BROWN, K. 2007a. Hierarchical hardness models for SAT. In CP: Proceedings of the International Conference on Principles and Practice of Constraint Programming. Lecture Notes in Computer Science, vol. 4741. Springer-Verlag, Berlin, Germany, 696-711.
80. XU, L., HUTTER, F., HOOS, H. H., AND LEYTON-BROWN, K. 2007b. SATzilla-07: The design and analysis of an algorithm portfolio for SAT. In CP: Proceedings of the International Conference on Principles and Practice of Constraint Programming. Lecture Notes in Computer Science, vol. 4741. Springer-Verlag, Berlin, Germany, 712-727.
81. XU, L., HUTTER, F., HOOS, H. H., AND LEYTON-BROWN, K. 2008. SATzilla: Portfolio-based algorithm selection for SAT. J. Artif. Intell. Res. 32, 565-606.
82. ZHANG, W. 1999. State-Space Search: Algorithms, Complexity, Extensions, and Applications. Springer-Verlag, Berlin, Germany.
83. ZHANG, W. 2001. Phase transitions and backbones of 3-SAT and maximum 3-SAT. In CP: Proceedings of the International Conference on Principles and Practice of Constraint Programming.
84. ZHENG, A., JORDAN, M., LIBLIT, B., AND AIKEN, A. 2003. Statistical debugging of sampled programs. In NIPS: Proceedings of the Neural Information Processing Systems Conference.