Cross-disciplinary perspectives on meta-learning for algorithm selection

ACM Computing Surveys

Volumn 41, Issue 1, 2008, Pages

  Smith Miles, Kate A ^a  

^a MONASH UNIVERSITY   (Australia)

Author keywords

Algorithm selection; Classification; Combinatorial optimization; Constraint satisfaction; Dataset characterization; Empirical hardness; Forecasting; Landscape analysis; Meta learning; Model selection; Sorting

Indexed keywords

ALGORITHM SELECTION; CLASSIFICATION; CONSTRAINT SATISFACTION; DATASET CHARACTERIZATION; EMPIRICAL HARDNESS; LANDSCAPE ANALYSIS; META-LEARNING; MODEL SELECTION;

BIOINFORMATICS; COMBINATORIAL OPTIMIZATION; CONSTRAINED OPTIMIZATION; EDUCATION; FORECASTING; HARDNESS; LEARNING SYSTEMS; STEELMAKING; TERMINOLOGY;

LEARNING ALGORITHMS;

EID: 49749086726     PISSN: 03600300     EISSN: 15577341     Source Type: Journal    
DOI: 10.1145/1456650.1456656     Document Type: Article

References (99)

1. ACHLIOPTAS, D., NAOR, A., AND PERES, Y. 2005. Rigorous location of phase transitions in hard optimization problems. Nature 435 (Jun.), 759-764.
2. AHA, D. 1992. Generalizing from case studies: A case study. In Proceedings of the 9th International Conference on Machine Learning, 1-10.
3. ALI, S. AND SMITH, K. A. 2005. Kernel width selection for SVM classification: A meta learning approach. Int. J. Data Warehousing Mining 1, 78-97.
4. ALI, S. AND SMITH, K. 2006. On learning algorithm selection for classification. Appl. Soft Comput. 6, 2, 119-138.
5. ALI, S. AND SMITH-MILES, K. 2006. A meta-learning approach to automatic kernel selection for support vector machines. Neurocomput. 70, 1-3, 173-186.
6. ANGEL, E. AND ZISSIMOPOULOS, V. 2002. On the hardness of the quadratic assignment problem with meta-heuristics. J. Heurist. 8, 399-414.
7. ARINZE, B. 1995. Selecting appropriate forecasting models using rule induction. Omega Int. J. Manage. Sci. 22, 6, 647-658.
8. ARINZE, B., KIM, S. L., AND ANANDARAJAN, M. 1997. Combining and selecting forecasting models using rule based induction. Comput. Oper. Res. 24, 5, 423-433.
9. ARMSTRONG, W., CHRISTEN, P., MCCREATH, E., AND RENDELL, A.P. 2006. Dynamic algorithm selection using reinforcement learning. In Proceedings of the International Workshop on Integrating AI and Data Mining, 18-24.
10. ASUNCION, A AND NEWMAN, D.J. 2007. UCI Machine Learning Repository (http://www.ics.uci.edu/∼ mlearn/MLRepository.html). Irvine, CA: University of California, Department of Information and Computer Science.
11. BACHELET, V. 1999. Métaheuristiques paralleles hybrides: Application au probleme d'affectation quadratique. Ph.D. dissertation, Universite des Sciences et Technologies de Lille, France. December.
12. BENSUSAN, H., GIRAUD-CARRIER, C., AND KENNEDY, C. 2000. A higher-order approach to meta-learning. In Proceedings of the European Conference on Machine Learning, Workshop on Meta-Learning: Building Automatic Advice Strategies for Model Selection and Method Combination, 109-117.
13. BENSUSAN, H. AND KALOUSIS, A. 2001. Estimating the predictive accuracy of a classifier. Lecture Notes in Computer Science, vol. 2167, 25-31.
14. BERNSTEIN, A., PROVOST, F., AND HILL, S. 2005. Toward intelligent assistance for a data mining process: An ontology-based approach for cost-sensitive classification. IEEE Trans. Knowl. Data Eng. 17, 4, 503-518.
15. BERRER, H., PATERSON, I., AND KELLER, J. 2000. Evaluation of machine-learning algorithm ranking advisors. In Proceedings of the PKDD-00 Workshop on Data Mining, Decision Support, Meta-Learning and ILP: Forum for Practical Problem Presentation and Prospective Solutions, P. Brazdil and A. Jorge, eds.
16. BORENSTEIN, Y., AND POLI, R. 2006. Kolmogorov complexity, optimization and hardness. IEEE Congress on Evol. Comput., 112-119.
17. BOUKEAS, G., HALATSIS, C., ZISSIMPOLOULOS, V., AND STAMATOPOULOS, P. 2004. Measures of instrinsic hardness for constraint satisfaction problem instances. Lecture Notes in Computer Science, vol. 2932, 184-195.
18. BRAZDIL, P. AND HENERY, R. 1994. Analysis of results. In Machine Learning, Neural and Statistical Classification, D. Michie et al., eds. Ellis Horwood, New York. Chapter 10.
19. BRAZDIL, P., SOARES, C., AND COSTA, J. 2003. Ranking learning algorithms: Using IBL and meta-learning on accuracy and time results. Mach. Learn. 50, 3, 251-277.
20. BRODLEY, C. E. 1993. Addressing the selective superiority problem: Automatic algorithm/model class selection. In Proceedings of the 10th International Machine Learning Conference, 17-24.
21. BURKE, E., KENDALL, G., NEWALL, J., HART, E., ROSS, P., AND SCHULENBURG, S. 2003. Hyper-Heuristics: An emerging direction in modern search technology. In Handbook of Metaheuristics, Glover and Kochen-berger, eds. Kluwer Academic, Dordrecht, 457-474.
22. CASTIELLO, C., CASTELLANO, G., AND FANELLI, A. M. 2005. Meta-Data: Characterization of input features for meta-learning. Lecture Notes in Artificial Intelligence, vol. 3558, 457-468.
23. CHAN, P. AND STOLFO, S. J. 1997. On the accuracy of meta-learning for scalable data mining. J. Intell. Inf. Syst. 8, 19, 5-28.
24. CHU, C. H. AND WIDJAJA, D. 1994. A neural network system for forecasting method selection. Decis. Support Syst. 12, 13-24.
25. DESJARDINS, M AND GORDON, D. F. 1995. Special issue on bias evaluation and selection. Mach. Learn. 20, 1-2.
26. DUCH, W. AND GRUDZINSKI, K. 2001. Meta-Learning: Searching in the model space. In Proceedings of the International Conference on Neural Information Processing (ICONIP) I, 235-240.
27. ESTIVILL-CASTRO, V. AND WOOD, D. 1992. A survey of adaptive sorting algorithms. ACM Comput. Surv. 24, 4, 441-476.
28. FREY, P. W. AND SLATE, D. J. 1991. Letter recognition using holland-style adaptive classifiers. Mach. Learn. 6, 161-182.
29. FURNKRANZ, J. AND PETRAK, J. 2001. An evaluation of landmarking variants. In Proceedings of the ECML/PKDD Workshop on Integrating Aspects of Data Mining, Decision Support and Meta-Learning (IDDM), C. Giraud-Carrier et al., eds., 57-68.
30. GAGLIOLO, M. AND SCHMIDHUBER, J. 2006. Learning dynamic algorithm portfolios. Ann. Math. Artif. Intell. 47, 3-4 (Aug.), 295-328.
31. GAMA, J. AND BRAZDIL, P. 2000. Cascade generalization. Mach. Learn. 41, 3, 315-343.
32. GAMA, J. AND BRAZDIL, P. 1995. Characterization of classification algorithms. In Proceedings of the 7th Portugese Conference in AI, 83-102.
33. GLOVER, F. AND KOCHENBERGER, G. 2003. Handbook of Metaheuristics. Kluwer Academic, Dordrecht.
34. GNANADESIKAN, R. 1997. Methods for Statistical Data Analysis of Multivariate Observations, 2nd ed. Wiley, New York.
35. GUO, H. 2003. Algorithm selection for sorting and probabilistic inference: A machine learning-based approach. Ph.D. dissertation, Kansas State University.
36. HILARIO, M. 2002. Model complexity and algorithm selection in classification. In Proceedings of the 5th International Conference on Discovery Science (DS-02), S. Lange et al., eds., 113-126.
37. HOOS, H. H. AND STÜTZLE, T. 1999. Towards a characterisation of the behaviour of stochastic local search algorithms for SAT. Artif. Intell. 112, 1-2, 213-232.
38. HORVITZ, E., RUAN, Y., GOMES, C., KAUTZ, H., SELMAN, B., AND CHICKERING, M. 2001. A Bayesian approach to tackling hard computational problems. In Proc. 17th Conference on Uncertainty in Artificial Intelligence.
39. HYNDMAN, R. 2002. Time Series Data Library, available from http://www-personal.buseco.monash. edu.au/∼hyndman/TSDL/.
40. JONES, N. AND PEVZNER, P. 2004. An Introduction to Bioinformatics Algorithms. MIT Press, Cambridge, MA.
41. JONES, T. AND FORREST, S. 1995. Fitness distance correlation as a measure of problem difficulty for genetic algorithms. In Proceedings of the International Conference on Genetic Algorithms, 184-192.
42. KALOUSIS, A. AND HILARIO, M. 2001. Model selection via meta-learning. International. Journal on AI Tools. 10,4.
43. KALOUSIS, A. AND THEOHARIS, T. 1999. Design, implementation and performance results of an intelligent assistant for classifier selection. Intell. Data Anal. 3, 5, 319-337.
44. KEOGH, E. AND FOLIAS, T. 2002. The UCR Time Series Data Mining Archive. http://www.cs. ucr.edu/∼eamonn/TSDMA/.
45. KNOWLES, J. D. AND CORNE, D. W. 2002. Towards landscape analysis to inform the design of a hybrid local search for the multiobjective quadratic assignment problem. In Soft Computing Systems: Design, Management and Applications, A. Abraham et al., eds. IOS Press, Amsterdam, 271-279.
46. KOHONEN, T. 1988. Self-Organisation and Associative Memory. Springer, New York.
47. KOPF, C., TAYLOR, C., AND KELLER, J. 2000. Meta-Analysis: From data characterisation for meta-learning to meta-regression. In Proceedings of the PKDD Workshop on Data Mining, Decision Support, Meta-Learning and ILP, P. Brazdil and A Jorge, eds.
48. KUBA, P., BRÁRZDIL, P., SOARES, C., AND WOZNICA, A. 2002. Exploiting sampling and meta-learning for parameter setting for support vector machines. In Proceedings of the Workshop Learning and Data Mining Associated with Iberamia VIII Iberoamerican Conference on Artificial Intelligence. F. Herrera et al., eds., 209-216.
49. LAGOUDAKIS, M., LITTMAN, M., AND PARR, R. 2001. Selecting the right algorithm. In Proceedings of the AAAI Fall Symposium Series: Using Uncertainty within Computation.
50. LER, D., KOPRINSKA, I., AND CHAWLA, S. 2005. Utilising regression-based landmarkers within a meta-learning framework for algorithm selection. In Proceedings of the Workshop on Meta-Learning, 22nd International Conference on Machine Learning (ICML), 44-51.
51. LEYTON-BROWN, K., NUDELMAN, E., AND SHOHAM, Y. 2002. Learning the empirical hardness of optimization problems: The case of combinatorial auctions. Lecture Notes in Computer Science, vol. 2470, 556-569.
52. LEYTON-BROWN, K., NUDELMAN, E., ANDREW, G., MCFADDEN, J., AND SHOHAM, Y. 2003a. A portfolio approach to algorithm selection. In Proceedings of the International Joint Conference on Artificial Intelligence, 1542-1543.
53. LEYTON-BROWN, K., NUDELMAN, E., ANDREW, G., MCFADDEN, J., AND SHOHAM, Y 2003b. Boosting as a metaphor for algorithm design. In Proceedings of the Conference on Principles and Practice of Constraint Programming, 899-903.
54. LIM, T.-S., LOH, W.-Y., AND SHIH, Y.-S. 2000. A comparison of prediction accuracy, complexity, and training time of thirty-three old and new classification algorithms. Mach. Learn. 40, 203-228.
55. LINDER, C. AND STUDER, R. 1999. AST: Support for algorithm selection with a CBR approach. In Proceedings of the 16th International Conference on Machine Learning.
56. MAKRIDAKIS, S. AND HIBON, M. 2000. The M3-competition: Results, conclusions and implications. Int. J. Forecast. 16, 4, 451-476.
57. MAROM, Y. ZUKERMAN, I., AND JAPKOWICZ, N. 2007. A meta-learning approach for selecting between response automation strategies in a help-desk domain. In Proceedings of the 22nd AAAI Conference on Artificial Intelligence, 907-912.
58. MARON, O. AND MOORE, A. W. 1997. The racing algorithm: Model selection for lazy learners. Artif Intell. Rev., 193-225.
59. MERZ, P. 2004. Advanced fitness landscape analysis and the performance of memetic algorithms. Evol. Comput. 12, 303-325.
60. MICHIE, D., SPIEGELHALTER, D. J., AND TAYLOR C. C. EDS. 1994. Machine Learning, Neural and Statistical Classification. Ellis Horwood, New York.
61. NUDELMAN, E., LEYTON-BROWN, K., HOOS, H., DEVKAR, A., AND SHOHAM, Y. 2004. Understanding random SAT: Beyond the clauses-to-variables ratio. Lecture Notes in Computer Science, vol. 3258, 438-452.
62. OPITZ, D. AND MACLIN, R. 1999. Popular ensemble methods: An empirical study. J. Artif. Intell. Res. 11, 169-198.
63. PENG, Y., FLACH, P., SOARES, C., AND BRAZDIL P. 2002. Improved dataset characterization for meta-learning. In Proceedings of the 5th International Conference on Discovery Science.
64. PETERSON, A. H. AND MARTINEZ, T. R. 2005. Estimating the potential for combining learning models. In Proceedings of the ICML Workshop on Meta-Learning, 68-75.
65. PFAHRINGER, B., BENSUSAN, H., AND GIRAUD-CARRIER, C. 2000. Meta-Learning by landmarking various learning algorithms. In Proceedings of the 17th International Conference on Machine Learning.
66. PRODROMIDIS, A. L., CHAN, P., AND STOLFO, S. J. 2000. Meta-Learning in distributed data mining systems: Issues and approaches. In Advances of Distributed Data Mining, H. Kargupta and P. Chan, eds. AAAI Press.
67. PRUDÊNCIO, R. AND LUDERMIR, T. 2004. Meta-Learning approaches to selecting time-series models. Neuro-comput. 61, 121-137.
68. RENDELL, L. AND CHO, H. 1990. Empirical learning as a function of concept character. Mach. Learn. 5, 267-298.
69. RICE, J. R. 1953. Classes of recursively enumerable sets and their decision problems. Trans. Amer. Math. Soc. 89, 29-59.
70. RICE, J. R. 1976. The algorithm selection problem. Advances in Comput. 15, 65-118.
71. SAMULOWITZ, H. AND MEMISEVIC, R. 2007. Learning to solve QBF. In Proceedings of the 22nd AAAI Conference on Artificial Intelligence, 255-260.
72. SEEWALD, A., PETRAK, J., AND WIDMER, G. 2001. Hybrid decision tree learners with alternative leaf classifiers: An empirical study. In Proceedings of the 14th International FLAIRS Conference.
73. SELMAN, B., MITCHELL, D. G., AND LEVESQUE, H. J. 1996. Generating hard satisfiability problems. Artif. Intell. 81, 17-29.
74. SHANNON, C. E. 1948. A mathematical theory of communication. Bell Syst. Tech. J. 27 (Jul., Oct.), 379-423 and 623-656.
75. SLANEY, J. AND WALSH, T. 2001. Backbones in optimization and approximation. In Proceedings of the International Joint Conference on Artificial Intelligence.
76. SMITH, K. A., WOO, F., CIESIELSKI, V., AND IBRAHIM, R. 2001. Modelling the relationship between problem characteristics and data mining algorithm performance using neural networks. In Smart Engineering System Design: Neural Networks, Fuzzy Logic, Evolutionary Programming, Data Mining, and Complex Systems, C. Dagli et al., eds. ASME Press, 11, 356-362.
77. SMITH, K. A., WOO, F., CIESIELSKI, V., AND IBRAHIM, R. 2002. Matching data mining algorithm suitability to data characteristics using a self-organising map. In Hybrid Information Systems, A. Abraham and M. Koppen, M., eds. Physica, Heidelberg, 169-180.
78. SMITH-MILES, K. A. 2008. Towards insightful algorithm selection for optimisation using meta-learning concepts. In Proceedings of the IEEE Joint Conference on Neural Networks. 4118-4124.
79. SOARES, C., PETRAK, J., AND BRAZDIL, P. 2001. Sampling-Based relative landmarks: Systematically test-driving algorithms before choosing. In Proceedings of the Portuguese AI Conference.
80. SOARES, C., BRAZDIL, P., AND KUBA, P. 2004. A meta-learning method to select the kernel width in support vector regression. Mach. Learn. 54, 3, 195-209.
81. STREETER, M., GOLOVIN, D., AND SMITH, S. F. 2007. Combining multiple heuristics online. In Proceedings of the 22nd AAAI Conference on Artificial Intelligence, 1197-1203.
82. STÜTZLE, T. AND FERNANDES, S. 2004. New benchmark instances for the QAP and the experimental analysis of algorithms. Lecture Notes in Computer Science, vol. 3004, 199-209.
83. TELESIS, O. AND STAMATOPOULOS, P. 2001. Combinatorial optimization through statistical instance-based learning. In Proceedings of the 13th International Conference on Tools with Artificial Intelligence, 203- 209.
84. TODOROVSKI, L., BLOCKEEL, H., AND DZEROSKI, S. 2002. Ranking with predictive clustering trees. In Proceedings of the European Conference on Machine Learning.
85. TODOROVSKI, L. AND DZEROSKI, S. 2003. Combining classifiers with meta decision trees. Mach. Learn.
86. VAN HEMERT, J. I. 2006. Evolving combinatorial problem instances that are difficult to solve. Evol. Comput. 14, 433-462.
87. VENKATACHALAM, A. R. AND SOHL, J. E. 1999. An intelligent model selection and forecasting system. Int. J. Forecast. 18, 3, 67-180.
88. VILALTA, R. AND DRISSI, Y. 2002. A perspective view and survey of meta-learning. Artif. Intell. Rev. 18, 77-95.
89. VOLLMANN, T. E. AND BUFFA, E. S. 1966. The facility layout problem in perspective. Manage. Sci. 12, 10, 450-468.
90. WALLACE, C. S. AND BOULTON D. M. 1968. An information measure for classification. Comput. J. 11, 2, 185-194.
91. WALLACE, M. AND SCHIMPF, J. 2002. Finding the right hybrid algorithm - A combinatorial meta-problem. Ann. Math. Artif Intell. 34, 259-269.
92. WANG, X. 2005. Characteristic-Based forecasting for time series data, Ph.D. dissertation, Monash University, Australia.
93. WANG, X., SMITH, K. A., AND HYNDMAN, R. 2006. Characteristic-Based clustering for time series data. Data Mining Knowl. Discov. 13, 335-364.
94. WITTEN, I. H. AND FRANK, E. 2005. Data Mining: Practical Machine Learning Tools and Techniques, 2nd ed. Morgan Kaufmann, San Francisco.
95. WOLPERT, D. AND MACREADY, W 1997. No free lunch theorems for optimization. IEEE Trans. Evol. Comput. 1, 67-82.
96. WOLPERT, D. H. 1992. Stacked generalization. Neural Netw. 5, 241-259.
97. XU, L., HUTTER, F., HOOS, H., AND LEYTON-BROWN, K. 2007a. Satzilla-07: The design and analysis of an algorithm portfolio for SAT. In Principles and Practices of Constraint Programming. Lecture Notes in Computer Science, 712-727.
98. XU, L., HOOS, H., AND LEYTON- BROWN, K. 2007b. Hierarchical hardness models for SAT. In Principles and Practices of Constraint Programming. Lecture Notes in Computer Science, 696-711.
99. YANG, J. AND JIU, B. 2006. Algorithm selection: A quantitative approach. Algorithmic Trading II: Precision, control, execution. http://www.itg.com/newsevents/papers/AlgoSelection20/.