(Psycho-)analysis of benchmark experiments: A formal framework for investigating the relationship between data sets and learning algorithms

Computational Statistics and Data Analysis

Volumn 71, Issue , 2014, Pages 986-1000

  Eugster, Manuel J A ^a   Leisch, Friedrich ^b   Strobl, Carolin ^c  

^a UNIVERSITY OF MUNICH   (Germany)

^b UNIVERSITY OF NATURAL RESOURCES AND LIFE SCIENCES   (Austria)

^c UNIVERSITY OF ZURICH   (Switzerland)

Author keywords

Benchmark experiments; Bradley Terry model; Data set characterization; Preference scaling; Recursive partitioning

Indexed keywords

BENCHMARK EXPERIMENTS; BRADLEY-TERRY MODELS; DATA SET; PREFERENCE SCALING; RECURSIVE PARTITIONING;

EXPERIMENTS; REGRESSION ANALYSIS;

LEARNING ALGORITHMS;

EID: 84889089766     PISSN: 01679473     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.csda.2013.08.007     Document Type: Article

References (36)

1. Asuncion, A., Newman, D., 2007. UCI Machine Learning Repository. URL http://www.ics.uci.edu/~mlearn/MLRepository.html.
2. Bischl, B., Wornowizki, M., Borg, K., 2012. mlr: Machine Learning in R. R package version 0.3.1206/r1206. URL http://R-Forge.R-project.org/projects/mlr/.
3. Bivand, R., 2011. classInt: choose univariate class intervals. R package version 0.1-17. URL http://CRAN.R-project.org/package=classInt.
4. R.A. Bradley, and M.E. Terry Rank analysis of incomplete block designs, I. The method of paired comparisons Biometrica 39 3/4 1952 324 345
5. L. Breiman, J.H. Friedman, R.A. Olshen, and C.J. Stone Classification and Regression Trees 1984 Chapman and Hall
6. Caruana, R., Karampatziakis, N., Yessenalina, A., 2008. An empirical evaluation of supervised learning in high dimensions. In: Proceedings of the 25th International Conference on Machine Learning. pp. 96-103.
7. D.E. Critchlow, and M.A. Fligner Paired comparison, triple comparison, and ranking experiments as generalized linear models, and their implementation on GLIM Psychometrika 56 3 1991 517 533
8. de Souto, M.C.P., Prudencio, R.B.C., Soares, R.G.F., Araujo, D.A.S., Costa, I.G., Ludermir, T.B., Schliep, A., 2008. Ranking and selecting clustering algorithms using a meta-learning approach. In: International Joint Conference on Neural Networks. pp. 3729-3735.
9. Dimitriadou, E., Hornik, K., Leisch, F., Meyer, D., Weingessel, A., 2011. e1071: misc functions of the department of statistics (e1071), TU Wien. R package version 1.6. URL http://CRAN.R-project.org/package=e1071.
10. Eugster, M.J.A., 2011. Benchmark Experiments - A Tool for Analyzing Statistical Learning Algorithms. Dr. Hut-Verlag, Ph.D. Thesis, Institut für Statistik, Ludwig-Maximilians-Universität München, Munich, Germany. URL http://edoc.ub.uni-muenchen.de/12990/.
11. Eugster, M.J.A., 2012. benchmark: benchmark experiments toolbox. R package version 0.3-4. URL http://CRAN.R-project.org/package=benchmark.
12. M.J.A. Eugster, T. Hothorn, and F. Leisch Domain-based benchmark experiments: exploratory and inferential analysis Austrian Journal of Statistics 41 1 2012 5 26 URL http://www.stat.tugraz.at/AJS/ausg121/121Leisch.pdf
13. G.H. Givens, J.R. Beveridge, P.J. Phillips, B. Draper, Y.M. Lui, and D.S. Bolme Introduction to face recognition and evaluation of algorithm performance Computational Statistics & Data Analysis 67 2013 236 247 URL http://www.sciencedirect.com/science/article/pii/S0167947313002120
14. T. Hastie, R. Tibshirani, and J. Friedman The Elements of Statistical Learning: Data Mining, Inference, and Prediction second ed. 2009 Springer
15. K. Hornik, and D. Meyer Deriving consensus rankings from benchmarking experiments R. Decker, H.-J. Lenz, Advances in Data Analysis (Proceedings of the 30th Annual Conference of the Gesellschaft für Klassifikation e.V., Freie Universität Berlin, March 8-10, 2006) Studies in Classification, Data Analysis, and Knowledge Organization 2007 Springer 163 170
16. T. Hothorn, K. Hornik, and A. Zeileis Unbiased recursive partitioning: a conditional inference framework Journal of Computational and Graphical Statistics 15 3 2006 651 674 (Pubitemid 44554201)
17. T. Hothorn, F. Leisch, A. Zeileis, and K. Hornik The design and analysis of benchmark experiments Journal of Computational and Graphical Statistics 14 3 2005 675 699 (Pubitemid 41442156)
18. A. Kalousis, J. ao Gama, and M. Hilario On data and algorithms: understanding inductive performance Machine Learning 54 2004 275 312
19. Kalousis, A., Hilario, M., 2000. Model selection via meta-learning: a comparative study. In: Proceedings of the 12th International IEEE Conference on Tools with Artificial Intelligence. pp. 406-413.
20. R.D. King, C. Feng, and A. Sutherland STATLOG: comparison of classification algorithms on large real-world problems Applied Artifcial Intelligence 9 1995 289 333
21. Kuhn, M., 2012. caret: classification and regression training. R package version 5.14-023. URL http://CRAN.R-project.org/package=caret.
22. A. Liaw, and M. Wiener Classification and regression by randomForest R News 2 3 2002 18 22 URL http://CRAN.R-project.org/doc/Rnews/
23. T.-S. Lim, W.-Y. Loh, and Y.-S. Shih A comparison of prediction accuracy, complexity, and training time of thirty-three old and new classification algorithms Machine Learning 40 3 2000 203 228
24. J.R. Quinlan C4.5: Programms for Machine Learning 1993 Morgan Kaufmann Publishers Inc. San Francisco
25. R Development Core Team, R: A Language and Environment for Statistical Computing 2011 R Foundation for Statistical Computing Vienna, Austria 3900051070 URL http://www.R-project.org
26. C. Strobl, A.-L. Boulesteix, A. Zeileis, and T. Hothorn Bias in random forest variable importance measures: illustrations, sources and a solution BMC Bioinformatics 8 25 2007
27. C. Strobl, J. Malley, and G. Tutz An introduction to recursive partitioning: rationale, application and characteristics of classification and regression trees, bagging and random forests Psychological Methods 14 4 2009 323 348
28. C. Strobl, F. Wickelmaier, and A. Zeileis Accounting for individual differences in Bradley-Terry models by means of recursive partitioning Journal of Educational and Behavioral Statistics 36 2 2011
29. Therneau, T.M., Ripley, B.D., 2011. rpart: recursive partitioning. R package version 3.1-50. URL http://CRAN.R-project.org/package=rpart.
30. S. Theußl, and A. Zeileis Collaborative software development using R-Forge The R Journal 1 1 2009 9 14 URL http://journal.r-project.org/
31. F. Torti, D. Perrotta, A.C. Atkinson, and M. Riani Benchmark testing of algorithms for very robust regression: FS, LMS and LTS Computational Statistics & Data Analysis 56 8 2012 2501 2512 URL http://dx.doi.org/10.1016/j.csda. 2012.02.003
32. W.N. Venables, and B.D. Ripley Modern Applied Statistics with S fourth ed. 2002 Springer URL http://www.stats.ox.ac.uk/pub/MASS4
33. R. Vilalta, and Y. Drissi A perspective view and survey of meta-learning Artificial Intelligence Review 18 2 2002 77 95
34. H. Wickham ggplot2: Elegant Graphics for Data Analysis 2009 Springer URL http://had.co.nz/ggplot2/book
35. A. Zeileis, and K. Hornik Generalized M-fluctuation tests for parameter instability Statistica Neerlandica 61 4 2007 488 508
36. A. Zeileis, T. Hothorn, and K. Hornik Model-based recursive partitioning Journal of Computational and Graphical Statistics 17 2 2008 492 514