The design and analysis of benchmark experiments

Journal of Computational and Graphical Statistics

Volumn 14, Issue 3, 2005, Pages 675-699

  Hothorn, Torsten ^a   Leisch, Friedrich ^b   Zeileis, Achim ^c   Hornik, Kurt ^c  

^a UNIVERSITY OF ERLANGEN NUREMBERG   (Germany)

^b VIENNA UNIVERSITY OF TECHNOLOGY   (Austria)

^c VIENNA UNIVERSITY OF ECONOMICS AND BUSINESS   (Austria)

Author keywords

Bootstrap; Cross validation; Hypothesis testing; Model comparison; Performance

Indexed keywords

EID: 26644464819     PISSN: 10618600     EISSN: None     Source Type: Journal    
DOI: 10.1198/106186005X59630     Document Type: Article

References (55)

1. Alpaydin, E. (1999), "Combined 5 × 2 cv F Test for Comparing Supervised Classification Learning Algorithms," Neural Computation, 11, 1885-1892.
2. Bartlett, P. L., Boucheron, S., and Lugosi, G. (2002), "Model Selection and Error Estimation," Machine Learning, 48, 85-113.
3. Bauer, E., and Kohavi, R. (1999), "An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants," Machine Learning, 36, 105-139.
4. Berger, V. W. (2000), "Pros and Cons of Permutation Tests in Clinical Trials," Statistics in Medicine, 19, 1319-1328.
5. Berger, V. W., Lunneborg, C., Ernst, M. D., and Levine, J. G. (2002), "Parametric Analyses in Randomized Clinical Trials," Journal of Modern Applied Statistical Methods, 1, 74-82.
6. Blake, C. L., and Merz, C. J. (1998), "UCI Repository of Machine Learning Databases," http://www.ics.uci.edu/~mlearn/MLRepository.html.
7. Blockeel, H., and Struyf, J. (2002), "Efficient Algorithms for Decision Tree Cross-Validation," Journal of Machine Learning Research, 3, 621-650.
8. Breiman, L. (1996a), "Bagging Predictors," Machine Learning, 24, 123-140.
9. _ (1996b), "Out-of-Bag Estimation," Technical report, Statistics Department, University of California Berkeley, Berkeley CA 94708, ftp://ftp.stat.berkeley.edu/pub/users/breiman/.
10. _ (2001 a), "Random Forests," Machine Learning, 45, 5-32.
11. _ (2001b), "Statistical Modeling: The Two Cultures" (with discussion), Statistical Science, 16, 199-231.
12. Breiman, L., and Friedman, J. H. (1985), "Estimating Optimal Transformations for Multiple Regression and Correlation," Journal of the American Statistical Association, 80, 580-598.
13. Bylander, T. (2002), "Estimating Generalization Error on Two-Class Datasets Using Out-of-Bag Estimates," Machine Learning, 48, 287-297.
14. Bühlmann, P. (2002), "Bootstraps for Time Series," Statistical Science, 17, 52-72.
15. Chang, C.-C., and Lin, C.-J. (2001), LIBSVM: A Library for Support Vector Machines, Department of Computer Science and Information Engineering, National Taiwan University, http://www.csie.ntu.edu.tw/~cjlin/libsvm.
16. Dietterich, T. G. (1998), "Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms," Neural Computation, 10, 1895-1923.
17. _ (2000), "An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees: Bagging, Boosting, and Randomization," Machine Learning, 40, 139-157.
18. Dimitriadou, E., Hornik, K., Leisch, F., Meyer, D., and Weingessel, A. (2004), e1071: Misc Functions of the Department of Statistics (e1071), TU Wien, R package version 1.5-1, http://CRAN.R-project.org.
19. Dudoit, S., and van der Laan, M. J. (2005), "Asymptotics of Cross-Validated Risk Estimation in Estimator Selection and Performance Assessment," Statistical Methodology, 2, 131-154.
20. Efron, B. (1983), "Estimating the Error Rate of a Prediction Rule: Improvements on Cross-Validation," Journal of the American Statistical Association, 78, 316-331.
21. _ (1986), "How Biased is the Apparent Error Rate of a Prediction Rule?" Journal of the American Statistical Association, 81, 461-470.
22. Efron, B., and Tibshirani, R. J. (1993), An Introduction to the Bootstrap, New York: Chapman & Hall.
23. _ (1997), "Improvements on Cross-Validation: The .632+ Bootstrap Method," Journal of the American Statistical Association, 92, 548-560.
24. Freund, Y., and Schapire, R. E. (1996), "Experiments with a New Boosting Algorithm," in Machine Learning: Proceedings of the Thirteenth International Conference, ed. L. Saitta, San Francisco: Morgan Kaufmann, pp. 148-156.
25. Friedman, J. H. (1991), "Multivariate Adaptive Regression Splines," The Annals of Statistics, 19, 1-67.
26. George, E. I. (2000), "The Variable Selection Problem," Journal of the American Statistical Association, 95, 1304-1308.
27. Gu, C., and Xiang, D. (2001), "Cross-Validating Non-Gaussian Data: Generalized Approximate Cross-Validation Revisited," Journal of Computational and Graphical Statistics, 10, 581-591.
28. Hájek, J., Šidák, Z., and Sen, P. K. (1999), Theory of Rank Tests (2nd ed.), London: Academic Press.
29. Hastie, T., Tibshirani, R., and Friedman, J. (2001), The Elements of Statistical Learning (Data Mining, Inference and Prediction), New York: Springer Verlag.
30. Hochberg, Y., and Tamhane, A. C. (1987), Multiple Comparison Procedures, New York: Wiley.
31. Hothorn, T. (2003), "Bundling Classifiers with an Application to Glaucoma Diagnosis," Ph.D. thesis, Department of Statistics. University of Dortmund, Germany, http://eldorado.uni-dortmund.de:8080/FB5/ls7/forschung/2003/ Hothorn.
32. Hothorn, T., and Lausen, B. (2003), "Double-Bagging: Combining Classifiers by Bootstrap Aggregation," Pattern Recognition, 36, 1303-1309.
33. _ (2005), "Bundling Classifiers by Bagging Trees," Computational Statistics & Data Analysis, 49, 1068-1075.
34. Ihaka, R., and Gentleman, R. (1996), "R: A Language for Data Analysis and Graphics," Journal of Computational and Graphical Statistics, 5, 299-314.
35. Kim, H., and Loh, W.-Y. (2003), "Classification Trees with Bivariate Linear Discriminant Node Models," Journal of Computational and Graphical Statistics, 12, 512-530.
36. Läuter, J. (1992), Stabile Multivariate Verfahren: Diskriminanzanalyse-Regressionsanalyse-Faktoranalyse, Berlin: Akademie Verlag.
37. Liaw, A., and Wiener, M. (2002), "Classification and Regression by randomForest, R News, 2, 18-22.
38. Lim, T.-S., Loh, W.-Y., and Shih, Y.-S. (2000), "A Comparison of Prediction Accuracy, Complexity, and Training Time of Thirty-Three Old and New Classification Algorithms." Machine Learning, 40, 203-228.
39. Meyer, D. (2001), "Support Vector Machines," R News, 1, 23-26.
40. Meyer, D., Leisch, F., and Hornik, K. (2003), "The Support Vector Machine under Test," Neurocomputing, 55, 169-186.
41. Nadeau, C., and Bengio, Y. (2003), "Inference for the Generalization Error," Machine Learning, 52, 239-281.
42. Patterson, J. G. (1992), Benchmarking Basics, Menlo Park, CA: Crisp Publications Inc.
43. Pesarin, F. (2001), Multivariate Permutation Tests: With Applications to Biostatistics, Chichester: Wiley.
44. Peters, A., Hothorn, T., and Lausen, B. (2002), "ipred: Improved Predictors, R News, 2, 33-36.
45. Pittman, J. (2002), "Adaptive Splines and Genetic Algorithms," Journal of Computational and Graphical Statistics, 11, 615-638.
46. Pizarro, J., Guerrero, E., and Galindo, P. L. (2002), "Multiple Comparison Procedures Applied to Model Selection," Neurocomputing, 48, 155-173.
47. R Development Core Team (2004), R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna, Austria, ISBN 3-900051-00-3.
48. Ripley, B. D. (1996), Pattern Recognition and Neural Networks, Cambridge. UK: Cambridge University Press.
49. Schiavo, R. A., and Hand, D. J. (2000), "Ten More Years of Error Rate Research," International Statistical Review, 68, 295-310.
50. Stone, M. (1974), "Cross-Validatory Choice and Assessment of Statistical Predictions," Journal of the Royal Statistical Society, Series B, 36, 111-147.
51. Therneau, T. M., and Atkinson, E. J. (1997), "An Introduction to Recursive Partitioning using the rpart Routine, Technical Report 61, Section of Biostatistics, Mayo Clinic, Rochester, http://www.mayo.edu/hsr/techrpt/61. pdf.
52. Vapnik, V. (1998), Statistical Learning Theory, New York: Wiley.
53. Vehtari, A., and Lampinen, J. (2002), "Bayesian Model Assessment and Comparison Using Cross-Validation Predictive Densities," Neural Computation, 14, 2439-2468.
54. Venables, W. N., and Ripley, B. D. (2002), Modern Applied Statistics with S (4th ed.), New York: Springer, http://www.stats.ox.ac.uk/pub/MASS4/.
55. Wolpert, D. H., and Macready, W. G. (1999), "An Efficient Method to Estimate Bagging's Generalization Error," Machine Learning, 35, 41-51.