A Statistical Framework for Hypothesis Testing in Real Data Comparison Studies

American Statistician

Volumn 69, Issue 3, 2015, Pages 201-212

  Boulesteix, Anne Laure ^a   Hable, Robert ^b   Lauer, Sabine ^a   Eugster, Manuel J A ^c  

^a UNIVERSITY OF MUNICH   (Germany)

^b UNIVERSITY OF BAYREUTH   (Germany)

^c AALTO UNIVERSITY   (Finland)

Author keywords

Benchmarking; Comparison; Performance; Supervised learning; Testing

Indexed keywords

EID: 84940653205     PISSN: 00031305     EISSN: 15372731     Source Type: Journal    
DOI: 10.1080/00031305.2015.1005128     Document Type: Article

References (38)

1. U.Alon,, N.Barkai,, D.A.Notterman,, K.Gish,, S.Ybarra,, D.Mack,, and A.J.Levine, (1999), “Broad Patterns of Gene Expression Revealed by Clustering Analysis of Tumor and Normal Colon Tissues Probed by Oligonucleotide Arrays,” Proceedings of the National Academy of Sciences, 96, 6745–6750.
2. P.Billingsley, (1986), Probability and Measure (2nd ed.), New York: Wiley.
3. J.Bock, (1998), Bestimmung des Stichprobenumfangs, München/Wien: Oldenbourg.
4. A.-L.Boulesteix, (2013), “On Representative and Illustrative Comparisons With Real Data in Bioinformatics: Response to the Letter to the Editor by Smith et al. Bioinformatics, 29, 2664–2666.
5. A.L.Boulesteix,, S.Lauer,, and M.J.E.Eugster, (2013), “A Plea for Neutral Comparison Studies in Computational Sciences,” PLoS One, 8, e61562.
6. A.L.Boulesteix,, C.Strobl,, T.Augustin,, and M.Daumer, (2008), “Evaluating Microarray-Based Classifiers: An Overview,” Cancer Informatics, 6, 77–97.
7. U.Braga-Neto,, and E.Dougherty, (2005), “Exact Performance of Error Estimators for Discrete Classifiers,” Pattern Recognition, 38, 1799–1814.
8. L.Dalton,, and E.Dougherty, (2012a), “Exact Sample Conditioned MSE Performance of the Bayesian MMSE Estimator for Classification Error—Part I: Representation,” IEEE Transactions on Signal Processing, 60, 2575–2587.
9. L.Dalton, (2012b), “Exact Sample Conditioned MSE Performance of the Bayesian MMSE Estimator for Classification Error—Part II: Consistency and Performance Analysis,” IEEE Transactions on Signal Processing, 60, 2588–2603.
10. J.Demšar, (2006), “Statistical Comparisons of Classifiers Over Multiple Data Sets,” Journal of Machine Learning Research, 7, 1–30.
11. B.de Souza,, A.de Carvalho,, and C.Soares, (2010), “A Comprehensive Comparison of ML Algorithms for Gene Expression Data Classification,” in The 2010 International Joint Conference on Neural Networks (IJCNN). IEEE, pp. 1–8.
12. R.Díaz-Uriarte,, and S.A.De Andres, (2006), “Gene Selection and Classification of Microarray Data Using Random Forest,” BMC Bioinformatics, 7, 3.
13. T.Dietterich, (1998), “Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms,” Neural Computation, 10, 1895–1923.
14. E.Dougherty,, A.Zollanvari,, and U.Braga-Neto, (2011), “The Illusion of Distribution-Free Small-Sample Classification in Genomics,” Current Genomics, 12, 333–341.
15. S.Dudoit,, J.Fridlyand,, and T.P.Speed, (2002), “Comparison of Discrimination Methods for the Classification of Tumors Using Gene Expression Data,” Journal of the American Statistical Association, 97, 77–87.
16. B.Efron,, and R.Tibshirani, (1997), “Improvements on Cross-Validation: the 632+ Bootstrap Method,” Journal of the American Statistical Association, 92, 548–560.
17. M.Eugster,, T.Hothorn,, and F.Leisch, (2012), “Domain-Based Benchmark Experiments: Exploratory and Inferential Analysis,” Austrian Journal of Statistics, 41, 5–26.
18. M.J.Eugster,, F.Leisch,, and C.Strobl, (2014), “(Psycho-) Analysis of Benchmark Experiments: A Formal Framework for Investigating the Relationship Between Data Sets and Learning Algorithms,” Computational Statistics & Data Analysis, 71, 986–1000.
19. S.Garcia,, A.Fernandez,, J.Luengo,, and F.Herrera, (2010), “Advanced Nonparametric Tests for Multiple Comparisons in the Design of Experiments in Computational Intelligence and Data Mining: Experimental Analysis of Power,” Information Sciences, 180, 2044–2064.
20. S.Garcia,, and F.Herrera, (2008), “An Extension on Statistical Comparisons of Classifiers Over Multiple Data Sets for All Pairwise Comparisons,” Journal of Machine Learning Research, 9, 2677–2694.
21. T.Golub,, D.Slonim,, P.Tamayo,, C.Huard,, M.Gaasenbeek,, J.Mesirov,, H.Coller,, M.Loh,, J.Downing,, and M.Caligiuri,, Bloomfield, C. D., Lander, E. S. (1999), “Molecular Classification of Cancer: Class Discovery and Class Prediction by Gene Expression Monitoring,” Science, 286, 531–537.
22. B.Hanczar,, and E.Dougherty, (2010), “On the Comparison of Classifiers for Microarray Data,” Current Bioinformatics, 5, 29–39.
23. D.J.Hand, (2006), “Classifier Technology and the Illusion of Progress,” Statistical Science, 21, 1–14.
24. T.Hastie,, R.Tibshirani,, J.Friedman,, and J.Franklin, (2009), The Elements of Statistical Learning: Data Mining, Inference and Prediction, New York: Springer.
25. W.Hoeffding,, and J.Wolfowitz, (1958), “Distinguishability of Sets of Distributions. (The Case of Independent and Identically Distributed Chance Variables),” Annals of Mathematical Statistics, 29, 700–718.
26. T.Hothorn,, F.Leisch,, A.Zeileis,, and K.Hornik, (2005), “The Design and Analysis of Benchmark Experiments,” Journal of Computational and Graphical Statistics, 14, 675–699.
27. M.Jelizarow,, V.Guillemot,, A.Tenenhaus,, K.Strimmer,, and A.L.Boulesteix, (2010), “Over-Optimism in Bioinformatics: An Illustration,” Bioinformatics, 26, 1990–1998.
28. C.Lai,, M.Reinders,, L.Van’t Veer,, and L.Wessels, (2006), “A Comparison of Univariate and Multivariate Gene Selection Techniques for Classification of Cancer Datasets,” BMC Bioinformatics, 7, 235.
29. J.Lee,, J.Lee,, M.Park,, and S.Song, (2005), “An Extensive Comparison of Recent Classification Tools Applied to Microarray Data,” Computational Statistics & Data Analysis, 48, 869–885.
30. C.Nadeau,, and Y.Bengio, (2003), “Inference for the Generalization Error,” Machine Learning, 52, 239–281.
31. B.D.Ripley, (2008), Pattern Recognition and Neural Networks, Cambridge, UK: Cambridge University Press.
32. M.Slawski,, M.Daumer,, and A.L.Boulesteix, (2008), “CMA—A Comprehensive Bioconductor Package for Supervised Classification With High Dimensional Data,” BMC Bioinformatics, 9, 439.
33. A.Statnikov,, C.Aliferis,, I.Tsamardinos,, D.Hardin,, and S.Levy, (2005), “A Comprehensive Evaluation of Multicategory Classification Methods for Microarray Gene Expression Cancer Diagnosis,” Bioinformatics, 21, 631–643.
34. G.Webb, (2000), “Multiboosting: A Technique for Combining Boosting and Wagging,” Machine Learning, 40, 159–196.
35. D.Wolpert, (2001), “The Supervised Learning No-Free-Lunch Theorems,” in Proceedings of the 6th Online World Conference on Soft Computing in Industrial Applications, pp. 10–24.
36. M.Yousefi,, J.Hua,, C.Sima,, and E.Dougherty, (2010), “Reporting Bias When Using Real Data Sets to Analyze Classification Performance,” Bioinformatics, 26, 68–76.
37. A.Zollanvari,, U.Braga-Neto,, and E.Dougherty, (2009), “On the Sampling Distribution of Resubstitution and Leave-One-Out Error Estimators for Linear Classifiers,” Pattern Recognition, 42, 2705–2723.
38. A.Zollanvari,, U.Braga-Neto,, and E.Dougherty, (2011), “Analytic Study of Performance of Error Estimators for Linear Discriminant Analysis,” IEEE Transactions on Signal Processing, 59, 4238–4255.