Should we really use post-hoc tests based on mean-ranks?

Journal of Machine Learning Research

Volumn 17, Issue , 2016, Pages

  Benavoli, Alessio ^a   Corani, Giorgio ^a   Mangili, Francesca ^a  

^a DALLE MOLLE INSTITUTE FOR ARTIFICIAL INTELLIGENCE IDSIA   (Switzerland)

Author keywords

Friedman test; Post hoc test; Statistical comparison

Indexed keywords

ARTIFICIAL INTELLIGENCE; CODES (SYMBOLS); LAKES; LEARNING SYSTEMS;

FRIEDMAN TEST; MULTIPLE ALGORITHMS; MULTIPLE COMPARISON; MULTIPLE DATA SETS; NULL HYPOTHESIS; STATISTICAL COMPARISONS; TECHNICAL NOTES; WILCOXON SIGNED RANK TEST;

ALGORITHMS;

EID: 84962407167     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (18)

1. A. Benavoli, F. Mangili, G. Corani, M. Zaffalon, and F. Ruggeri. A Bayesian Wilcoxon signed-rank test based on the Dirichlet process. In Proceedings of the 30th International Conference on Machine Learning (ICML 2014), pages 1-9, 2014.
2. A. Benavoli, F. Mangili, F. Ruggeri, and M. Zaffalon. Imprecise Dirichlet Process with application to the hypothesis test on the probability that X ≤ Y. Journal of Statistical Theory and Practice, February 2014. doi: 10.1080/15598608.2014.985997. Accepted for publication.
3. G. Corani and A. Benavoli. A Bayesian approach for comparing cross-validated algorithms on multiple data sets. Machine Learning, 2015. Accepted for publication.
4. Janez Demšar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7:1-30, 2006.
5. Janez Demšar. On the appropriateness of statistical tests in machine learning. In Workshop on Evaluation Methods for Machine Learning in conjunction with ICML, 2008.
6. Michael A. Fligner. A note on two-sided distribution-free treatment versus control multiple comparisons. Journal of the American Statistical Association, 79(385):pp. 208-211, 1984.
7. K Ruben Gabriel. Simultaneous test procedures-some theory of multiple comparisons. The Annals of Mathematical Statistics, pages 224-250, 1969.
8. Salvador Garcia and Francisco Herrera. An extension on" Statistical Comparisons of Classifiers over Multiple Data Sets" for all pairwise comparisons. Journal of Machine Learning Research, 9(12), 2008.
9. Jean Dickinson Gibbons and Subhabrata Chakraborti. Nonparametric Statistical Inference. Springer, 2011.
10. Steven N Goodman. Toward evidence-based medical statistics: The p-value fallacy. Annals of Internal Medicine, 130(12):995-1004, 1999.
11. Myles Hollander, Douglas A Wolfe, and Eric Chicken. Nonparametric Statistical Methods, volume 751. John Wiley & Sons, 2013.
12. John K Kruschke. Bayesian data analysis. Wiley Interdisciplinary Reviews: Cognitive Science, 1(5):658-676, 2010.
13. Paul H Kvam and Brani Vidakovic. Nonparametric Statistics With Applications to Science and Engineering, volume 653. John Wiley & Sons, 2007.
14. B. J. McDonald and Jr Thompson, W. A. Rank sum multiple comparisons in one-and two-way classifications. Biometrika, 54(3/4):pp. 487-497, 1967.
15. Rupert G Miller. Simultaneous Statistical Inference. Springer, 1966.
16. P. Nemenyi. Distribution-free multiple comparisons. Ph.D. thesis, Princeton University, 1963.
17. David J Sheskin. Handbook of Parametric and Nonparametric Statistical Procedures. CRC Press, 2003.
18. Ian H Witten and Eibe Frank. Data Mining: Practical machine learning tools and techniques. Morgan Kaufmann, 2005.