A Bayesian approach for comparing cross-validated algorithms on multiple data sets

Machine Learning

Volumn 100, Issue 2-3, 2015, Pages 285-304

  Corani, Giorgio ^a   Benavoli, Alessio ^a  

^a DALLE MOLLE INSTITUTE FOR ARTIFICIAL INTELLIGENCE IDSIA   (Switzerland)

Author keywords

Bayesian hypothesis tests; Cross validation; Evaluation of classifiers; Hypothesis test; Poisson binomial; Signed rank test

Indexed keywords

ALGORITHMS; BAYESIAN NETWORKS; INFERENCE ENGINES; TESTING;

BAYESIAN HYPOTHESIS TESTS; CROSS VALIDATION; HYPOTHESIS TESTS; POISSON-BINOMIAL; SIGNED RANK TEST;

STATISTICAL TESTS;

EID: 84939272573     PISSN: 08856125     EISSN: 15730565     Source Type: Journal    
DOI: 10.1007/s10994-015-5486-z     Document Type: Article

References (13)

1. Benavoli, A., Mangili, F., Corani, G., Zaffalon, M., & Ruggeri, F. (2014). A Bayesian Wilcoxon signed-rank test based on the Dirichlet process. In Proceedings of the 31st International Conference on Machine Learning (ICML 2014) (pp. 1026–1034).
2. Bernardo, J. M., & Smith, A. F. M. (2009). Bayesian theory (Vol. 405). Chichester: Wiley.
3. Bouckaert, R. R. (2003). Choosing between two learning algorithms based on calibrated tests. In Proceedings of the 20th International Conference on Machine Learning (ICML-03) (pp. 51–58).
4. Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. The Journal of Machine Learning Research, 7, 1–30.
5. Dietterich, T. G. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10(7), 1895–1923.
6. Hong, Y. (2013). On computing the distribution function for the Poisson binomial distribution. Computational Statistics and Data Analysis, 59, 41–51.
7. Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. In Proceedings of the 14th International Joint Conference on Artificial intelligence-Volume 2 (pp. 1137–1143). Morgan Kaufmann Publishers Inc.
8. Lacoste, A., Laviolette, F., & Marchand, M. (2012). Bayesian comparison of machine learning algorithms on single and multiple datasets. In Proceeding of the Fifteenth International Conference on Artificial Intelligence and Statistics (AISTATS-12) (pp. 665–675).
9. Murphy, K. P. (2012). Machine learning: A probabilistic perspective. Cambridge: MIT press.
10. Nadeau, C., & Bengio, Y. (2003). Inference for the generalization error. Machine Learning, 52(3), 239–281.
11. Otero, J., Sánchez, L., Couso, I., & Palacios, A. (2014). Bootstrap analysis of multiple repetitions of experiments using an interval-valued multiple comparison procedure. Journal of Computer and System Sciences, 80(1), 88–100.
12. Press, S. J. (2012). Applied multivariate analysis: Using Bayesian and frequentist methods of inference. Mineola: Courier Dover Publications.
13. Witten, I. H., Frank, E., & Hall, M. A. (2011). Data mining: Practical machine learning tools and techniques (3rd ed.). Burlington, USA: Morgan Kaufmann.