Fast cross-validation via sequential testing

Journal of Machine Learning Research

Volumn 16, Issue , 2015, Pages 1103-1155

  Krueger, Tammo ^a   Panknin, Danny ^a   Braun, Mikio ^a  

^a TECHNISCHE UNIVERSITÄT BERLIN   (Germany)

Author keywords

Cross validation; Nonparametric methods; Statistical testing

Indexed keywords

ARTIFICIAL INTELLIGENCE;

CROSS VALIDATION; EXPERIMENTAL EVALUATION; NONPARAMETRIC METHODS; NONPARAMETRIC TESTING; SEQUENTIAL ANALYSIS; SEQUENTIAL TESTING; STATISTICAL TESTING; TIME-CONSUMING TASKS;

SOFTWARE ENGINEERING;

EID: 84962236224     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (56)

1. D. W. Alling. Closed sequential tests for binomial probabilities. Biometrika, 53(1/2):73-84, 1966.
2. S. Arlot, A. Celisse, and P. Painleve. A survey of cross-validation procedures for model selection. Statistics Surveys, 4:40-79, 2010.
3. P. Armitage. Sequential Medical Trials. Blackwell Scientific Publications, 1960.
4. N. Aronszajn. Theory of reproducing kernels. Transactions of the American Mathematical Society, 68:337-404, 1950.
5. P. L. Bartlett, P. M. Long, and R. C. Williamson. Fat-shattering and the learnability of real-valued functions. Journal of Computer and Systems Science, 52:434-452, 1996.
6. Y. Bengio. Gradient-based optimization of hyperparameters. Neural Computation, 12(8):1889-1900, 2000.
7. J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13:281-305, March 2012.
8. J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl. Algorithms for hyper-parameter optimization. In Advances in Neural Information Processing Systems 24, pages 2546-2554, 2011.
9. D. Berry and B. Fristedt. Bandit problems: Sequential Allocation of Experiments. Chapman & Hall, 1985.
10. M. Birattari. Tuning Metaheuristics: A Machine Learning Perspective. Springer, 2009.
11. M. Birattari, T. Stützle, L. Paquete, and K. Varrentrapp. A racing algorithm for configuring metaheuristics. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 11-18, 2002.
12. J. A. Blackard and D. J. Dean. Comparative accuracies of artificial neural networks and discriminant analysis in predicting forest cover types from cartographic variables. Computers and Electronics in Agriculture, 24:131-151, 1999.
13. J. K. Bradley and R. Schapire. Filterboost: Regression and classification on large datasets. In Advances in Neural Information Processing Systems 20, pages 185-192, 2008.
14. M. L. Braun, J. Buhmann, and K.-R. Müller. On relevant dimensions in kernel feature spaces. Journal of Machine Learning Research, 9:1875-1908, 2008.
15. N. Cesa-Bianchi and G. Lugosi. Prediction, Learning, and Games. Cambridge University Press, 2006.
16. S. Chien, J. Gratch, and M. Burl. On the efficient allocation of resources for hypothesis evaluation: A statistical approach. IEEE Transactions Pattern Analysis and Machine Intelligence, 17:652-665, 1995.
17. S. Chien, A. Stechert, and D. Mutz. Efficient heuristic hypothesis ranking. Journal of Artificial Intelligence Research, 10:375-397, 1999.
18. W. G. Cochran. The comparison of percentages in matched samples. Biometrika, 37(3-4):256-266, 1950.
19. L. Devroye, L. Györfi, and G. Lugosi. A Probabilistic Theory of Pattern Recognition. Springer, 1996.
20. P. Domingos and G. Hulten. A general method for scaling up machine learning algorithms and its application to clustering. In Proceedings of the Eighteenth International Conference on Machine Learning, pages 106-113, 2001.
21. D. L. Donoho and J. M. Johnstone. Ideal spatial adaptation by wavelet shrinkage. Biometrika, 81(3):425-455, 1994.
22. T. Evgeniou and M. Pontil. On the V gamma dimension for regression in reproducing kernel hilbert spaces. In O. Watanabe and T. Yokomori, editors, Algorithmic Learning Theory, pages 106-117, 1999.
23. M. Friedman. The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association, 32(200):675-701, 1937.
24. S. Geisser. The predictive sample reuse method with applications. Journal of the American Statistical Association, 70(350):320-328, 1975.
25. N. A. Heckert and J. J. Filliben. NIST Handbook 148: DATAPLOT Reference Manual, Volume I: Commands. National Institute of Standards and Technology Handbook Series, 2003.
26. V. Heidrich-Meisner and C. Igel. Hoeffding and Bernstein races for selecting policies in evolutionary direct policy search. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 401-408, 2009.
27. S. S. Keerthi, V. Sindhwani, and O. Chapelle. An efficient method for gradient-based adaptation of hyperparameters in SVM models. In Advances in Neural Information Processing Systems 19, pages 673-680, 2006.
28. R. Kohavi and G. H. John. Automatic parameter selection by minimizing estimated error. In Proceedings of the Twelfth International Conference on Machine Learning, pages 304-312, 1995.
29. W. H. Kruskal and W. A. Wallis. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 47(260):583-621, 1952.
30. O. Maron and A. W. Moore. Hoeffding races: Accelerating model selection search for classification and function approximation. In Advances in Neural Information Processing Systems 6, pages 59-66, 1994.
31. O. Maron and A. W. Moore. The racing algorithm: Model selection for lazy learners. Artificial Intelligence Review, 11:193-225, 1997.
32. C. K. McPherson and P. Armitage. Repeated significance tests on accumulating data when the null hypothesis is not true. Journal of the Royal Statistical Society. Series A, 134(1):15-25, 1971.
33. V. Mnih, C. Szepesvári, and J.-Y. Audibert. Empirical Bernstein stopping. In Proceedings of the 25th International Conference on Machine learning, ICML'08, pages 672-679, 2008.
34. M. Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations Of Machine Learning. MIT Press, 2012.
35. F. Mosteller and J. W. Tukey. Handbook of Social Psychology, volume 2, chapter Data analysis, including statistics, pages 80-203. Addison-Wesley, 2nd edition, 1968.
36. K. D. Patil. Cochran's Q test: Exact distribution. Journal of the American Statistical Association, 70(349):186-189, 1975.
37. R. Pelossof and M. Jones. Curtailed online boosting. Technical report, Columbia University, 2009.
38. R. Pelossof and Z. Ying. The attentive perceptron. Computing Research Repository, abs/1009.5972, 2010.
39. R. Pelossof and Z. Ying. Rapid learning with stochastic focus of attention. Computing Research Repository, abs/1105.0382, 2011.
40. G. Rätsch, T. Onoda, and K.-R. Müller. Soft margins for AdaBoost. Machine Learning, 42(3):287-320, 2001.
41. V. Roth. Probabilistic discriminative kernel classifiers for multi-class problems. In Proceedings of the 23rd DAGM-Symposium on Pattern Recognition, pages 246-253, 2001.
42. B. Schölkopf, A. J. Smola, R. C. Williamson, and P. L. Bartlett. New support vector algorithms. Neural Computation, 12:1207-1245, 2000.
43. S. Smale and D.-X. Zhou. Estimating the approximation error in learning theory. Analysis and Applications, 1(1):1-25, 2003.
44. J. Snoek, H. Larochelle, and R. P. Adams. Practical bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems 25, pages 2960-2968, 2012.
45. C. C. Spicer. Some new closed sequential designs for clinical trials. Biometrics, 18(2):203-211, 1962.
46. A. Stanski. Konstruktives Probabilistisches Lernen. PhD thesis, Technische Universität Berlin, 2012.
47. I. Steinwart and C. Scovel. Fast rates for support vector machines using Gaussian kernels. Annals of Statistics, 35:575-607, 2007.
48. M. Stone. Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society. Series B, 36(2):111-147, 1974.
49. M. W. Tate and S. M. Brown. Note on the Cochran Q test. Journal of the American Statistical Association, 65(329):155-160, 1970.
50. C. Thornton, F. Hutter, H. H. Hoos, and K. Leyton-Brown. Auto-WEKA: Automated selection and hyper-parameter optimization of classification algorithms. CoRR, abs/1208.3719, 2012.
51. V. Vapnik. Statistical Learning Theory. Wiley, 1998.
52. A. Wald. Sequential Analysis. Wiley, 1947.
53. A. Wald and J. Wolfowitz. Optimum character of the sequential probability ratio test. The Annals of Mathematical Statistics, 19(3):326-339, 1948.
54. G. B. Wetherill and K. D. Glazebrook. Sequential Methods in Statistics. Chapman and Hall, 1986.
55. E. B. Wilson. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158):209-212, 1927.
56. A. X. Zheng and M. Bilenko. Lazy paired hyper-parameter tuning. In Proceedings of the 23rd International Joint Conference on Artificial Intelligence, pages 1924-1931, 2013.