Model selection: Beyond the bayesian/frequentist divide

Journal of Machine Learning Research

Volumn 11, Issue , 2010, Pages 61-87

  Guyon, Isabelle ^a   Saffari, Amir ^b   Dror, Gideon ^c   Cawley, Gavin ^d  

^a ClopiNet   (United States)

^b GRAZ UNIVERSITY OF TECHNOLOGY   (Austria)

^c ACADEMIC COLLEGE OF TEL AVIV YAFFO   (Israel)

^d UNIVERSITY OF EAST ANGLIA   (United Kingdom)

Author keywords

Bayesian priors; Bias variance tradeoff; Ensemble methods; Guaranteed risk minimization; Model selection; Multilevel inference; Multilevel optimization; Over fitting; Performance prediction; Regularization; Structural risk minimization

Indexed keywords

BAYESIAN; ENSEMBLE METHODS; MODEL SELECTION; MULTILEVEL OPTIMIZATION; OVERFITTING; PERFORMANCE PREDICTION; RISK MINIMIZATION; STRUCTURAL RISK MINIMIZATION;

BAYESIAN NETWORKS; COMMUNICATION CHANNELS (INFORMATION THEORY);

OPTIMIZATION;

EID: 76749118521     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (84)

1. M. Adankon and M. Cheriet. Unified framework for SVM model selection. In I. Guyon, et al., editor, Hands on Pattern Recognition. Microtome, 2009.
2. H. Akaike. Information theory and an extension of the maximum likelihood principle. In B.N. Petrov and F. Csaki, editors, 2nd International Symposium on Information Theory, pages 267- 281. Akademia Kiado, Budapest, 1973.
3. C. F. Aliferis, I. Tsamardinos, and A. Statnikov. HITON, a novel Markov blanket algorithm for optimal variable selection. In 2003 American Medical Informatics Association (AMIA) Annual Symposium, pages 21-25, 2003.
4. C.-A. Azencott and P. Baldi. Virtual high-throughput screening with two-dimensional kernels. In I. Guyon, et al., editor, Hands on Pattern Recognition. Microtome, 2009.
5. P. L. Bartlett. For valid generalization the size of the weights is more important than the size of the network. In M. C. Mozer, M. I. Jordan, and T. Petsche, editors, Advances in Neural Information Processing Systems, volume 9, page 134, Cambridge, MA, 1997. MIT Press.
6. A. Ben-Hur, A. Elisseeff, and I. Guyon. A stability based method for discovering structure in clustered data. In Pacific Symposium on Biocomputing, pages 6-17, 2002.
7. A. Blum and P. Langley. Selection of relevant features and examples in machine learning. Artificial Intelligence, 97(1-2):245-271, December 1997.
8. B. Boser, I. Guyon, and V. Vapnik. A training algorithm for optimal margin classifiers. In COLT, pages 144-152, 1992.
9. M. Boullé. Compression-based averaging of selective naive bayes classifiers. In I. Guyon and A. Saffari, editors, JMLR, Special Topic on Model Selection, volume 8, pages 1659-1685, Jul 2007. URL http://www.jmlr.org/papers/ volume8/boulle07a/boulle07a.pdf.
10. M. Boullé. Data grid models for preparation and modeling in supervised learning. In I. Guyon, et al., editor, Hands on Pattern Recognition. Microtome, 2009.
11. L. Breiman. Random forests. Machine Learning, 45(1):5-32, 2001.
12. L. Breiman. Bagging predictors. Machine Learning, 24(2):123-140, 1996.
13. G. Cawley. Leave-one-out cross-validation based model selection criteria for weighted ls-svms. In IJCNN, pages 1661-1668, 2006.
14. G. Cawley and N. Talbot. Over-fitting in model selection and subsequent selection bias in performance evaluation. JMLR, submitted, 2009.
15. G. Cawley and N. Talbot. Preventing over-fitting during model selection via Bayesian regularization of the hyper-parameters. In I. Guyon and A. Saffari, editors, JMLR, Special Topic on Model Selection, volume 8, pages 841-861, Apr 2007a. URL http://www.jmlr.org/papers/volume8/cawley07a/cawley07a.pdf.
16. G. C. Cawley and N. L. C. Talbot. Agnostic learning versus prior knowledge in the design of kernel machines. In Proc. IJCNN07, Orlando, Florida, Aug 2007b. INNS/IEEE.
17. G.C. Cawley, G.J. Janacek, and N.L.C. Talbot. Generalised kernel machines. In International Joint Conference on Neural Networks, pages 1720 - 1725. IEEE, August 2007.
18. W. Chu, S. Keerthi, C. J. Ong, and Z. Ghahramani. Bayesian Support Vector Machines for feature ranking and selection. In I. Guyon, et al., editor, Feature Extraction, Foundations and Applications, 2006.
19. G. Claeskens, C. Croux, and J. Van Kerckhoven. An information criterion for variable selection in Support Vector Machines. In I. Guyon and A. Saffari, editors, JMLR, Special Topic on Model Selection, volume 9, pages 541-558, Mar 2008. URL http://www.jmlr.org/papers/volume9/claeskens08a/claeskens08a.pdf.
20. Clopinet. Challenges in machine learning, 2004-2009. URL: http://clopinet.com/challenges.
21. C. Dahinden. An improved Random Forests approach with application to the performance prediction challenge datasets. In I. Guyon, et al., editor, Hands on Pattern Recognition. Microtome, 2009.
22. M. Debruyne, M. Hubert, and J. Suykens. Model selection in kernel based regression using the influence function. In I. Guyon and A. Saffari, editors, JMLR, Special Topic on Model Selection, volume 9, pages 2377-2400, Oct 2-8. URL http://www.jmlr.org/papers/volume9/debruyne08a/debruyne08a.pdf.
23. Y. Freund and R. E. Schapire. Experiments with a new boosting algorithm. In Proc. 13th International Conference on Machine Learning, pages 148-1146 Morgan Kaufmann, 1996.
24. J. Friedman. Greedy function approximation: A gradient boosting machine. Annals of Statistics, 29:1189-1232, 2000.
25. J. Friedman, T. Hastie, and R. Tibshirani. Additive logistic regression, a statistical view of boosting. Annals of Statistics, 28:337374, 2000.
26. J. Friedman, T. Hastie, and R. Tibshirani. Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software (to appear), 2009.
27. P. Germain, A. Lacasse, F. Laviolette, and M. Marchand. PAC-Bayesian learning of linear classifiers. In ICML '09: Proceedings of the 26th Annual International Conference on Machine Learning, pages 353-360, New York, NY, USA, 2009. ACM. ISBN 978-1-60558-516-1.
28. Y. Guermeur. VC theory of large margin multi-category classifiers. In I. Guyon and A. Saffari, editors, JMLR, Special Topic on Model Selection, volume 8, pages 2551-2594, Nov 2007. URL http://www.jmlr.org/papers/volume8/guermeur07a/ guermeur07a.pdf.
29. I. Guyon. A practical guide to model selection. In J. Marie, editor, Machine Learning Summer School. Springer, to appear, 2009.
30. I. Guyon, S. Gunn, M. Nikravesh, and L. Zadeh, Editors. Feature Extraction, Foundations and Applications. Studies in Fuzziness and Soft Computing. With data, results and sample code for the NIPS 2003 feature selection challenge. Physica-Verlag, Springer, 2006a. URL http: //clopinet.com/fextract-book/.
31. I. Guyon, A. Saffari, G. Dror, and J. Buhmann. Performance prediction challenge. In IEEE/INNS conference IJCNN 2006, Vancouver, Canada, July 16-21 2006b.
32. I. Guyon, A. Saffari, G. Dror, and G. Cawley. Agnostic learning vs. prior knowledge challenge.In IEEE/INNS conference IJCNN 2007, Orlando, Florida, August 12-17 2007.
33. I. Guyon, C. Aliferis, G. Cooper, A. Elisseeff, J.-P. Pellet, P. Spirtes, and A. Statnikov. Design and analysis of the causation and prediction challenge. In JMLR W&CP, volume 3, pages 1-33, WCCI2008 workshop on causality, Hong Kong, June 3-4 2008a. URL http://jmlr.csail.mit.edu/papers/ topic/causality.html.
34. I. Guyon, A. Saffari, G. Dror, and G. Cawley. Analysis of the IJCNN 2007 agnostic learning vs. prior knowledge challenge. In Neural Networks, volume 21, pages 544-550, Orlando, Florida, March 2008b.
35. I. Guyon, D. Janzing, and B. Schölkopf. Causality: objectives and assessment. In NIPS 2008 workshop on causality, volume 7. JMLR W&CP, in press, 2009a.
36. I. Guyon, V. Lemaire, M. Boullé, Gideon Dror, and David Vogel. Analysis of the KDD cup 2009: Fast scoring on a large orange customer database. In KDD cup 2009, in press, volume 8. JMLR W&CP, 2009b.
37. L. E. Sucar H. J. Escalante, M. Montes. Particle swarm model selection. In I. Guyon and A. Saffari, editors, JMLR, Special Topic on Model Selection, volume 10, pages 405-440, Feb 2009. URL http://www.jmlr.org/papers/volume10/ escalante09a/escalante09a.pdf.
38. T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning, Data Mining, Inference and Prediction. Springer Verlag, 2000.
39. T. Hastie, S. Rosset, R. Tibshirani, and J. Zhu. The entire regularization path for the support vector machine. JMLR, 5:1391-1415, 2004. URL http://jmlr.csail.mit.edu/papers/volume5/hastie04a/hastie04a.pdf.
40. D. Haussler, M. Kearns, and R. Schapire. Bounds on the sample complexity of Bayesian learning using information theory and the vc dimension. Machine Learning, 14(1):83-113, 1994. ISSN 0885-6125.
41. A. E. Hoerl. Application of ridge analysis to regression problems. Chemical Engineering Progress, 58:54-59, 1962.
42. C. Hue and M. Boullé. A new probabilistic approach in rank regression with optimal Bayesian partitioning. In I. Guyon and A. Saffari, editors, JMLR, Special Topic on Model Selection, volume 8, pages 2727-2754, Dec 2007. URL http://www.jmlr.org/papers/volume8/hue07a/hue07a.pdf.
43. IBM team. Winning the KDD cup orange challenge with ensemble selection. In KDD cup 2009, in press, volume 8. JMLR W&CP, 2009.
44. R. Kohavi and G. John. Wrappers for feature selection. Artificial Intelligence, 97(1-2):273-324, December 1997.
45. I. Koo and R. M. Kil. Model selection for regression with continuous kernel functions using the modulus of continuity. In I. Guyon and A. Saffari, editors, JMLR, Special Topic on Model Selection, volume 9, pages 2607-2633, Nov 2008. URL http://www.jmlr.org/papers/volume9/koo08b/koo08b.pdf.
46. G. Kunapuli, J.-S. Pang, and K. Bennett. Bilevel cross-validation-based model selection.In I. Guyon, et al., editor, Hands on Pattern Recognition. Microtome, 2009.
47. J. Langford. Tutorial on practical prediction theory for classification. JMLR, 6:273-306, Mar 2005. URL http://jmlr.csail.mit.edu/papers/volume6/ langford05a/langford05a.pdf.
48. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. J. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1:541-551, 1989.
49. R. W. Lutz. Logitboost with trees applied to the WCCI 2006 performance prediction challenge datasets. In Proc. IJCNN06, pages 2966-2969, Vancouver, Canada, July 2006. INNS/IEEE.
50. D. MacKay. A practical Bayesian framework for backpropagation networks. Neural Computation, 4:448-472, 1992.
51. O. Madani, D. M. Pennock, and G. W. Flake. Co-validation: Using model disagreement to validate classification algorithms. In NIPS, 2005.
52. M. Momma and K. Bennett. A pattern search method for model selection of Support Vector Regression. In In Proceedings of the SIAM International Conference on Data Mining. SIAM, 2002.
53. R. Neal and J. Zhang. High dimensional classification with Bayesian neural networks and dirichlet diffusion trees. In I. Guyon, et al., editor, Feature Extraction, Foundations and Applications, 2006.
54. V. Nikulin. Classification with random sets, boosting and distance-based clustering. In I. Guyon, et al., editor, Hands on Pattern Recognition. Microtome, 2009.
55. T. Poggio and F. Girosi. Regularization algorithms for learning that are equivalent to multilayer networks. Science, 247(4945):978-982, February 1990.
56. A. Pozdnoukhov and S. Bengio. Invariances in kernel methods: From samples to objects. Pattern Recogn. Lett., 27(10):1087-1097, 2006. ISSN 0167-8655.
57. E. Pranckeviciene and R. Somorjai. Liknon feature selection: Behind the scenes. In I. Guyon, et al., editor, Hands on Pattern Recognition. Microtome, 2009.
58. J. Reunanen. Model selection and assessment using cross-indexing. In Proc. IJCNN07, Orlando, Florida, Aug 2007. INNS/IEEE.
59. S. Rosset and J. Zhu. Sparse, flexible and efficient modeling using L1 regularization. In I. Guyon, et al., editor, Feature Extraction, Foundations and Applications, 2006.
60. S. Rosset, J. Zhu, and T. Hastie. Boosting as a regularized path to a maximum margin classifier. Journal of Machine Learning Research, 5:941-973, 2004.
61. M. Saeed. Hybrid learning using mixture models and artificial neural networks. In I. Guyon, et al., editor, Hands on Pattern Recognition. Microtome, 2009.
62. A. Saffari and I. Guyon. Quick start guide for CLOP. Technical report, Graz University of Technology and Clopinet, May 2006. URL http://clopinet.com/ CLOP/.
63. D. Schuurmans and F. Southey. Metric-based methods for adaptive model selection and regularization. Machine Learning, Special Issue on New Methods for Model Selection and Model Combination, 48:51-84, 2001.
64. G. Schwarz. Estimating the dimension of a model. The Annals of Statistics, 6(2):461-464, 1978.
65. M. Seeger. PAC-Bayesian generalisation error bounds for Gaussian process classification. JMLR, 3:233-269, 2003. URL http://jmlr.csail.mit.edu/papers/ volume3/seeger02a/seeger02a.pdf.
66. M. Seeger. Bayesian inference and optimal design for the sparse linear model. JMLR, 9:759-813, 2008. ISSN 1533-7928.
67. P. Simard, Y. LeCun, and J. Denker. Efficient pattern recognition using a new transformation distance. In S. J. Hanson, J. D. Cowan, and C. L. Giles, editors, Advances in Neural Information Processing Systems 5, pages 50-58, San Mateo, CA, 1993. Morgan Kaufmann.
68. A. Singh, R. Nowak, and X. Zhu. Unlabeled data: Now it helps, now it doesn't. In NIPS, 2008.
69. A. Smola, S. Mika, B. Schölkopf, and R. Williamson. Regularized principal manifolds. JMLR, 1:179-209, 2001. URL http://jmlr.csail.mit.edu/ papers/volume1/smola01a/smola01a.pdf.
70. R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B, 58:267-288, 1994.
71. E. Tuv, A. Borisov, G. Runger, and K. Torkkola. Feature selection with ensembles, artificial variables, and redundancy elimination. In I. Guyon and A. Saffari, editors, JMLR, Special Topic on Model Selection, volume 10, pages 1341-1366, Jul 2009. URL http://www.jmlr.org/papers/volume10/tuv09a/tuv09a.pdf.
72. L. Valiant. A theory of the learnable. Communications of the ACM, 27(11):1134-1142, 1984.
73. V. Vapnik. Statistical Learning Theory. John Wiley and Sons N.Y., 1998.
74. V. Vapnik. Estimation of Dependences Based on Empirical Data [in Russian]. Nauka, Moscow, 1979. (English translation: Springer Verlag, New York, 1982).
75. V. Vapnik and A. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theory Probab. Appl., 16:264-1180, 1971.
76. S. Vishwanathan and A. Smola. Fast kernels for string and tree matching. In Advances in Neural Information Processing Systems 15, pages 569-576. MIT Press, 2003. URL http://books.nips.cc/papers/files/nips15/AA11.pdf.
77. G. Wahba. Spline Models for Observational Data, volume 59 of CBMS-NSF Regional Conference Series in Applied Mathematics. SIAM, Philadelphia, 1990.
78. A. Waibel. Consonant recognition by modular construction of large phonemic time-delay neural networks. In NIPS, pages 215-223, 1988.
79. C. Watkins. Dynamic alignment kernels. In A.J. Smola, P.L. Bartlett, B. Schölkopf, and D. Schuurmans, editors, Advances in Large Margin Classifiers, pages 39-50, Cambridge, MA, 2000. MIT Press. URL http://www.cs.rhul.ac.uk/home/chrisw/dynk.ps.gz.
80. P. Werbos. Backpropagation: Past and future. In International Conference on Neural Networks, pages 343-353. IEEE, IEEE press, 1988.
81. J.Weston, A. Elisseff, B. Schoelkopf, and M. Tipping. Use of the zero norm with linear models and kernel methods. JMLR, 3:1439-1461, 2003.
82. J. Wichard. Agnostic learning with ensembles of classifiers. In Proc. IJCNN07, Orlando, Florida, Aug 2007. INNS/IEEE.
83. J. Ye, S. Ji, and J. Chen. Multi-class discriminant kernel learning via convex programming. In I. Guyon and A. Saffari, editors, JMLR, Special Topic on Model Selection, volume 9, pages 719- 758, Apr 2008. URL http://www.jmlr.org/ papers/volume9/ye08b/ye08b.pdf.
84. J. Zhu, S. Rosset, T. Hastie, and R. Tibshirani. 1-norm support vector machines. In NIPS, 2003.