Regularization in regression: Comparing Bayesian and frequentist methods in a poorly informative situation

Bayesian Analysis

Volumn 7, Issue 2, 2012, Pages 477-502

  Celeux, Gilles ^a   El Anbari, Mohammed ^b   Marin, Jean Michel ^c   Robert, Christian P ^d  

^a INRIA SACLAY   (France)

^b CADI AYYAD UNIVERSITY   (Morocco)

^c UNIV MONTPELLIER   (France)

^d UNIVERSITÉ PARIS DAUPHINE   (France)

Author keywords

Calibration; Dantzig selector; Elastic net; Lasso; Model choice; Noninformative priors; Regularization methods; Zellner's g prior

Indexed keywords

EID: 84865715409     PISSN: 19360975     EISSN: 19316690     Source Type: Journal    
DOI: 10.1214/12-BA716     Document Type: Article

References (44)

1. Bartlett, M. (1957). A comment on D.V. Lindley's statistical paradox. Biometrika, 44:533-534.
2. Berger, J., Pericchi, L., and Varshavsky, I. (1998). Bayes factors and marginal distributions in invariant situations. Sankhya A, 60:307-321.
3. Bottolo, L. and Richardson, S. (2010). Evolutionary stochastic search for Bayesian model exploration. Bayesian Analysis, 5(3):583-618.
4. Breiman, L. and Friedman, J.H. (1985). Estimating optimal transformations for multiple regression and correlation. Journal of the American Statistical Association, 85(391):580-598.
5. Brown, J. and Vannucci, M. (1998). Multivariate Bayesian variable selection and prediction. Journal of the Royal Statistical Society Series B, 60(3):627-641.
6. Butler, R. and Wood, A. (2002). Laplace approximations for hypergeometric functions with matrix arguments. Annals of Statistics, 30:1155-1177.
7. Candes, E. and Tao, T. (2007). The Dantzig Selector: statistical estimation when p is much larger than n. Annals of Statistics, 35(6):2313-2351.
8. Casella, G. and Moreno, E. (2006). Objective Bayesian variable selection. Journal of the American Statistical Association, 101(473):157-167.
9. Celeux, G., Marin, J.-M., and Robert, C. (2006). Sélection bayésienne de variables en régression linéaire. Journal de la Société Française de Statistique, 147(1):59-79.
10. Chipman, H. (1996). Bayesian variable selection with related predictors. Canadian Journal of Statistics, 1:17-36.
11. Cui, W. and George, E. (2008). Empirical Bayes vs. fully Bayes variable selection. Journal of Statistical Planning and Inference, 138:888-900.
12. de Finetti, B. (1972). Probability, Induction and Statistics. John Wiley, New York.
13. DeGroot, M. (1973). Doing what comes naturally: Interpreting a tail area as a posterior probability or as a likelihood ratio. Journal of the American Statistical Association, 68:966-969.
14. Dupuis, J. and Robert, C. (2003). Bayesian variable selection in qualitative models by Kullback-Leibler projections. Journal of Statistical Planning and Inference, 111:77-94.
15. Fernandez, C., Ley, E., and Steel, M. (2001). Benchmark priors for Bayesian model averaging. Journal of Econometrics, 100:381-427.
16. Foster, D. and George, E. (1994). The risk ination criterion for multiple regression. Annals of Statistics, 22:1947-1975.
17. George, E. (2000). The variable selection problem. Journal of the American Statistical Association, 95:1304-1308.
18. George, E. and Foster, D. (2000). Calibration and empirical Bayes variable selection. Biometrika, 87(4):731-747.
19. George, E. and McCulloch, R. (1993). Variable selection via Gibbbs sampling. Journal of the American Statistical Association, 88:881-889.
20. George, E. and McCulloch, R. (1997). Approaches to Bayesian variable selection. Sta-tistica Sinica, 7:339-373.
21. Guo, R. and Speckman, P. (2009). Bayes factor consistency in linear models. In The 2009 International Workshop on Objective Bayes Methodology, Philadelphia, June 5-9, 2009. URL http://www-stat.wharton.upenn.edu/statweb/Conference/ OBayes09/AbstractPapers/speckman.pdf.
22. Hoerl, A. and Kennard, R. (1970). Ridge regression: biased estimation for non orthogonal problems. Technometrics, 12:55-67.
23. Kass, R. and Raftery, A. (1995). Bayes factor and model uncertainty. Journal of the American Statistical Association, 90:773-795.
24. Kass, R. and Wasserman, L. (1995). A reference Bayesian test for nested hypotheses and its relationship to the Schwarz criterion. Journal of the American Statistical Association, 90:928-934.
25. Kohn, R., Smith, M., and Chan, D. (2001). Nonparametric regression using linear combinations of basis functions. Statistics and Computing, 11:313-322.
26. Liang, F., Paulo, R., Molina, G., Clyde, M., and Berger, J. (2008). Mixtures of g-priors for Bayesian variable selection. Journal of the American Statistical Association, 103(481):410-423.
27. Lindley, D. (1957). A statistical paradox. Biometrika, 44:187-192.
28. Marin, J. and Robert, C. (2007). Bayesian Core: A Practical Approach to Computa-tional Bayesian Statistics. Springer-Verlag, New York.
29. Mitchell, T. and Beauchamp, J. (1988). Bayesian variable selection in linear regression. Journal of the American Statistical Association, 83:1023-1032.
30. Nott, D. J. and Green, P. J. (2004). Bayesian variable selection and the Swendsen-Wang algorithm. Journal of Computational and Graphical Statistics, 13:1-17.
31. Park, T. and Casella, G. (2008). The Bayesian lasso. Journal of the American Statistical Association, 103(473):681-686.
32. Penrose, K., Nelson, A., and Fisher, A. (1985). Generalized body composition prediction equation for men using simple measurement techniques. Medicine and Science in Sports and Exercise, 17(2):189.
33. Philips, R. and Guttman, I. (1998). A new criterion for variable selection. Statistics and Probability Letters, 38:11-19.
34. Rao, C. (1973). Linear Statistical Inference and its Applications. John Wiley, New York.
35. Robert, C. (1993). A note on the Je_reys-Lindley paradox. Statistica Sinica, 3:601-608.
36. Robert, C. (2001). The Bayesian Choice. Springer-Verlag, 2 edition.
37. Schneider, U. and Corcoran, J. (2004). Perfect sampling for Bayesian variable selection in a linear regression model. Journal of Statistical Planning and Inference, 126:153-171.
38. Smith, M. and Kohn, R. (1996). Nonparametric regression using Bayesian variable selection. Journal of Econometrics, 75:317-343.
39. Tibshirani, R. (1996). Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society Series B, 58(1):267-288.
40. Yuan, M. and Lin, Y. (2007). Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society Series B, 68(1):49-67.
41. Zellner, A. (1986). On assessing prior distributions and Bayesian regression analysis with g-prior distribution regression using Bayesian variable selection. In Bayesian inference and decision techniques: Essays in Honor of Bruno De Finetti, pages 233-243. North-Holland / Elsevier.
42. Zellner, A. and Siow, A. (1980). Posterior odds ratios for selected regression hypotheses. In Bayesian Statistics, pages 585-603. Valencia: University Press. (Proceedings of the first Valencia meeting).
43. Zou, H. (2006). The adaptive Lasso and its oracle properties. Journal of the American Statistical Association, 101:1418-1429.
44. Zou, H. and Hastie, T. (2005). Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society Series B, 67(2):301-320. 478, 484, 485