A survey of Bayesian predictive methods for model assessment, selection and comparison

Statistics Surveys

Volumn 6, Issue 1, 2012, Pages 142-228

  Vehtari, Aki ^a   Ojanen, Janne ^a  

^a Department of Biomedical Engineering and Computational Science ^*  (Finland)

Author keywords

Bayesian; Cross validation; Decision theory; Expected utility; Information criteria; Model assessment; Model selection; Predictive

Indexed keywords

EID: 84874387472     PISSN: None     EISSN: 19357516     Source Type: Journal    
DOI: 10.1214/12-ss102     Document Type: Article

References (183)

1. Aitkin, M. (1991). Posterior Bayes Factors (with discussion). Journal of the Royal Statistical Society. Series B (Methodological) 53 111-142.
2. Akaike, H. (1973). Information Theory and an Extension of the Maximum Likelihood Principle. In Second International Symposium on Information Theory (B. N. Petrov and F. Csaki, eds.) 267-281. Academiai Kiado, Budapest. Reprinted in Kotz, S. and Johnson, N. L., editors, (1992). Breakthroughs in Statistics Volume I: Foundations and Basic Theory, pp. 610-624. Springer-Verlag. Mathematical Reviews (MathSciNet): MR483125, Zentralblatt MATH: 0283.62006
3. Akaike, H. (1974). A New Look at the Statistical Model Identification. IEEE Transactions on Automatic Control AC-19 716-723. Mathematical Reviews (MathSciNet): MR423716, Digital Object Identifier: doi:10.1109/TAC.1974.1100705
4. Akaike, H. (1979). A Bayesian Extension of the Minimum AIC Procedure of Autoregressive Model Fitting. Biometrika 66 237-242. Mathematical Reviews (MathSciNet): MR548189, Zentralblatt MATH: 0407.62064, Digital Object Identifier: doi:10.1093/biomet/66.2.237
5. Ando, T. (2007). Bayesian predictive information criterion for the evaluation of hierarchical Bayesian and empirical Bayes models. Biometrika 94 443-458. Mathematical Reviews (MathSciNet): MR2380571, Zentralblatt MATH: 1132.62005, Digital Object Identifier: doi:10.1093/biomet/asm017
6. Ando, T. and Tsay, R. (2010). Predictive likelihood for Bayesian model selection and averaging. International Journal of Forecasting 26 744-763.
7. Arlot, S. and Celisse, A. (2010). A survey of cross-validation procedures for model selection. Statistics surveys 4 40-79. Mathematical Reviews (MathSciNet): MR2602303, Zentralblatt MATH: 1190.62080, Digital Object Identifier: doi:10.1214/09-SS054, Project Euclid: euclid.ssu/1268143839
8. Barbieri, M. M. and Berger, J. O. (2004). Optimal Predictive Model Selection. The Annals of Statistics 32 870-897. Mathematical Reviews (MathSciNet): MR2065192, Zentralblatt MATH: 1092.62033, Digital Object Identifier: doi:10.1214/009053604000000238, Project Euclid: euclid.aos/1085408489
9. Bayarri, M. J. (1987). Comment to J. O. Berger and M. Delampady. Statistical Science 3 342-344.
10. Bayarri, M. J. (2003). Which 'base' distribution for model criticism? In Highly Structured Stochastic Systems (P. J. Green, N. L. Hjort and S. Richardson, eds.) 445-453. Oxford University Press. Mathematical Reviews (MathSciNet): MR2082403
11. Bayarri, M. J. and Berger, J. O. (1999). Quantifying Surprise in the Data and Model Verification. In Bayesian Statistics 6 (J. M. Bernardo, J. O. Berger and A. P. Dawid, eds.) 53-82. Oxford University Press.Mathematical Reviews (MathSciNet): MR1723493, Zentralblatt MATH: 0974.62021
12. Bayarri, M. J. and Berger, J. O. (2000). P Values for Composite Null Models. Journal of the American Statistical Association 95 1127-1142. Mathematical Reviews (MathSciNet): MR1804239, Zentralblatt MATH: 1004.62022
13. Berger, J. O. (1985). Statistical Decision Theory and Bayesian Analysis, 2nd ed. Springer-Verlag. Mathematical Reviews (MathSciNet): MR804611, Zentralblatt MATH: 0572.62008
14. Berger, J. O. and Bernardo, J. M. (1992). On the Development of Reference Priors. In Bayesian Statistics 4 (J. M. Bernardo, J. O. Berger, A. P. Dawid and A. F. M. Smith, eds.) 35-60. Oxford University Press. Mathematical Reviews (MathSciNet): MR1380269
15. Berger, J. and Pericchi, L. (1996). The intrinsic Bayes factor for model selection and prediction. Journal of the American Statistical Association 91 109-122. Mathematical Reviews (MathSciNet): MR1394065, Zentralblatt MATH: 0870.62021, Digital Object Identifier: doi:10.1080/01621459.1996.10476668
16. Bernardo, J. M. (1979). Expected Information as Expected Utility. Annals of Statistics 7 686-690. Mathematical Reviews (MathSciNet): MR527503, Zentralblatt MATH: 0407.62002, Digital Object Identifier: doi:10.1214/aos/1176344689, Project Euclid: euclid.aos/1176344689
17. Bernardo, J. M. (1999). Nested Hypothesis Testing: The Bayesian Reference Criterion. In Bayesian Statistics 6 (J. M. Bernardo, J. O. Berger and A. P. Dawid, eds.) 101-130. Oxford University Press. Mathematical Reviews (MathSciNet): MR1723495, Zentralblatt MATH: 0973.62019
18. Bernardo, J. M. (2005a). Reference Analysis. In Handbook of Statistics, (D. Dey and C. R. Rao, eds.) 25 Elsevier 17-90. Mathematical Reviews (MathSciNet): MR2490522, Zentralblatt MATH: 0682.62018, Digital Object Identifier: doi:10.1016/S0169-7161(05)25002-2
19. Bernardo, J. M. (2005b). Intrinsic credible regions: An objective Bayesian approach to interval estimation. Test 14. 317-384. Mathematical Reviews (MathSciNet): MR2211385, Zentralblatt MATH: 1087.62036, Digital Object Identifier: doi:10.1007/BF02595408
20. Bernardo, J. M. and Bayarri, M. J. (1985). Bayesian model criticism. In Model choice: proceedings of the 4th Franco-Belgian meeting of statisticians (J. P. Florens, M. Mouchart, J. P. Raoult and L. Simar, eds.). Facultés universitaires Saint-Louis, Bruxelles.
21. Bernardo, J. M. and Bermúdez, J. D. (1985). The Choice of Variables in Probabilistic Classification. In Bayesian Statistics 2 (J. M. Bernardo, M. H. deGroot, D. V. Lindley and A. F. M. Smith, eds.) 67-82. Elsevier Science Publishers. Mathematical Reviews (MathSciNet): MR862484, Zentralblatt MATH: 0671.62059
22. Bernardo, J. M. and Juárez, M. A. (2003). Intrinsic Estimation. In Bayesian Statistics 7 (J. M. Bernardo, M. J. Bayarri, J. O. Berger, A. P. Dawid, D. Heckerman, A. F. M. Smith and M. West, eds.) 456-476. Oxford University Press. Mathematical Reviews (MathSciNet): MR2003181, Zentralblatt MATH: 1044.62002
23. Bernardo, J. M. and Rueda, R. (2002). Bayesian hypothesis testing: a reference approach. International Statistical Review 70 351-372.
24. Bernardo, J. M. and Smith, A. F. M. (1994). Bayesian Theory. John Wiley & Sons. Mathematical Reviews (MathSciNet): MR1274699
25. Bhattacharya, S. and Haslett, J. (2007). Importance Re-sampling MCMC for Cross-Validation in Inverse Problems. Bayesian Analysis 2 385-408. Mathematical Reviews (MathSciNet): MR2312288, Digital Object Identifier: doi:10.1214/07-BA217
26. Birgé, L. and Massart, P. (2007). Minimal Penalties for Gaussian Model Selection. Probability Theory and Related Fields 138 33-73. Mathematical Reviews (MathSciNet): MR2288064, Zentralblatt MATH: 1112.62082, Digital Object Identifier: doi:10.1007/s00440-006-0011-8
27. Bornn, L., Doucet, A. and Gottardo, R. (2010). An efficient computational approach for prior sensitivity analysis and cross-validation. The Canadian Journal of Statistics 38 47-64. Mathematical Reviews (MathSciNet): MR2676929
28. Box, G. E. P. (1980). Sampling and Bayes' Inference in Scientific Modelling and Robustness. Journal of the Royal Statistical Society. Series A (General) 143 383-430. Mathematical Reviews (MathSciNet): MR603745, Digital Object Identifier: doi:10.2307/2982063
29. Breiman, L., Friedman, J., Olshen, R. and Stone, C. (1984). Classification and Regression Trees. Chapman and Hall. Mathematical Reviews (MathSciNet): MR726392, Zentralblatt MATH: 0541.62042
30. Brown, P. J., Fearn, T. and Vannucci, M. (1999). The choice of variables in multivariate regression: a non-conjugate Bayesian decision theory approach. Biometrika 86 635-648. Mathematical Reviews (MathSciNet): MR1723783, Zentralblatt MATH: 1072.62510, Digital Object Identifier: doi:10.1093/biomet/86.3.635
31. Brown, P. J., Vannucci, M. and Fearn, T. (1998). Multivariate Bayesian variable selection and prediction. Journal of the Royal Statistical Society. Series B (Statistical Methodology) 60 627-641. Mathematical Reviews (MathSciNet): MR1626005, Zentralblatt MATH: 0909.62022, Digital Object Identifier: doi:10.1111/1467-9868.00144
32. Brown, P. J., Vannucci, M. and Fearn, T. (2002). Bayes model averaging with selection of regressors. Journal of the Royal Statistical Society. Series B (Statistical Methodology) 64 519-536. Mathematical Reviews (MathSciNet): MR1924304, Zentralblatt MATH: 1073.62004, Digital Object Identifier: doi:10.1111/1467-9868.00348
33. Burman, P. (1989). A Comparative Study of Ordinary Cross-Validation, v $v$-Fold Cross-Validation and the Repeated Learning-Testing Methods. Biometrika 76 503-514. Mathematical Reviews (MathSciNet): MR1040644, Zentralblatt MATH: 0677.62065
34. Burman, P., Chow, E. and Nolan, D. (1994). A Cross-Validatory Method for Dependent Data. Biometrika 81 351-358. Mathematical Reviews (MathSciNet): MR1294896, Zentralblatt MATH: 0825.62669, Digital Object Identifier: doi:10.1093/biomet/81.2.351
35. Burman, P. and Nolan, D. (1992). Data dependent estimation of prediction functions. Journal of Time Series Analysis 13 189-207. Mathematical Reviews (MathSciNet): MR1168164, Zentralblatt MATH: 0754.62018, Digital Object Identifier: doi:10.1111/j.1467-9892.1992.tb00102.x
36. Burnham, K. P. and Anderson, D. R. (1998). Model selection and inference. Springer.
37. Burnham, K. P. and Anderson, D. R. (2002). Model Selection and Multi-Model Inference: A Practical Information-Theoretic Approach, 2nd ed. Springer. Mathematical Reviews (MathSciNet): MR1919620, Zentralblatt MATH: 1005.62007
38. Carlin, B. P. and Louis, T. A. (1996). Bayes and Empirical Bayes Methods for Data Analysis 69. Chapman & Hall. Mathematical Reviews (MathSciNet): MR1427749
39. Carlin, B. P. and Spiegelhalter, D. J. (2007). Discussion to 'Estimating the Integrated Likelihood via Posterior Simulation Using the Harmonic Mean Identity'. In Bayesian Statistics 8 (J. M. Bernardo, M. J. Bayarri, J. O. Berger, A. P. Dawid, D. Heckerman, A. F. M. Smith and M. West, eds.) 33-36. Oxford University Press. Mathematical Reviews (MathSciNet): MR2452343
40. Cawley, G. C. and Talbot, N. L. C. (2010). On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation. Journal of Machine Learning Research 11 2079-2107. Mathematical Reviews (MathSciNet): MR2678023, Zentralblatt MATH: 1242.62051
41. Celeux, G., Forbes, F., Robert, C. P. and Titterington, D. M. (2006). Deviance Information Criteria for Missing Data Models. Bayesian Analysis 1 651-674. Mathematical Reviews (MathSciNet): MR2282197, Digital Object Identifier: doi:10.1214/06-BA122
42. Chakrabarti, A. and Ghosh, J. K. (2007). Some Aspects of Bayesian Model Selection for Prediction. In Bayesian Statistics 8 (J. M. Bernardo, M. J. Bayarri, J. O. Berger, A. P. Dawid, D. Heckerman, A. F. M. Smith and M. West, eds.) 51-90. Oxford University Press. Mathematical Reviews (MathSciNet): MR2433189, Zentralblatt MATH: 1252.62014
43. Chen, M.-H., Dey, D. K. and Ibrahim, J. G. (2004). Bayesian criterion based model assessment for categorical data. 91 45-63. http://biomet.oxfordjournals.org/content/91/1/45.abstract Mathematical Reviews (MathSciNet): MR2050459, Zentralblatt MATH: 1132.62301, Digital Object Identifier: doi:10.1093/biomet/91.1.45
44. Chen, M.-H., Shao, Q.-M. and Ibrahim, J. Q. (2000). Monte Carlo Methods in Bayesian Computation. Springer-Verlag. Mathematical Reviews (MathSciNet): MR1742311, Zentralblatt MATH: 0949.65005
45. Chow, G. C. (1981). A comparison of the information and posterior probability criteria for model selection. Journal of Econometrics 16 21-33.
46. Corander, J. and Marttinen, P. (2006). Bayesian Model Learning Based on Predictive Entropy. Journal of Logic, Language, and Information 15 5-20. http://www.jstor.org/stable/40180417 Mathematical Reviews (MathSciNet): MR2254565, Zentralblatt MATH: 1100.62031, Digital Object Identifier: doi:10.1007/s10849-005-9004-8
47. Dietterich, T. G. (1998). Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms. Neural Computation 10 1895-1924.
48. Draper, D. and Fouskakis, D. (2000). A Case Study of Stochastic Optimization in Health Policy: Problem Formulation and Preliminary Results. Journal of Global Optimization 18 399-416.
49. Dupuis, J. A. and Robert, C. P. (1997). Bayesian Variable Selection in Qualitative Models by Kullback-Leibler Projections Working papers, Centre de Recherche en Economie et Statistique.
50. Dupuis, J. A. and Robert, C. P. (2003). Variable selection in qualitative models via an entropic explanatory power. Journal of Statistical Planning and Inference 111 77-94. Mathematical Reviews (MathSciNet): MR1955873, Zentralblatt MATH: 1033.62066, Digital Object Identifier: doi:10.1016/S0378-3758(02)00286-0
51. Efron, B. and Tibshirani, R. J. (1993). An Introduction to the Bootstrap 57. Chapman & Hall. Mathematical Reviews (MathSciNet): MR1270903, Zentralblatt MATH: 0835.62038
52. Epifani, I., MacEachern, S. N. and Peruggia, M. (2008). Case-deletion importance sampling estimators: Central limit theorems and related results. Electronic Journal of Statistics 2 774-806. Mathematical Reviews (MathSciNet): MR2443196, Digital Object Identifier: doi:10.1214/08-EJS259, Project Euclid: euclid.ejs/1221569998
53. Fearn, T., Brown, P. J. and Besbeas, P. (2002). A Bayesian decision theory approach to variable selection for discrimination. Statistics and Computing 12 253-260. Mathematical Reviews (MathSciNet): MR1933511, Digital Object Identifier: doi:10.1023/A:1020702927247
54. Fouskakis, D. and Draper, D. (2008). Comparing stochastic optimization methods for variable selection in binary outcome prediction with application to health policy. Journal of the American Statistical Association 103 1367-1381. Mathematical Reviews (MathSciNet): MR2655719, Digital Object Identifier: doi:10.1198/016214508000001048
55. Fouskakis, D., Ntzoufras, I. and Draper, D. (2009). Population-based reversible-jump Markov chain Monte Carlo for Bayesian variable selection and evaluation under cost limit restrictions. Journal of the Royal Statistical Society, Series C: Applied Statistics 58 383-403. Mathematical Reviews (MathSciNet): MR2750012, Digital Object Identifier: doi:10.1111/j.1467-9876.2008.00658.x
56. Friel, N. and Wyse, J. (2012). Estimating the evidence - a review. Statistica Neerlandica. Early view online. DOI: 10.1111/j.1467-9574.2011.00515.x. Mathematical Reviews (MathSciNet): MR2955421, Digital Object Identifier: doi:10.1111/j.1467-9574.2011.00515.x
57. Fushiki, T. (2011). Estimation of prediction error by using K-fold cross-validation. Statistics and Computing 21 137-146. Mathematical Reviews (MathSciNet): MR2774847, Zentralblatt MATH: 06113565, Digital Object Identifier: doi:10.1007/s11222-009-9153-8
58. Geisser, S. (1975). The Predictive Sample Reuse Method with Applications. Journal of the American Statistical Association 70 320-328.
59. Geisser, S. and Eddy, W. F. (1979). A Predictive Approach to Model Selection. Journal of the American Statistical Association 74 153-160. Mathematical Reviews (MathSciNet): MR529531, Digital Object Identifier: doi:10.1080/01621459.1979.10481632
60. Gelfand, A. E. (1996). Model determination using sampling-based methods. In Markov Chain Monte Carlo in Practice (W. R. Gilks, S. Richardson and D. J. Spiegelhalter, eds.) 145-162. Chapman & Hall. Mathematical Reviews (MathSciNet): MR1397969, Zentralblatt MATH: 0840.62003
61. Gelfand, A. E. (2003). Some comments on model criticism. In Highly Structured Stochastic Systems (P. J. Green, N. L. Hjort and S. Richardson, eds.) 449-453. Oxford University Press. Mathematical Reviews (MathSciNet): MR2082403
62. Gelfand, A. E., Dey, D. K. and Chang, H. (1992). Model Determination using Predictive Distributions with Implementation via Sampling-Based Methods (with discussion). In Bayesian Statistics 4 (J. M. Bernardo, J. O. Berger, A. P. Dawid and A. F. M. Smith, eds.) 147-167. Oxford University Press. Mathematical Reviews (MathSciNet): MR1380275
63. Gelfand, A. E. and Dey, D. K. (1994). Bayesian Model Choice: Asymptotics and Exact Calculations. Journal of the Royal Statistical Society. Series B (Methodological) 56 501-514. Mathematical Reviews (MathSciNet): MR1278223
64. Gelfand, A. E. and Ghosh, S. K. (1998). Model Choice: A Minimum Posterior Predictive Loss Approach. Biometrika 85 1-11. Mathematical Reviews (MathSciNet): MR1627258, Zentralblatt MATH: 0904.62036, Digital Object Identifier: doi:10.1093/biomet/85.1.1
65. Gelman, A., Meng, X.-L. and Stern, H. (1996). Posterior Predictive Assessment of Model Fitness via Realized Discrepancies (with discussion). Statistica Sinica 6 733-807. Mathematical Reviews (MathSciNet): MR1422404, Zentralblatt MATH: 0859.62028
66. Gelman, A., Carlin, J. B., Stern, H. S. and Rubin, D. R. (1995). Bayesian Data Analysis. Chapman & Hall. Mathematical Reviews (MathSciNet): MR1385925
67. Gelman, A., Carlin, J. B., Stern, H. S. and Rubin, D. R. (2003). Bayesian Data Analysis, 2nd ed. Chapman & Hall. Mathematical Reviews (MathSciNet): MR2027492
68. George, E. I. and McCulloch, R. E. (1993). Variable Selection Via Gibbs Sampling. Journal of the American Statistical Association 88 881-889.
69. Geweke, J. (1989). Bayesian Inference in Econometric Models Using Monte Carlo Integration. Econometrica 57 1317-1339. Mathematical Reviews (MathSciNet): MR1035115, Digital Object Identifier: doi:10.2307/1913710
70. Gneiting, T. (2011). Making and Evaluating Point Forecasts. Journal of the American Statistical Association 106 746-762. Mathematical Reviews (MathSciNet): MR2847988, Zentralblatt MATH: 1232.62028, Digital Object Identifier: doi:10.1198/jasa.2011.r10138
71. Gneiting, T., Balabdaoui, F. and Raftery, A. E. (2007). Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society Series B: Statistical Methodology 69 243-268. Mathematical Reviews (MathSciNet): MR2325275, Zentralblatt MATH: 1120.62074, Digital Object Identifier: doi:10.1111/j.1467-9868.2007.00587.x
72. Gneiting, T. and Raftery, A. E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of American Statistical Association 102 359-378. Mathematical Reviews (MathSciNet): MR2345548, Zentralblatt MATH: 05191574, Digital Object Identifier: doi:10.1198/016214506000001437
73. Good, I. J. (1952). Rational Decisions. Journal of the Royal Statistical Society. Series B (Methodological) 14 107-114. Mathematical Reviews (MathSciNet): MR77033
74. Goutis, C. and Robert, C. P. (1998). Model choice in generalised linear models: A Bayesian approach via Kullback-Leibler projections. Biometrika 85 29-37. Mathematical Reviews (MathSciNet): MR1627250, Zentralblatt MATH: 0903.62061, Digital Object Identifier: doi:10.1093/biomet/85.1.29
75. Grünwald, P. D. and Dawid, A. P. (2004). Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory. Annals of Statistics 32 1367-1433. Mathematical Reviews (MathSciNet): MR2089128, Zentralblatt MATH: 1048.62008, Digital Object Identifier: doi:10.1214/009053604000000553, Project Euclid: euclid.aos/1091626173
76. Gutiérrez-Peña, E. (1992). Expected logarithmic divergence for exponential families. In Bayesian Statistics 4 (J. M. Bernardo, J. O. Berger, A. P. Dawid and A. F. M. Smith, eds.) 669-674. Oxford University Press. Mathematical Reviews (MathSciNet): MR1380301
77. Gutiérrez-Peña, E. (1997). A Bayesian Predictive Semiparametric Approach to Variable Selection and Model Comparison in Regression. In Bulletin of the International Statistical Institute, Tome 57. (Proceedings of the 51st Session of the ISI, Invited Papers, Book 1.) 17-29.
78. Gutiérrez-Peña, E. and Walker, S. G. (2001). A Bayesian predictive approach to model selection. Journal of Statistical Planning and Inference 93 259-276. Mathematical Reviews (MathSciNet): MR1822401, Zentralblatt MATH: 1072.62537, Digital Object Identifier: doi:10.1016/S0378-3758(00)00172-5
79. Gutiérrez-Peña, E. and Walker, S. G. (2005). Statistical decision problems and Bayesian nonparametric methods. International Statistical Review 73 309-330.
80. Guttman, I. (1967). The Use of the Concept of a Future Observation in Goodness-of-Fit Problems. Journal of the Royal Statistical Society. Series B (Methodological) 29 83-100. Mathematical Reviews (MathSciNet): MR216699
81. Han, C. and Carlin, B. P. (2000). MCMC methods for computing Bayes factors: A comparative review Research Report No. 2000-001, Division of Biostatistics, University of Minnesota.
82. Held, L., Schrödle, B. and Rue, H. (2010). Posterior and Cross-validatory Predictive Checks: A Comparison of MCMC and INLA. In Statistical Modelling and Regression Structures (T. Kneib and G. Tutz, eds.) 91-110. Springer. Mathematical Reviews (MathSciNet): MR2664630, Digital Object Identifier: doi:10.1007/978-3-7908-2413-1_6
83. Hoeting, J., Madigan, D., Raftery, A. and Volinsky, C. (1999). Bayesian Model Averaging: A Tutorial. Statistical Science 14 382-401. Mathematical Reviews (MathSciNet): MR1765176, Digital Object Identifier: doi:10.1214/ss/1009212519, Project Euclid: euclid.ss/1009212519
84. Hurvich, C. M. and Tsai, C.-L. (1989). Regression and Time Series Model Selection in Small Samples. Biometrika 76 297-307. Mathematical Reviews (MathSciNet): MR1016020, Zentralblatt MATH: 0669.62085, Digital Object Identifier: doi:10.1093/biomet/76.2.297
85. Hurvich, C. M. and Tsai, C.-L. (1991). Bias of the Corrected AIC Criterion for Underfitted Regression and time Series Models. Biometrika 78 499-509. Mathematical Reviews (MathSciNet): MR1130918, Zentralblatt MATH: 1193.62159
86. Ibrahim, J. G. and Chen, M.-H. (1997). Predictive Variable Selection for the Multivariate Linear Model. Biometrics 53 465-478. http://www.jstor.org/stable/2533950
87. Ibrahim, J. G., Chen, M.-H. and Sinha, D. (2001). Criterion-based methods for Bayesian model assessment. Statistica Sinica 11 419-443. Mathematical Reviews (MathSciNet): MR1844533, Zentralblatt MATH: 1037.62017
88. Ibrahim, J. G. and Laud, P. W. (1994). A Predictive Approach to the Analysis of Designed Experiments. Journal of the American Statistical Association 89 309-319. Mathematical Reviews (MathSciNet): MR1266302, Zentralblatt MATH: 0791.62080, Digital Object Identifier: doi:10.1080/01621459.1994.10476472
89. Jaakkola, T. S. (2001). Tutorial on variational approximation methods. In Advanced Mean Field Methods (M. Opper and D. Saad, eds.) 129-160. The MIT Press.
90. Jeffreys, H. (1961). Theory of Probability, 3rd ed. Oxford University Press (1st edition 1939).Mathematical Reviews (MathSciNet): MR187257
91. Jonathan, P., Krzanowski, W. J. and McCarthy, W. V. (2000). On the use of cross-validation to assess performance in multivariate prediction. Statistics and Computing 10 209-229.
92. Jordan, M. I., Ghahramani, Z., Jaakkola, T. S. and Saul, L. K. (1999). An introduction to variational methods for graphical models. Machine Learning 37 183-233. Zentralblatt MATH: 1033.68081
93. Jylänki, P., Vanhatalo, J. and Vehtari, A. (2011). Gaussian Process Regression with a Student-t Likelihood. Journal of Machine Learning Research 12 3227-3257. Mathematical Reviews (MathSciNet): MR2877599
94. Karabatsos, G. (2006). Bayesian nonparametric model selection and model testing. Journal of Mathematical Psychology 50. Mathematical Reviews (MathSciNet): MR2215142, Zentralblatt MATH: 1099.62005, Digital Object Identifier: doi:10.1016/j.jmp.2005.07.003
95. Kass, R. E. and Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association 90 773-795.
96. Key, J. T., Pericchi, L. R. and Smith, A. F. M. (1999). Bayesian Model Choice: What and Why? In Bayesian Statistics 6 (J. M. Bernardo, J. O. Berger and A. P. Dawid, eds.) 343-370. Oxford University Press. Mathematical Reviews (MathSciNet): MR1723504, Zentralblatt MATH: 0956.62007
97. Kullback, S. and Leibler, R. A. (1951). On Information and Sufficiency. Annals of Mathematical Statistics 22 79-86. Mathematical Reviews (MathSciNet): MR39968, Zentralblatt MATH: 0042.38403, Digital Object Identifier: doi:10.1214/aoms/1177729694, Project Euclid: euclid.aoms/1177729694
98. Lacoste-Julien, S., Huszár, F. and Ghahramani, Z. (2011). Approximate inference for the loss-calibrated Bayesian. Journal of Machine Learning Research: Workshop and Conference Proceedings 15 416-424. AISTATS 2011 special issue.
99. Laud, P. and Ibrahim, J. (1995). Predictive model selection. Journal of the Royal Statistical Society. Series B (Methodological) 57 247-262. Mathematical Reviews (MathSciNet): MR1325389
100. Leamer, E. E. (1979). Information Criteria for Choice of Regression Models: A Comment. Econometrica 47 507-510. Mathematical Reviews (MathSciNet): MR525791, Digital Object Identifier: doi:10.2307/1914197
101. Leung, D. H.-Y. (2005). Cross-validation in nonparametric regression with outliers. Annals of Statistics 33 2291-2310. Mathematical Reviews (MathSciNet): MR2211087, Zentralblatt MATH: 1086.62055, Digital Object Identifier: doi:10.1214/009053605000000499, Project Euclid: euclid.aos/1132936564
102. Lindley, D. V. (1968). The Choice of Variables in Multiple Regression. Journal of the Royal Statistical Society. Series B (Methodological) 30 31-66. Mathematical Reviews (MathSciNet): MR231492
103. Lo, A. Y. (1987). A Large Sample Study of the Bayesian Bootstrap. Annals of Statistics 15 360-375. Mathematical Reviews (MathSciNet): MR885742, Zentralblatt MATH: 0617.62032, Digital Object Identifier: doi:10.1214/aos/1176350271, Project Euclid: euclid.aos/1176350271
104. MacKay, D. J. C. (1992). A Practical Bayesian Framework for Backpropagation Networks. Neural Computation 4 448-472.
105. Marin, J.-M. and Robert, C. P. (2010). Importance sampling methods for Bayesian discrimination between embedded models. In Frontiers of Statistical Decision Making and Bayesian Analysis (M. H. Chen, D. K. Dey, P. Müller, D. Sun and K. Ye, eds.) 14, 513-553. Springer.
106. Marriott, J. M., Spencer, N. M. and Pettitt, A. N. (2001). A Bayesian Approach to Selecting Covariates for Prediction. Scandinavian Journal of Statistics 28 87-97. Mathematical Reviews (MathSciNet): MR1844350, Digital Object Identifier: doi:10.1111/1467-9469.00225
107. Marshall, E. C. and Spiegelhalter, D. J. (2003). Approximate cross-validatory predictive checks in disease mapping models. Statistics in Medicine 22 1649-1660.
108. San Martini, A. and Spezzaferri, F. (1984). A Predictive Model Selection Criterion. Journal of the Royal Statistical Society. Series B (Methodological) 46 296-303. Mathematical Reviews (MathSciNet): MR781890
109. Mason, D. M. and Newton, M. A. (1992). A Rank Statistics Approach to the Consistency of a General Bootstrap. Annals of Statistics 20 1611-1624. Mathematical Reviews (MathSciNet): MR1186268, Zentralblatt MATH: 0777.62045, Digital Object Identifier: doi:10.1214/aos/1176348787, Project Euclid: euclid.aos/1176348787
110. McCullagh, P. and Nelder, J. A. (1989). Generalized Linear Models 37, Second ed. Chapman & Hall. Mathematical Reviews (MathSciNet): MR727836
111. McCulloch, R. E. (1989). Local Model Influence. Journal of the American Statistical Association 84 473-478. http://www.jstor.org/stable/2289932
112. Meng, X.-L. (1994). Posterior Predictive p-Values. Annals of Statistics 22 1142-1160. Mathematical Reviews (MathSciNet): MR1311969, Zentralblatt MATH: 0820.62027, Digital Object Identifier: doi:10.1214/aos/1176325622, Project Euclid: euclid.aos/1176325622
113. Meyer, M. C. and Laud, P. W. (2002). Predictive Variable Selection in Generalized Linear Models. Journal of the American Statistical Association 97 859-871. Mathematical Reviews (MathSciNet): MR1941415, Zentralblatt MATH: 1048.62071, Digital Object Identifier: doi:10.1198/016214502388618654
114. Minka, T. (2001). A Family of Algorithms for Approximate Bayesian Inference PhD thesis, Massachusetts Institute of Technology. Mathematical Reviews (MathSciNet): MR2717007
115. Mitchell, T. J. and Beauchamp, J. J. (1988). Bayesian Variable Selection in Linear Regression (with discussion). Journal of the American Statistical Association 83. Mathematical Reviews (MathSciNet): MR997578, Zentralblatt MATH: 0673.62051, Digital Object Identifier: doi:10.1080/01621459.1988.10478694
116. Miyamoto, J. M. (1999). Quality-Adjusted Life Years (QALY) Utility Models under Expected Utility and Rank Dependent Utility Assumptions. Journal of Mathematical Psychology 43 201-237. Mathematical Reviews (MathSciNet): MR1689346, Zentralblatt MATH: 0942.91032, Digital Object Identifier: doi:10.1006/jmps.1999.1256
117. Moody, J. E. (1992). The Effective Number of Parameters: An Analysis of Generalization and Regularization in Nonlinear Learning Systems. In Advances in Neural Information Processing Systems 4 (J. E. Moody, S. J. Hanson and R. P. Lippmann, eds.) 847-854. Morgan Kaufmann Publishers.
118. Murata, N., Yoshizawa, S. and Amari, S.-I. (1994). Network Information Criterion-Determining the number of hidden units for an Artificial Neural Network model. IEEE Transactions on Neural Networks 5 865-872.
119. Nadeau, C. and Bengio, S. (2000). Inference for the Generalization Error. In Advances in Neural Information Processing Systems 12 (S. A. Solla, T. K. Leen and K.-R. Müller, eds.) 307-313. MIT Press.
120. Neal, R. M. (1998). Assessing Relevance Determination Methods Using DELVE. In Neural Networks and Machine Learning (C. M. Bishop, ed.) 97-129. Springer-Verlag.
121. Newton, M. A. and Raftery, A. E. (1994). Approximate Bayesian Inference with the Weighted Likelihood Bootstrap (with discussion). Journal of the Royal Statistical Society. Series B (Methodological) 56 3-48. Mathematical Reviews (MathSciNet): MR1257793
122. Nickisch, H. and Rasmussen, C. E. (2008). Approximations for Binary Gaussian Process Classification. Journal of Machine Learning Research 9 2035-2078. Mathematical Reviews (MathSciNet): MR2452620, Zentralblatt MATH: 1225.62087
123. Nott, D. J. and Leng, C. (2010). Bayesian projection approaches to variable selection in generalized linear models. Computational Statistics & Data Analysis 54 3227-3241. http://dx.doi.org/10.1016/j.csda.2010.01.036 Mathematical Reviews (MathSciNet): MR2727748
124. O'Hagan, A. (1995). Fractional Bayes Factors for Model Comparison (with discussion). Journal of the Royal Statistical Society. Series B (Methodological) 57 99-138. Mathematical Reviews (MathSciNet): MR1325379
125. O'Hagan, A. (2003). HSSS model criticism. In Highly Structured Stochastic Systems (P. J. Green, N. L. Hjort and S. Richardson, eds.) 423-444. Oxford University Press. Mathematical Reviews (MathSciNet): MR2082418
126. O'Hagan, A. and Forster, J. (2004). Bayesian Inference, 2nd ed. Kendalls's Advanced Theory of Statistics 2B. Arnold.
127. Opper, M. and Winther, O. (2000). Gaussian Processes for Classification: Mean-Field Algorithms. Neural Computation 12 2655-2684.
128. Orr, M. J. L. (1996). Introduction to Radial Basis Function Networks [online] Technical Report, Centre for Cognitive Science, University of Edinburgh. April 1996. Available at http://www.anc.ed.ac.uk/mjo/papers/intro.ps.gz.
129. Peruggia, M. (1997). On the Variability of Case-Deletion Importance Sampling Weights in the Bayesian Linear Model. Journal of the American Statistical Association 92 199-207. Mathematical Reviews (MathSciNet): MR1436108, Zentralblatt MATH: 0889.62020, Digital Object Identifier: doi:10.1080/01621459.1997.10473617
130. Plummer, M. (2008). Penalized loss functions for Bayesian model comparison. Biostatistics (Oxford, England) 9 523-39.
131. Raftery, A. E. and Zheng, Y. (2003). Discussion: Performance Of Bayesian Model Averaging. Journal of American Statistical Association 98 931-938.
132. Raftery, A. E., Newton, M. A., Satagopan, J. M. and Krivitsky, P. (2007). Estimating the Integrated Likelihood via Posterior Simulation Using the Harmonic Mean Identity (with discussion). In Bayesian Statistics 8 (J. M. Bernardo, M. J. Bayarri, J. O. Berger, A. P. Dawid, D. Heckerman, A. F. M. Smith and M. West, eds.) 1-45. Oxford University Press. Mathematical Reviews (MathSciNet): MR2433201, Zentralblatt MATH: 1252.62038
133. Raiffa, H. and Schlaifer, R. (2000). Applied Statistical Decision Theory. John Wiley & Sons. Mathematical Reviews (MathSciNet): MR1789469
134. Rasmussen, C. E. and Ghahramani, Z. (2003). Bayesian Monte Carlo. In Advances in Neural Information Processing Systems 15 (S. Becker, S. Thrun and K. Obermayer, eds.) 489-496. MIT Press, Cambridge, MA. Mathematical Reviews (MathSciNet): MR2003529
135. Rasmussen, C. E. and Williams, C. K. I. (2006). Gaussian Processes for Machine Learning. The MIT Press. Mathematical Reviews (MathSciNet): MR2514435
136. Rasmussen, C. E., Neal, R. M., Hinton, G. E., van Camp, D., Revow, M., Ghahramani, Z., Kustra, R. and Tibshirani, R. (1996). The DELVE Manual [online]. Version 1.1. Available at http//ftp.cs.utoronto.ca/pub/neuron/delve/doc/manual.ps.gz.
137. Rencher, A. C. and Pun, F. C. (1980). Inflation of R2 in Best Subset Regression. Technometrics 22 49-53.
138. Reunanen, J. (2003). Overfitting in Making Comparisons Between Variable Selection Methods. Journal of Machine Learning Research 3 1371-1382.
139. Richardson, S. (2002). Discussion to 'Bayesian measures of model complexity and fit' by Spiegelhalter et al. Journal of the Royal Statistical Society. Series B (Statistical Methodology) 64 626-627. Mathematical Reviews (MathSciNet): MR1979380, Zentralblatt MATH: 1067.62010, Digital Object Identifier: doi:10.1111/1467-9868.00353
140. Robert, C. P. (1996). Intrinsic losses. Theory and decision 40 191-214. Mathematical Reviews (MathSciNet): MR1385186, Digital Object Identifier: doi:10.1007/BF00133173
141. Robert, C. P. (2001). The Bayesian Choice: from Decision-Theoretic Motivations to Computational Implementation, 2nd ed. Springer. Mathematical Reviews (MathSciNet): MR1835885
142. Robert, C. P. and Wraith, D. (2009). Computational methods for Bayesian model choice. In The 29th International Workshop on Bayesian Inference and Maximum Entropy Methods in Science and Engineering. AIP Proceedings 1193 251-262.
143. Robins, J. M., van Der Vaart, A. and Ventura, V. (2000). Asymptotic Distribution of P Values in Composite Null Models. Journal of the American Statistical Association 95 1143-1156. Mathematical Reviews (MathSciNet): MR1804240, Zentralblatt MATH: 1072.62522
144. Rubin, D. B. (1981). The Bayesian Bootstrap. Annals of Statistics 9 130-134. Mathematical Reviews (MathSciNet): MR600538, Digital Object Identifier: doi:10.1214/aos/1176345338, Project Euclid: euclid.aos/1176345338
145. Rubin, D. B. (1984). Bayesianly Justifiable and Relevant Frequency Calculations for the Applied Statistician. Annals of Statistics 12 1151-1172. Mathematical Reviews (MathSciNet): MR760681, Zentralblatt MATH: 0555.62010, Digital Object Identifier: doi:10.1214/aos/1176346785, Project Euclid: euclid.aos/1176346785
146. Rueda, R. (1992). A Bayesian alternative to parametric hypothesis testing. Test 1 61-67. http://www.springerlink.com/content/37501636313583g2/ Mathematical Reviews (MathSciNet): MR1266129, Zentralblatt MATH: 0764.62003, Digital Object Identifier: doi:10.1007/BF02562662
147. Sawa, T. (1978). Information Criteria for Discriminating Among Alternative Regression Models. Econometrica 46 1273-1291. Mathematical Reviews (MathSciNet): MR513693, Digital Object Identifier: doi:10.2307/1913828
148. Shao, J. (1993). Linear Model Selection by Cross-Validation. Journal of the American Statistical Association 88 486-494. Mathematical Reviews (MathSciNet): MR1224373, Zentralblatt MATH: 0773.62051, Digital Object Identifier: doi:10.1080/01621459.1993.10476299
149. Shen, X., Huang, H.-C. and Ye, J. (2004). Inference after Model Selection. Journal of the American Statistical Association 99 751-762. http://www.jstor.org/stable/27590445 Mathematical Reviews (MathSciNet): MR2090908, Zentralblatt MATH: 1117.62423, Digital Object Identifier: doi:10.1198/016214504000001097
150. Shibata, R. (1989). Statistical aspects of model selection. In From data to model (J. C. Willems, ed.) 215-240. Springer-Verlag.
151. Shimodaira, H. (2000). Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference 90 227-244. Mathematical Reviews (MathSciNet): MR1795598, Zentralblatt MATH: 0958.62011, Digital Object Identifier: doi:10.1016/S0378-3758(00)00115-4
152. Sinha, D., Chen, M.-H. and Ghosh, S. K. (1999). Bayesian Analysis and Model Selection for Interval-Censored Survival Data. Biometrics 55 585-590. Mathematical Reviews (MathSciNet): MR1705161, Digital Object Identifier: doi:10.1111/j.0006-341X.1999.00585.x
153. Skare, Ø., Bølviken, E. and Holden, L. (2003). Improved sampling-importance resampling and reduced bias importance sampling. Scandivanian Journal of Statistics 30 719-737. Mathematical Reviews (MathSciNet): MR2155479, Digital Object Identifier: doi:10.1111/1467-9469.00360
154. Spiegelhalter, D. J., Best, N. G., Carlin, B. P. and van der Linde, A. (2002). Bayesian measures of model complexity and fit (with discussion). Journal of the Royal Statistical Society. Series B (Statistical Methodology) 64 583-639. Mathematical Reviews (MathSciNet): MR1979380, Zentralblatt MATH: 1067.62010, Digital Object Identifier: doi:10.1111/1467-9868.00353
155. Stern, H. S. and Cressie, N. (2000). Posterior predictive model checks for disease mapping models. Statistics in Medicine 19 2377-2397.
156. Stone, M. (1974). Cross-Validatory Choice and Assessment of Statistical Predictions (with discussion). Journal of the Royal Statistical Society. Series B (Methodological) 36 111-147. Mathematical Reviews (MathSciNet): MR356377
157. Stone, M. (1977). An Asymptotic Equivalence of Choice of Model by Cross-Validation and Akaike's Criterion. Journal of the Royal Statistical Society. Series B (Methodological) 39 44-47. Mathematical Reviews (MathSciNet): MR501454
158. Sugiyama, M., and Müller, K.-R. (2005). Input-dependent estimation of generalization error under covariate shift. Statistics & Decisions 23 249-279. Mathematical Reviews (MathSciNet): MR2255627
159. Sugiyama, M., Krauledat, M. and M̈ller, K.-R. (2007). Covariate Shift Adaptation by Importance Weighted Cross Validation. Journal of Machine Learning Research 8 985-1005.
160. Sundararajan, S. and Keerthi, S. S. (2001). Predictive Approaches for Choosing Hyperparameters in Gaussian Processes. Neural Computation 13 1103-1118.
161. Takeuchi, K. (1976). Distribution of Informational Statistics and a Criterion of Model Fitting (in Japanese). Suri-Kagaku (Mathematic Sciences) 153 12-18.
162. Tibshirani, R. (1996). Regression Shrinkage and Selection via the Lasso. Journal of the Royal Statistical Society. Series B (Methodological) 58 267-288. Mathematical Reviews (MathSciNet): MR1379242
163. Tibshirani, R. J. and Tibshirani, R. (2009). A bias correction for the minimum error rate in cross-validation. Annals of Applied Statistics 3 822-829. Mathematical Reviews (MathSciNet): MR2750683, Zentralblatt MATH: 1166.62311, Digital Object Identifier: doi:10.1214/08-AOAS224, Project Euclid: euclid.aoas/1245676196
164. Tierney, L. and Kadane, J. B. (1986). Accurate Approximations for Posterior Moments and Marginal Densities. Journal of the American Statistical Association 81 82-86. Mathematical Reviews (MathSciNet): MR830567, Zentralblatt MATH: 0587.62067, Digital Object Identifier: doi:10.1080/01621459.1986.10478240
165. Tran, M.-N., Nott, D. J. and Leng, C. (2011). The predictive Lasso. Statistics and Computing 1-16. Mathematical Reviews (MathSciNet): MR2950086, Zentralblatt MATH: 1252.62075, Digital Object Identifier: doi:10.1007/s11222-011-9279-3
166. Trottini, M. and Spezzaferri, F. (2002). A generalized predictive criterion for model selection. The Canadian Journal of Statistics 30 79-96. Mathematical Reviews (MathSciNet): MR1907678, Digital Object Identifier: doi:10.2307/3315866
167. Vanhatalo, J., Pietiläinen, V. and Vehtari, A. (2010). Approximate inference for disease mapping with sparse Gaussian processes. Statistics in Medicine 29 1580-1607. Mathematical Reviews (MathSciNet): MR2758849
168. Vannucci, M., Brown, P. J. and Fearn, T. (2003). A Decision theoretical approach to wavelet regression on curves with a high number of regressors. Journal of Statistical Planning and Inference 112 195-212. Mathematical Reviews (MathSciNet): MR1961730, Zentralblatt MATH: 1032.62004, Digital Object Identifier: doi:10.1016/S0378-3758(02)00333-6
169. Varma, S. and Simon, R. (2006). Bias in error estimation when using cross-validation for model selection. BMC Bioinformatics 7 91. http://www.ncbi.nlm.nih.gov/pubmed/16504092
170. Vehtari, A. (2002). Discussion of "Bayesian measures of model complexity and fit" by Spiegelhalter et al. Journal of the Royal Statistical Society. Series B (Statistical Methodology) 64 620. Mathematical Reviews (MathSciNet): MR1979380, Zentralblatt MATH: 1067.62010, Digital Object Identifier: doi:10.1111/1467-9868.00353
171. Vehtari, A. and Lampinen, J. (2002). Bayesian Model Assessment and Comparison Using Cross-Validation Predictive Densities. Neural Computation 14 2439-2468.
172. Vehtari, A. and Lampinen, J. (2004). Model Selection via Predictive Explanatory Power Technical Report No. B38, Helsinki University of Technology, Laboratory of Computational Engineering.
173. Vlachos, P. K. and Gelfand, A. E. (2003). On the Calibration of Bayesian Model Choice Criteria. Journal of Statistical Planning and Inference 111 223-234. Mathematical Reviews (MathSciNet): MR1955883, Zentralblatt MATH: 1033.62028, Digital Object Identifier: doi:10.1016/S0378-3758(02)00304-X
174. Watanabe, S. (2009). Algebraic Geometry and Statistical Learning Theory. Cambridge University Press. Mathematical Reviews (MathSciNet): MR2554932, Zentralblatt MATH: 1180.93108
175. Watanabe, S. (2010a). Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory. Journal of Machine Learning Research 11 3571-3594. Mathematical Reviews (MathSciNet): MR2756194, Zentralblatt MATH: 1242.62024
176. Watanabe, S. (2010b). Equations of states in singular statistical estimation. Neural Networks 23 20-34.
177. Watanabe, S. (2010c). A limit theorem in singular regression problem. Advanced Studies of Pure Mathematics 57 473-492. Mathematical Reviews (MathSciNet): MR2648274, Zentralblatt MATH: 1210.62102
178. Weng, C.-S. (1989). On a Second-Order Asymptotic Property of the Bayesian Bootstrap Mean. Annals of Statistics 17 705-710. Mathematical Reviews (MathSciNet): MR994261, Zentralblatt MATH: 0672.62027, Digital Object Identifier: doi:10.1214/aos/1176347136, Project Euclid: euclid.aos/1176347136
179. Yang, Y. (2005). Can the Strengths of AIC and BIC Be Shared? A Conflict between Model Indentification and Regression Estimation. Biometrika 92 937-950. Mathematical Reviews (MathSciNet): MR2234196, Digital Object Identifier: doi:10.1093/biomet/92.4.937
180. Yang, Y. (2007). Consistency of Cross Validation for Comparing Regression Procedures. The Annals of Statistics 35 2450-2473. Mathematical Reviews (MathSciNet): MR2382654, Zentralblatt MATH: 1129.62039, Digital Object Identifier: doi:10.1214/009053607000000514, Project Euclid: euclid.aos/1201012968
181. Young, A. S. (1987a). On a Bayesian criterion for choosing predictive sub-models in linear regression. Metrika 34 325-339. Mathematical Reviews (MathSciNet): MR916417, Digital Object Identifier: doi:10.1007/BF02613164
182. Young, A. S. (1987b). On the information criterion for selecting regressors. Metrika 34 185-194. Mathematical Reviews (MathSciNet): MR899029, Digital Object Identifier: doi:10.1007/BF02613148
183. Zhu, L. and Carlin, B. P. (2000). Comparing hierarchical models for spatio-temporally misaligned data using the deviance information criterion. Statistics in Medicine 19 2265-2278