Combining linear regression models: When and how?

Journal of the American Statistical Association

Volumn 100, Issue 472, 2005, Pages 1202-1214

  Yuan, Zheng ^a   Yang, Yuhong ^b  

^a UNIVERSITY OF MICHIGAN   (United States)

^b UNIVERSITY OF MINNESOTA   (United States)

Author keywords

Adaptive regression by mixing; Bayesian model averaging; Instability index; Model combining; Model selection; Model uncertainty; Perturbation instability in estimation

Indexed keywords

EID: 29144505412     PISSN: 01621459     EISSN: None     Source Type: Journal    
DOI: 10.1198/016214505000000088     Document Type: Article

References (37)

1. Akaike, H. (1973), "Information Theory and an Extension of the Maximum Likelihood Principle," in Proceedings 2nd International Symposium on Information Theory, eds. B. N. Petrov and F. Csaki, Budapest: Akademia Kiado, pp. 267-281.
2. Allen, D. M. (1974), "The Relationship Between Variable Selection and Data Augmentation and a Method for Prediction," Technometrics, 16, 125-127.
3. Barron, A. R., Birgé, L., and Massart, P. (1999), "Risk Bounds for Model Selection via Penalization," Probability Theory and Related Fields, 113, 301-413.
4. Barron, A. R., Rissanen, J., and Yu, B. (1998), "The Minimum Description Length Principle in Coding and Modeling," IEEE Transactions on Information Theory, 44, 2743-2760.
5. Breiman, L. (1996a), "Bagging Predictors," Machine Learning, 24, 123-140.
6. _ (1996b), "Heuristics of Instability and Stabilization in Model Selection," The Annals of Statistics, 24, 2350-2383.
7. Buckland, S. T., Burnham, K. P., and Augustin, N. H. (1997), "Model Selection: An Integral Part of Inference," Biometrics, 53, 603-618.
8. Chatfield, C. (1995), "Model Uncertainty, Data Mining and Statistical Inference" (with discussion), Journal of the Royal Statistical Society, Ser. A, 158, 419-466.
9. Clarke, B. (2003), "Comparing Bayes Model Averaging and Stacking When Model Approximation Error Cannot Be Ignored," Journal of Machine Learning Research, 4, 683-712.
10. Cook, R. D. (1986), "Assessment of Local Influence" (with discussion), Journal of the Royal Statistical Society, Ser. B, 48, 133-155.
11. Cook, R. D., and Weisberg, S. (1982), Residuals and Influence in Regression, New York: Chapman & Hall.
12. Draper, D. (1995), "Assessment and Propagation of Model Uncertainty" (with discussion), Journal of the Royal Statistical Society, Ser. B, 57, 45-70.
13. Ehrlich, I. (1973), "Participation in Illegitimate Activities: A Theoretical and Empirical Investigation," Journal of Political Economy, 81, 521-565.
14. Fernández, C., Ley, E., and Steel, M. F. J. (2001), "Benchmark Priors for Bayesian Model Averaging," Journal of Econometrics, 100, 381-427.
15. George, E. (2000), "The Variable Selection Problem," Journal of the American Statistical Association, 95, 1304-1308.
16. Guyon, X., and Yao, J. (1999), "On the Underfitting and Overfitting Sets of Models Chosen by Order Selection Criteria," Journal of Multivariate Analysis, 70, 221-249.
17. Hoeting, J., Madigan, D., Raftery, A., and Volinsky, C. (1999), "Bayesian Model Averaging: A Tutorial" (with discussion), Statistical Science, 14, 382-417.
18. Hjort, N. L., and Claeskens, G. (2003), "Frequentist Model Average Estimators" (with discussion), Journal of the American Statistical Association, 98, 879-899.
19. Johnson, R. W. (1996), "Fitting Percentage of Body Fat to Simple Body Measurements," Journal of Statistics Education, 4.
20. Leung, G., and Barron, A. R. (2004), "Information Theory and Mixing Least-Squares Regressions," unpublished manuscript.
21. Madigan, D., and York, J. (1995), "Bayesian Graphical Models for Discrete Data," International Statistical Review, 63, 215-232.
22. Myers, R. H., Montgomery, D. C., and Vining, G. G. (2002), Generalized Linear Models: With Applications in Engineering and the Sciences, New York: Wiley.
23. Penrose, K., Nelson, A., and Fisher, A. (1985), "Generalized Body Composition Prediction Equation for Men Using Simple Measurement Techniques" (abstract), Medicine and Science in Sports and Exercises, 19, 189.
24. Raftery, A., Madigan, D., and Hoeting, J. (1997), "Bayesian Model Averaging for Linear Regression Models," Journal of American Statistical Association, 92, 179-191.
25. Rissanen, J. (1984), "Universal Coding, Information, Prediction, and Estimation," IEEE Transactions on Information Theory, 30, 629-636.
26. Schwarz, G. (1978), "Estimating the Dimension of a Model," The Annals of Statistics, 6, 461-464.
27. Shao, J. (1993), "Linear Model Selection by Cross-Validation," Journal of the American Statistical Association, 88, 486-494.
28. Simonoff, J. S. (1996), Smoothing Methods in Statistics, New York: Springer-Verlag.
29. Stone, M. (1974), "Cross-Validation Choice and Assessment of Statistical Predictions," Journal of the Royal Statistical Society, Ser. B, 36, 111-147.
30. Vandaele, W. (1978), "Participation in Illegitimate Activities; Ehrlich Revisited," in Deterrence and Incapacitation, eds. A. Blumstein, J. Cohen, and D. Nagin, Washington, DC: National Academy of Sciences Press, pp. 270-335.
31. Weisberg, S. (1985), Applied Linear Regression, New York: Wiley.
32. Yang, Y. (2001), "Adaptive Regression by Mixing," Journal of American Statistical Association, 96, 574-588.
33. _ (2003), "Regression With Multiple Candidate Models: Selecting or Mixing?" Statistica Sinica, 13, 783-809.
34. _ (2004a), "Aggregating Regression Procedures to Improve Performance," Bernoulli, 10, 25-47.
35. _ (2004b), "Can the Strengths of AIC and BIC Be Shared? A Conflict Between Model Identification and Regression Estimation," Biometrika, accepted.
36. Yang, Y., and Barron, A. R. (1998), "Asymptotic Property of Model Selection Criteria," IEEE Transactions on Information Theory, 44, 95-116.
37. Zhang, P. (1993), "On the Convergence Rate of Model Selection Criteria," Communications in Statistics, Pan A - Theory and Methods, 22, 2765-2775.