When does more regularization imply fewer degrees of freedom? Sufficient conditions and counterexamples

Biometrika

Volumn 101, Issue 4, 2014, Pages 771-784

  Kaufman, S ^a   Rosset, S ^a  

^a TEL AVIV UNIVERSITY   (Israel)

Author keywords

Degrees of freedom; Model selection; Optimism; Regularization

Indexed keywords

EID: 84985994363     PISSN: 00063444     EISSN: 14643510     Source Type: Journal    
DOI: 10.1093/biomet/asu034     Document Type: Article

References (24)

1. AKAIKE, H. (1974). A new look at the statistical model identification. IEEE Trans. Auto. Contr. 19, 716-23.
2. BOYD, S. P. & VANDENBERGHE, L. (2004). Convex Optimization. Cambridge: Cambridge University Press.
3. CHERKASSKY, V. S. &MULIER, F. M. (2007). Learning from Data: Concepts, Theory, and Methods. 2nd ed. New York: Wiley.
4. DAVIDOV, O. (2006). Constrained estimation and the theorem of Kuhn-Tucker. J. Appl. Math. Decis. Sci. 2006, article no. 92970.
5. EFRON, B. (1983). Estimating the error rate of a prediction rule: Improvement on cross-validation. J. Am. Statist. Assoc. 78, 316-31.
6. EFRON, B. (2004). The estimation of prediction error: Covariance penalties and cross-validation (with Discussion). J. Am. Statist. Assoc. 99, 619-32.
7. EFRON, B., HASTIE, T. J., JOHNSTONE, I. M. & TIBSHIRANI, R. J. (2004). Least angle regression (with Discussion). Ann. Statist. 32, 407-99.
8. GOLUB, G. H., HEATH, M. & WAHBA, G. (1979). Generalized cross-validation as a method for choosing a good ridge parameter. Technometrics 21, 215-23.
9. HASTIE, T. J., TIBSHIRANI, R. J. & FRIEDMAN, J. H. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd ed. New York: Springer.
10. HOERL, A. E. (1962). Application of ridge analysis to regression problems. Chem. Eng. Progr. 58, 54-9.
11. KATO, K. (2009). On the degrees of freedom in shrinkage estimation. J. Mult. Anal. 100, 1338-52.
12. KR äMER, N. & SUGIYAMA, M. (2011). The degrees of freedom of partial least squares regression. J. Am. Statist. Assoc. 106, 697-705.
13. KUMAR KATTUMANNIL, S. (2009). On Stein's identity and its applications. Statist. Prob. Lett. 79, 1444-9.
14. MALLOWS, C. L. (1973). Some comments on Cp. Technometrics 15, 661-75.
15. SCHWARZ, G. (1978). Estimating the dimension of a model. Ann. Statist. 6, 461-4.
16. STEIN, C. M. (1981). Estimation of the mean of a multivariate normal distribution. Ann. Statist. 9, 1135-51.
17. STONE, M. (1974). Cross-validatory choice and assessment of statistical predictions (with Discussion). J. R. Statist. Soc. B, 111-47.
18. TIBSHIRANI, R. J. (1996). Regression shrinkage and selection via the lasso. J. R. Statist. Soc. B 58, 267-88.
19. TIBSHIRANI, R. J. & TAYLOR, J. (2011). The solution path of the generalized lasso. Ann. Statist. 39, 1335-71.
20. VAPNIK, V. N. (2000). The Nature of Statistical Learning Theory. 2nd ed. New York: Springer.
21. WAHBA, G. (1983). Bayesian "confidence intervals" for the cross-validated smoothing spline. J. R. Statist. Soc. B 45, 133-50.
22. YE, J. (1998). On measuring and correcting the effects of data mining and model selection. J. Am. Statist. Assoc. 93, 120-31.
23. ZOU, H. & HASTIE, T. J. (2005). Regularization and variable selection via the elastic net. J. R. Statist. Soc. B 67, 301-20.
24. ZOU, H., HASTIE, T. J. & TIBSHIRANI, R. J. (2007). On the degrees of freedom of the lasso. Ann. Statist. 35, 2173-92.