Estimation Stability With Cross-Validation (ESCV)

Journal of Computational and Graphical Statistics

Volumn 25, Issue 2, 2016, Pages 464-492

  Lim, Chinghway ^a   Yu, Bin ^b  

^a NATIONAL UNIVERSITY OF SINGAPORE   (Singapore)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Lasso; Model selection; Parameter estimation; Prediction

Indexed keywords

EID: 84971449545     PISSN: 10618600     EISSN: 15372715     Source Type: Journal    
DOI: 10.1080/10618600.2015.1020159     Document Type: Article

References (53)

1. H.Akaike, (1974), “A New Look at the Statistical Model Identification,” IEEE Transactions on Automatic Control, 19, 716--723.
2. D.M.Allen, (1974), “The Relationship Between Variable Selection and Data Argumentation and a Method of Prediction,” Technometrics, 16, 125–127.
3. S.Arlot,, and M.Lerasle, (2012), “V-Fold Cross Validation and V-Fold Penalization in Least-Squares Density Estimation,” arVix: 1210.5830.
4. F.Bach, (2008), “Bolasso: Model Consistent Lasso Estimation through the Bootstrap,” in Proceedings of the 25th International Conference on Machine Learning, pp. 33--40.
5. P.Bickel,, Y.Ritov,, and A.Tsybakov, (2009), “Simultaneous Analysis of Lasso and Dantzig Selector,” Annals of Statistics, 37, 1705–1732.
6. O.Bousquet,, and A.Elisseeff, (2002), “Stability and Generalization,” Journal of Machine Learning Research, 2, 499–426.
7. L.Breiman, (1995), “Better Subset Regression Using the Nonnegative Garrote,” Technometrics, 37373--384.
8. L.Breiman, (1996), “Heuristics of Instability and Stabilization in Model Selection,” The Annals of Statistics, 24, 2350–2383.
9. L.Breiman, (2001), “Statistical Modeling: The Two Cultures,” Statistical Science, 16, 199–231.
10. P.Bühlmann,, and B.Yu, (2003), “Boosting with the L2 Loss: Regression and Classification,” Journal of the American Statistical Association, 98, 324–339.
11. J.Chen,, and Z.Chen, (2008), “Extended Bayesian Information Criteria for Model Selection With Large Model Spaces,” Biometrika, 95, 759–771.
12. B.Efron, (1979), “Bootstrap Methods: Another Look at the Jackknife,” The Annals of Statistics, 7, 1–26.
13. B.Efron,, T.Hastie,, I.Johnstone,, and R.Tibshirani, (2004), “Least Angle Regression,” The Annals of Statistics, 32, 407–499.
14. S.P.Ellis, (1998), “Instability of Least Squares, Least Absolute Deviation and Least Median of Squares Linear Regression,” Statistical Science, 13, 337–350.
15. J.Friedman,, T.Hastie,, and R.Tibshirani, (2010), “Regularization Paths for Generalized Linear Models via Coordinate Descent,” Journal of Statistical Software, 33, 1–22.
16. J.H.Friedman, (2001), “Greedy Function Approximation: A Gradient Boosting Machine,” The Annals of Statistics, 29, 1189–1232.
17. T.Hastie,, R.Tibshirani,, and J.Friedman, (2002), The Elements of Statistical Learning: Data Mining, Inference, and Prediction, New York, NY: Springer.
18. A.-C.Haury,, F.Mordelet,, P.Vera-Licona,, and J.-P.Vert, (2012), “TIGRESS: Trustful Inference of Gene REgulation using Stability Selection,” BMC Systems Biology, 6.
19. N.J.Higham, (1996), Accuracy and Stability of Numerical Algorithms, Philadelphia, PA: SIAM.
20. A.Hoerl, (1962), “Application of Ridge Analysis to Regression Problems,” Chemical Engineering Progress58.
21. C.Hurvich,, and C.-L.Tsai, (1989), “Regression and Time Series Model Selection in Small Samples,” Biometrika, 76, 297–307.
22. K.Kay,, T.Naselaris,, R.Prenger,, and J.Gallant, (2008), “Identifying Natural Images from Human Brain Activity,” Nature, 452, 352–355.
23. S.Kutin,, and P.Niyogi, (2002), “Almost-Everywhere Algorithmic Stability and Generalization Error,” in Proceedings of the Eighteenth Conference on Uncertainty in Artificial Intelligence, pp. 275–282.
24. J.P.La Salle, (1976), The Stability of Dynamical Systems, CBMS-NSF Regional Conference Series in Applied Mathematics, Philadelphia, PA: SIAM.
25. C.Leng,, Y.Lin,, and G.Wahba, (2006), “A Note on the Lasso and Related Procedures in Model Selection,” Statistica Sinica, 16, 1273–1284.
26. H.Liu,, K.Roeder,, and L.Wasserman, (2010), “Stability Approach to Regularization Selection (StARS) for High Dimensional Graphical Models,” Advances in Neural Information Processing Systems, 23.
27. J.Mairal,, and B.Yu, (2012), “Complexity Analysis of the Lasso Regularization Path,” in Proceedings of the 29th International Conference on Machine Learning (ICML), Edinburgh, Scotland, UK, pp. 353--360.
28. J.Mairal, (2006), “High-Dimensional Graphs and Variable Selection with the Lasso,” The Annals of Statistics, 34, 1436–1462.
29. N.Meinshausen,, and P.Bühlmann, (2010), “Stability Selection,” Journal of the Royal Statistical Society, Series B, 72, 417--473.
30. N.Meinshausen,, G.Rocha,, and B.Yu, (2007), “Discussion: A Tale of Three Cousins: Lasso, L2Boosting and Dantzig,” The Annals of Statistics, 35, 2373–2384.
31. N.Meinshausen,, and B.Yu, (2009), “Lasso-Type Recovery of Sparse Representations for High-Dimensional Data,” Annals of Statistics, 37, 246–270.
32. S.Mukherjee,, P.Niyogi,, T.Poggio,, and R.Rifkin, (2006), “Learning Theory: Stability is Sufficient for Generalization and Necessary and Sufficient for Consistency of Empirical Risk Minimization,” Advances in Computational Mathematics, 26, 161–193.
33. Y.Nan,, and Y.Yang, (2014), “Variable Selection Diagnostics Measures for High-Dimensional Regression,” Journal of Computational and Graphical Statistics, 23, 636–656.
34. S.Pradervand,, M.Maurya,, and S.Subramaniam, (2006), “Identification of Signaling Components Required for the Prediction of Cytokine Release in RAW 264.7 Macrophages,” Genome Biology, 7.
35. G.Schwarz, (1978), “Estimating the Dimension of a Model,” The Annals of Statistics.
36. R.D.Shah,, and R.J.Samworth, (2013), “Variable Selection With Error Control: Another Look at Stability Selection,” Journal of the Royal Statistical Society, Series B, 75, 55–80.
37. J.Shao, (1993), “Linear Model Selection by Cross-Validation,” Journal of the American Statistical Association, 88, 486–494.
38. J.Shao, (1996), “Bootstrap Model Selection,” Journal of the American Statistical Association, 91, 353–360.
39. M.Stone, (1974), “Cross-Validation Choice and Assessment of Statistical Prediction,” Journal of the Royal Statistical Society, Series B, 39, 44–47.
40. R.Tibshirani, (1996), “Regression Shrinkage and Selection via the Lasso,” Journal of the Royal Statistical Society, Series B, 58, 267–288.
41. A.N.Tikhonov, (1943), “On the Stability of Inverse Problems,” Doklady Akademii Nauk SSSR, 39, 195–198.
42. J.Tropp, (2006), “Just Relax: Convex Programming Methods for Identifying Sparse Signals in Noise,” IEEE Transactions on Information Theory, 52.
43. 1-Constrained Quadratic Programming (Lasso),” IEEE Transactions on Information Theory, 55, 2183–2202.
44. L.Wasserman,, and K.Roeder, (2009), “High-Dimensional Variable Selection,” The Annals of Statistics, 37, 2178–2201.
45. Y.Yang, (2007), “Consistency of Cross Validation for Comparing Regression Procedures,” The Annals of Statistics, 35, 2450–2473.
46. B.Yu, (2013), “Stability,” Bernoulli, 19, 1484–1500.
47. Z.Yuan,, and Y.Yang, (2005), “Combining Linear Regression Models: When and How?,” Journal of the American Statistical Association, 100, 1202–1214.
48. C.-H.Zhang,, and J.Huang, (2008), “The Sparsity and Bias of the Lasso Selection in High-Dimensional Linear Regression,” The Annals of Statistics, 36, 1567–1594.
49. P.Zhang, (1993), “Model Selection Via Multifold Cross-validation,” The Annals of Statistics, 21, 299–313.
50. T.Zhang, (2011), “Adaptive Forward-Backward Greedy Algorithm for Learning Sparse Representations,” IEEE Transactions on Information Theory, 57, 4689–4708.
51. T.Zhang,, and B.Yu, (2005), “Boosting With Early Stopping: Convergence and Consistency,” The Annals of Statistics, 33, 1538–1579.
52. P.Zhao,, and B.Yu, (2006), “On Model Selection Consistency of Lasso,” Journal of Machine Learning Research, 7, 2541–2563.
53. P.Zhao, (2007), “Stagewise Lasso,” The Journal of Machine Learning Research, 8, 2701–2726.