When do stepwise algorithms meet subset selection criteria

Annals of Statistics

Volumn 35, Issue 2, 2007, Pages 870-887

  Huo, Xiaoming ^a,b   Ni, Xuelei ^a,b  

^a GEORGIA INSTITUTE OF TECHNOLOGY   (United States)

^b Georgia Institute of Technology   (United States)

Author keywords

Concurrent optimal subset; Convex optimization; Model selection; Stepwise algorithms; Subset selection

Indexed keywords

EID: 49749140446     PISSN: 00905364     EISSN: None     Source Type: Journal    
DOI: 10.1214/009053606000001334     Document Type: Article

References (51)

1. AKAIKE, H. (1973). Information theory and an extension of the maximum likelihood principle. In Second International Symposium on Information Theory (B. N. Petrov and F. Csáki, eds.) 267-281. Akadémiai Kiadó, Budapest. MR0483125
2. BURNHAM, K. P. and ANDERSON, D. A. (2002). Model Selection andMultimodel Inference: A Practical Information-Theoretic Approach, 2nd ed. Springer, New York.
3. CHEN, S. S. (1995). Basis pursuit. Ph.D. dissertation, Dept. Statistics, Stanford Univ.
4. CHEN, S. S., DONOHO, D. L. and SAUNDERS, M. A. (1998). Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20 33-61.
5. Reprinted at SIAM Rev. 43 (2001) 129-159. MR1639094, MR1854649
6. DEVORE, R. A. and TEMLYAKOV, V. N. (1996). Some remarks on greedy algorithms. Adv. Comput. Math. 5 173-187. MR1399379
7. 1 minimization. Proc. Natl. Acad. Sci. USA 100 2197-2202. MR1963681
8. DONOHO, D. L., ELAD, M. and TEMLYAKOV, V. N. (2006). Stable recovery of sparse overcomplete representations in the presense of noise. IEEE Trans. Inform. Theory 52 6-18. MR2237332
9. DONOHO, D. L. and Huo, X. (2001). Uncertainty principles and ideal atomic decomposition. IEEE Trans. Inform. Theory 47 2845-2862. MR1872845
10. DONOHO, D. L. and JOHNSTONE, I. M. (1995). Adapting to unknown smoothness via wavelet shrinkage. J. Amer. Statist. Assoc. 90 1200-1224. MR1379464
11. EFRON, B., HASTIE, T., JOHNSTONE, I. and TIBSHIRANI, R. (2004). Least angle regression (with discussion). Ann. Statist. 32 407-499. MR2060166
12. ELAD, M. and BRUCKSTEIN, A. M. (2002). A generalized uncertainty principle and sparse representation in pairs of bases. IEEE Trans. Inform. Theory 48 2558-2567. MR1929464
13. FAN, J. and LI, R. (2001). Variable selection via nonconvave penalized likelihood and its oracle properties. J. Amer. Statist. Assoc. 96 1348-1360. MR1946581
14. FAN, J. and LI, R. (2006). Statistical challenges with high dimensionality: Feature selection in knowledge discovery. In International Congress of Mathematicians 3 595-622. European Math. Soc., Zürich. MR2275698
15. FAN, J. and PENG, H. (2004). Nonconcave penalized likelihood with a diverging number of parameters. Ann. Statist. 32 928-961. MR2065194
16. FOSTER, D. P. and GEORGE, E. I. (1994). The risk inflation criterion for multiple regression. Ann. Statist. 22 1947-1975. MR1329177
17. FUCHS, J.-J. (2004). On sparse representations in arbitrary redundant bases. IEEE Trans. Inform. Theory 50 1341-1344. MR2094894
18. FURNIVAL, G. and WILSON, R. (1974). Regressions by leaps and bounds. Technometrics 16 499-511.
19. GATU, C. and KONTOGHIORGHES, E. J. (2006). Branch-and-bound algorithms for computing the best-subset regression models. J. Comput. Graph. Statist. 15 139-156. MR2269366
20. GEORGE, E. I. (2000). The variable selection problem. J. Amer. Statist. Assoc. 95 1304-1308. MR1825282
21. p-statistic. The Statistician 45 49-56.
22. GOLUB, G. H. and VAN LOAN, C. F. (1996). Matrix Computations, 3rd ed. Johns Hopkins Univ. Press, Baltimore. MR1417720
23. GRIBONVAL, R., FIGUERAS I VENTURA, R. M. and VANDERGHEYNST, P. (2005). A simple test to check the optimality of sparse signal approximations. In Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP'05) S 717-720. IEEE Press, Piscataway, NJ. Available at Its1pc19.epfl.ch/repository/Gribonval2005_1167.pdf. A longer version is available at ftp.iri.sa.fr/techreports/2004/PI-1661.pdf.
24. GRIBONVAL, R. and NIELSEN, M. (2003). Sparse representations in unions of bases. IEEE Trans. Inform. Theory 49 3320-3325. MR2045813
25. HASTIE, T., ROSSET, S., TIBSHIRANI, R. and ZHU, J. (2004). The entire regularization path for the support vector machine. J. Mack Learn. Res. 5 1391-1415.
26. HASTIE, T., TIBSHIRANI, R. and FRIEDMAN, J. (2001). The Elements of Statistical Learning: Data Mining, Inference and Prediction. Springer, New York. MR1851606
27. HUO, X. and NI, X. S. (2005). When do stepwise algorithms meet subset selection criteria? Technical report, Georgia Inst. Technology. Available at www2.isye.gatech.edu/statistics/papers/.
28. JAIN, A. and ZONGKER, D. (1997). Feature selection: Evaluation, application and small sample performance. IEEE Trans. Pattern Analysis and Machine Intelligence 19 153-158.
29. KOHAVI, R. and JOHN, G. H. (1997). Wrappers for feature subset selection. Artificial Intelligence 97 273-324.
30. MALIOUTOV, D. M., CETIN, M. and WILLSKY, A. S. (2005). Homotopy continuation for sparse signal representation. In Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP'05) S 733-736. IEEE Press, Piscataway, NJ.
31. P. Technometrics 15 661-675.
32. MILLER, A. J. (2002). Subset Selection in Regression, 2nd ed. Chapman and. Hall/CRC, Boca Raton, FL. MR2001193
33. NARENDRA, P. M. and FUKUNAGA, K. (1977). A branch and bound algorithm for feature subset selection. IEEE Trans. Comput. 26 917-922.
34. NATARAJAN, B. K. (1995). Sparse approximate solutions to linear systems. SIAM J. Comput. 24 227-234. MR1320206
35. NI, X. S. (2006). New results in detection, estimation and model selection. Ph.D. dissertation, Georgia Inst. Technology. Available at etd.gatech.edu/.
36. NI, X. S. and HUO, X. (2005). Enhanced leaps-and-bounds methods in subset selections with additional optimality tests. INFORMS QSR student paper competition finalist. Available atqsr.section.informs.org/ qsr_activities.htm.
37. NI, X. S. and HUO, X. (2006). Regressions by enhanced leaps-and-bounds via optimality tests (LBOT). Technical report, Georgia Inst. Technology. Available at www2.isye.gatech.edu/statistics/papers/.
38. OSBORNE, M. R. (1985). Finite Algorithms in Optimization and Data Analysis. Wiley, Chichester. MR0817717
39. OSBORNE, M. R., PRESNELL, B. and TURLACH, B. A. (2000). On the LASSO and its dual. J. Comput. Graph. Statist. 9 319-337. MR1822089
40. OSBORNE, M. R., PRESNELL, B. and TURLACH, B. A. (2000). A new approach to variable selection in least squares problems. IMA J. Numer. Anal. 20 389-403. MR1773265
41. RIDOUT, M. S. (1988). Algorithm AS 233: An improved branch and bound algorithm for feature subset selection. Appl. Statist. 37 139-147.
42. ROBERTS, S. J. (1984). Algorithm AS .199: A branch and bound algorithm for determining the optimal feature subset of given size. Appl. Statist. 33 236-241.
43. ROCKAFELLAR, R. T. (1970). Convex Analysis. Princeton Univ. Press. MR0274683
44. SANTOSA, F. and SYMES, W. W. (1986). Linear inversion of band-limited reflection seismograms. SIAM J. Sci. Statist. Comput. 7 1307-1330. MR0857796
45. SCHWARZ, G. (1978). Estimating the dimension of a model. Ann. Statist. 6 461-464. MR0468014
46. TIBSHIRANI, R. (1996). Regression shrinkage and selection via the lasso. J. Roy. Statist. Soc. Ser. B 58 267-288. MR1379242
47. TROPP, J. A. (2004). Greed is good: Algorithmic results for sparse approximation. IEEE Trans. Inform. Theory 50 2231-2242. MR2097044
48. TROPP, J. A. (2006). Just relax: Convex programming methods for identifying sparse signals in noise. IEEE Trans. Inform. Theory 52 1030-1051. MR2238069
49. TURLACH, B. A. (2004). Discussion of "Least angle regression," by B. Efron, T. Hastie, I. Johnstone and R. Tibshirani. Ann. Statist. 32 481-490.
50. Wu, C.-F. J. (1993). Construction of supersaturated designs through partially aliased interactions. Biometrika 80 661-669. MR1248029
51. ZOU, H. (2005). The adaptive Lasso and its oracle properties. Unpublished manuscript.