Adaptive forward-backward greedy algorithm for learning sparse representations

IEEE Transactions on Information Theory

Volumn 57, Issue 7, 2011, Pages 4689-4708

  Zhang, Tong ^a  

^a RUTGERS UNIVERSITY   (United States)

Author keywords

Estimation theory; feature selection; greedy algorithms; sparse recovery; statistical learning

Indexed keywords

BASIS FUNCTIONS; ESTIMATION THEORY; GREEDY ALGORITHMS; LEAST SQUARES REGRESSION; LINEAR COMBINATIONS; NOISY OBSERVATIONS; NON-ZERO COEFFICIENTS; NUMBER OF SAMPLES; SPARSE RECOVERY; SPARSE REPRESENTATION; STATISTICAL LEARNING; TARGET FUNCTIONS; THEORETICAL RESULT;

ALGORITHMS; FEATURE EXTRACTION;

NUMBER THEORY;

EID: 79959549699     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2011.2146690     Document Type: Article

References (29)

1. A. Barron, "Universal approximation bounds for superpositions of a sigmoidal function," IEEE Trans. Inf. Theory, vol. 39, no. 3, pp. 930-945, Mar. 1993.
2. A. Barron, A. Cohen, W. Dahmen, and R. DeVore, "Approximation and learning by greedy algorithms," Ann. Statist., vol. 36, pp. 64-94, 2008.
3. P. Bickel, Y. Ritov, and A. Tsybakov, "Simultaneous analysis of Lasso and Dantzig selector," Ann. Statist., vol. 37, no. 4, pp. 1705-1732, 2009.
4. P. Bühlmann, "Boosting for high-dimensional linear models," Ann. Statist., vol. 34, pp. 559-583, 2006. (Pubitemid 44091386)
5. F. Bunea, A. Tsybakov, and M. H.Wegkamp, "Sparsity oracle inequalities for the Lasso," Electron. J. Statist., vol. 1, pp. 169-194, 2007.
6. F. Bunea, A. B. Tsybakov, and M. H. Wegkamp, "Aggregation for Gaussian regression," Ann. Statist., vol. 35, pp. 1674-1697, 2007.
7. E. J. Candes and T. Tao, "Decoding by linear programming," IEEE Trans. Inf. Theory, vol. 51, no. 12, pp. 4203-4215, Dec. 2005. (Pubitemid 41800353)
8. D. M. Chickering, "Optimal structure identification with greedy search," J. Mach. Learn. Res., vol. 3, pp. 507-554, 2003.
9. 1 Penalization for Flexible Function Libraries, 2008, Yale Tech. Rep.
10. C. Couvreur and Y. Bresler, "On the optimality of the backward greedy algorithm for the subset selection problem," SIAM J. Matrix Anal. Appl., vol. 21, no. 3, pp. 797-808, 2000.
11. D. L. Donoho, M. Elad, and V. N. Temlyakov, "Stable recovery of sparse overcomplete representations in the presence of noise," IEEE Trans. Inf. Theory, vol. 52, no. 1, pp. 6-18, Jan. 2006. (Pubitemid 43263116)
12. J. Duchi, S. Shalev-Shwartz, Y. Singer, and A. Tewari, "Composite objective mirror descent," presented at the COLT, 2010.
13. B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, "Least angle regression," Ann. Statist., vol. 32, no. 2, pp. 407-499, 2004. (Pubitemid 41250302)
14. T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning. New York: Springer, 2001.
15. L. Jones, "A simple lemma on greedy approximation in Hilbert space and convergence rates for projection pursuit regression and neural network training," Ann. Statist., vol. 20, no. 1, pp. 608-613, 1992.
16. V. Koltchinskii, "Sparsity in penalized empirical risk minimization," Annales de l'Institut Henri Poincaré, 2008.
17. N. Littlestone, "Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm," Mach. Learn., vol. 2, pp. 285-318, 1988.
18. S. Mallat and Z. Zhang, "Matching pursuits with time-frequency dictionaries," IEEE Trans. Signal Process., vol. 41, no. 12, pp. 3397-3415, Dec. 1993.
19. N. Meinshausen and P. Buhlmann, "High-dimensional graphs and variable selection with the Lasso," The Ann. Statist., vol. 34, pp. 1436-1462, 2006. (Pubitemid 44231168)
20. N. Meinshausen and B. Yu, "Lasso-type recovery of sparse representations for high-dimensional data," Ann. Statist., vol. 37, pp. 246-270, 2009.
21. S. Negahban, P. Ravikumar, M. Wainwright, and B. Yu, "A unified framework for high-dimensional analysis of M-estimators with decomposable regularizers," presented at the NIPS, 2009.
22. G. Pisier, The Volume of Convex Bodies and Banach Space Geometry. New York: Cambridge Univ. Press, 1989.
23. J. A. Tropp, "Greed is good: Algorithmic results for sparse approximation," IEEE Trans. Inf. Theory, vol. 50, no. 10, pp. 2231-2242, Oct. 2004.
24. L. Xiao, "Dual averaging methods for regularized stochastic learning and online optimization," J. Mach. Learn. Res., vol. 11, pp. 2543-2596, 2010.
25. C.-H. Zhang and J. Huang, Model-Selection Consistency of the Lasso in High-Dimensional Linear Regression, Rutgers Univ., 2006, Tech. Rep.
26. T. Zhang, "On the consistency of feature selection using greedy least squares regression," J. Mach. Learn. Res., vol. 10, pp. 555-568, 2009.
27. 1 regularization," Ann. Statist., vol. 37, no. 5A, pp. 2109-2144, 2009.
28. T. Zhang, "Analysis of multi-stage convex relaxation for sparse regularization," J. Mach. Learn. Res., vol. 11, pp. 1087-1107, 2010.
29. P. Zhao and B.Yu, "On model selection consistency of Lasso," J. Mach. Learn. Res., vol. 7, pp. 2541-2567, 2006.