An interior-point method for large-scale ℓ1-regularized least squares

IEEE Journal on Selected Topics in Signal Processing

Volumn 1, Issue 4, 2007, Pages 606-617

  Kim, Seung Jean ^a   Koh, Kwangmoo ^a   Lustig, Michael ^a   Boyd, Stephen ^a   Gorinevsky, Dimitry ^a  

^a Stanford University   (United States)

Author keywords

1 regularization; Basis pursuit denoising; Compressed sensing; Compressive sampling; Convex optimization; Interior point methods; Least squares; Preconditioned conjugate gradients

Indexed keywords

ALGORITHMS; MAGNETIC RESONANCE IMAGING; MATHEMATICAL PROGRAMMING; OPTIMIZATION; SAMPLING; SPURIOUS SIGNAL NOISE;

BASIS PURSUIT DENOISING; COMPRESSIVE SAMPLING; CONVEX OPTIMIZATION; INTERIOR-POINT METHODS; LEAST-SQUARES PROGRAMS (LSPS); PRECONDITIONED CONJUGATE GRADIENTS; SPARSE SIGNAL RECOVERY;

SIGNAL RECONSTRUCTION;

EID: 39449109476     PISSN: 19324553     EISSN: None     Source Type: Journal    
DOI: 10.1109/JSTSP.2007.910971     Document Type: Article

References (63)

1. R. Barlow, D. Bartholomew, J. Bremner, and H. Brunk, Statistical Inference Under Order Restrictions: The Theory and Application of Isotonic Regression. New York: Wiley, 1972.
2. J. Borwein and A. Lewis, Convex Analysis and Nonlinear Optimization New York: Springer, 2000.
3. S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge, U.K.: Cambridge University Press, 2004.
4. E. Candès, "Compressive sampling," Proc. Int. Congr. Mathematics 2006.
5. 1-magic: A Collection of MATLAB Routines for Solving the Convex Optimization Programs Central to Compressive Sampling 2006 Online. Available: Www.acm.caltech.edu/11magic/
6. E. Candès, J. Romberg, and T. Tao, "Stable signal recovery from incomplete and inaccurate measurements," Commun. Pure Appl. Math. vol. 59, no. 8, pp. 1207-1223, 2005.
7. E. Candès, J. Romberg, and T. Tao, "Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information," IEEE Trans. Inf. Theory, vol. 52, no. 2, pp. 489-509, Feb. 2006.
8. S. Chen, D. Donoho, and M. Saunders, "Atomic decomposition by basis pursuit," SIAM Rev., vol. 43, no. 1, pp. 129-159, 2001.
9. I. Daubechies, M. Defrise, and C. De Mol, "An iterative thresholding algorithm for linear inverse problems with a sparsity constraint," Commun. Pure Appl. Mathe., vol. 57, pp. 1413-1541, 2004.
10. J. Demmel, Applied Numerical Linear Algebra. Cambridge, U.K.: Cambidge Univ. Press, 1997.
11. D. Donoho, "Compressed sensing," IEEE Trans. Inf. Theory, vol. 52, no. 4, pp. 1289-1306, Apr. 2006.
12. D. Donoho, M. Elad, and V. Temlyakov, "Stable recovery of sparse overcomplete representations in the presence of noise," IEEE Trans. Inf. Theory, vol. 52, no. 1, pp. 6-18, Jan. 2006.
13. D. Donoho, V. Stodden, and Y. Tsaig, SparseLab: Seeking Sparse Solutions to Linear Systems of Equations 2006 [Online]. Available: http:// sparselab.stanford.edu/
14. 1-norm minimization problems when the solution may be sparse," Manuscript 2006 [Online]. Available: http://www.stanford.edu/
15. B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, "Least angle regression," Ann. Statist., vol. 32, no. 2, pp. 407-499, 2004.
16. M. Elad, B. Matalon, and M. Zibulevsky, "Image denoising with shrinkage and redundant representations," in Proc. IEEE Computer Society Conf. Computer Vision and Pattern Recognition (CVPR), 2006, vol. 2, pp. 1924-1931.
17. M. Figueiredo and R. Nowak, "A bound optimization approach to wavelet-based image deconvolution," in Proc. IEEE Int. Conf. Image Processing (ICIP), 2005, pp. 782-785.
18. M. Figueiredo, R. Nowak, and S. Wright, "Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems," IEEE J. Select. Topics Signal Process., 2007.
19. J. Friedman, T. Hastie T, and R. Tibshirani, Pathwise Coordinate Optimization 2007 [Online]. Available: Www-stat.stanford.edu/hastie/ pub.htm
20. J. Fuchs, "Recovery of exact sparse representations in the presence of noise," IEEE Trans. Inf. Theory, vol. 51, no. 10, pp. 3601-3608, Oct. 2005.
21. D. Gorinevsky, "Monotonic regression filters for trending gradual deterioration faults," in Proc. American Control Conf. (ACC), 2004, pp. 5394-5399.
22. D. Gorinevsky, S.-J. Kim, S. Bear, S. Boyd, and G. Gordon, "Optimal estimation of deterioration from diagnostic image sequence," IEEE Trans. Signal Process. 2007 [Online]. Available: http:// www.stanford.edu/gorin/papers/MonoImage07twocol.pdf, Available from, Submitted to
23. 1 regularized minimization with applications to compressed sensing," Manuscript 2007 [Online]. Available: http://www.dsp.ece.rice.edu/cs/
24. T. Hastie, S. Rosset, R. Tibshirani, and J. Zhu, "The entire regularization path for the support vector machine," J. Mach. Learning Res., vol. 5, pp. 1391-1415, 2004.
25. T. Hastie, J. Taylor, R. Tibshirani, and G. Walther, "Forward stagewise regression and the monotone lasso," Electron. J. Statist., vol. 1, no. 1-29, 2007.
26. T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning. New York: Springer-Verlag, 2001, Springer Series in Statistics.
27. C. Johnson, J. Seidel, and A. Sofer, "Interior point methodology for 3-D PET reconstruction," IEEE Trans. Med. Imag., vol. 19, no. 4, pp. 271-285, 2000.
28. C. T. Kelley, "Iterative methods for linear and nonlinear equations," Frontiers in Applied Mathematics, vol. 16, 1995, SIAM: Philadelphia, PA.
29. K. Knight and W. Fu, "Asymptotics for lasso-type estimators," Ann. Statist., vol. 28, no. 5, pp. 1356-1378, 2000.
30. K. Koh, S.-J. Kim, and S. Boyd, "An interior-point method for l 1-regularized logistic regression," J. Mach. Learning Res., vol. 8, pp. 1519-1555, 2007.
31. M. Lobo, L. Vandenberghe, S. Boyd, and H. Lebret, "Applications of second-order cone programming," Linear algebra and its Applications vol. 284, pp. 193-228, 1998.
32. D. Luenberger, Linear and Nonlinear Programming, 2nd ed. Reading, MA: Addison-Wesley, 1984.
33. N. Meinshausen and P. Büuhlmann, "High-dimensional graphs and variable selection with the lasso," Ann. Statist., vol. 34, no. 3, pp. 1436-1462, 2006.
34. The MOSEK Optimization Tools Version 2.5. User's Manual and Reference 2002 Online. Available: Www.mosek.com, MOSEK ApS Available from
35. G. Narkiss and M. Zibulevsky, Sequential Subspace Optimization Method for Large-Scale Unconstrained Problems The Technion, Haifa, Israel, Tech. Rep. CCIT No 559, 2005.
36. Y. Nesterov, "Gradient methods for minimizing composite objective function," 2007, CORE Discussion Paper 2007/76 [Online]. Available: http://www.optimization-online.org/DB_HTML/2007/09/1784.html
37. Y. Nesterov and A. Nemirovsky, "Interior-point polynomial methods in convex programming," Studies in Applied Mathematics, vol. 13, 1994, SIAM: Philadelphia, PA.
38. A. Neumaier, "Solving ill-conditioned and singular linear systems: A tutorial on regularization," SIAM Rev., vol. 40, no. 3, pp. 636-666, 1998.
39. J. Nocedal and S. Wright, "Numerical optimization," Springer Series in Operations Research, 1999, Springer.
40. M. Osborne, B. Presnell, and B. Turlach, "A new approach to variable selection in least squares problems," IMA J. Numer. Anal., vol. 20, no. 3, pp. 389-403, 2000.
41. S. Osher, M. Burger, D. Goldfarb, J. Xu, and W. Yin, "An iterative regularization method for total variation based image restoration," SIAM J. Multiscale Modeling and Simulation, vol. 4, no. 2, pp. 460-489, 2005.
42. C. Paige and M. Saunders, "LSQR: An algorithm for sparse linear equations and sparse least squares," ACM Trans. Mathemat. Software vol. 8, no. 1, pp. 43-71, 1982.
43. B. Polyak, "Introduction to optimization," Optimization Software 1987, Translated from Russian.
44. L. Portugal, M. Resende, G. Veiga, and J. Júdice, "A truncated primal-infeasible dual-feasible network interior point method," Networks, vol. 35, no. 2, pp. 91-108, 2000.
45. T. Robertson, F. Wright, R., and Dykstra, Order Restricted Statistical Inference. New York: Wiley, 1988.
46. S. Rosset,, L. Saul, Y. Weiss, and L. Bottou, Eds., "Tracking curved regularized optimization solution paths," in Advances in Neural Information Processing Systems 17. Cambridge, MA: MIT Press, 2005.
47. L. Rudin, S. Osher, and E. Fatemi, "Nonlinear total variation based noise removal algorithms," Physics D, vol. 60, pp. 259-268, 1992.
48. Y. Saad, Iterative Methods for Sparse Linear Systems, 2nd ed. Philadelphia, PA: SIAM, 2003.
49. S. Samar, D. Gorinevsky, and S. Boyd, "Moving horizon filter for monotonic trends," in Proc. 43rd IEEE Conf. Decision and Control, 2004, vol. 3, pp. 3115-3120.
50. M. Saunders, PDCO: Primal-Dual Interior Method for Convex Objectives 2002 [Online]. Available: http://www.stanford.edu/group/SOL/software/ pdco.html
51. N. Z. Shor, "Minimization methods for non-differentiable functions," Springer Series in Computational Mathematics, 1985, Springer.
52. R. Tibshirani, "Regression shrinkage and selection via the lasso," J. Roy. Statist. Soc., ser. B, vol. 58, no. 1, pp. 267-288, 1996.
53. J. Tropp, "Just relax: Convex programming methods for identifying sparse signals in noise," IEEE Trans. Inf. Theory, vol. 52, no. 3, pp. 1030-1051, Mar. 2006.
54. Y. Tsaig and D. Donoho, "Extensions of compressed sensing," Signal Process., vol. 86, no. 3, pp. 549-571, 2006.
55. L. Vandenberghe and S. Boyd, "A primal-dual potential reduction method for problems involving matrix inequalities," Mathemat. Programm., ser. B, vol. 69, pp. 205-236, 1995.
56. M. Wainwright, "Sharp thresholds for high-dimensional and noisy recovery of sparsity," in Proc. 44th Annu. Allerton Conf. Communication, Control, and Computing, 2006.
57. S. Wright, "Primal-dual interior-point methods," Society for Industrial and Applied Mathematics, 1997, SIAM: Philadelphia, PA.
58. Y. Ye, Interior Point Algorithms: Theory and Analysis. New York: Wiley, 1997.
59. W. Yin, S. Osher, J. Darbon, and D. Goldfarb, Bregman Iterative Algorithms for Compessed Sensing and Related Problems 2007 [Online]. Available: http://www.math.ucla.edu/applied/cam/index.shtml
60. M. Yuan and L. Lin, "Model selection and estimation in regression with grouped variables," J. Roy. Statist. Soc., ser. B, vol. 68, no. 1, pp. 49-67, 2006.
61. P. Zhao and B. Yu, "On model selection consistency of lasso," J. Mach. Learning Res., vol. 7, pp. 2541-2563, 2006.
62. H. Zou, "The adaptive lasso and its oracle properties," J. Amer. Statist. Assoc., vol. 101, no. 476, pp. 1418-1429, 2006.
63. H. Zou and T. Hastie, "Regularization and variable selection via the elastic net," Journal of the Roy. Statist. Soc., ser. B, vol. 67, no. 2, pp. 301-320, 2005.