A fast hybrid algorithm for large-scale ℓ1-regularized logistic regression

Journal of Machine Learning Research

Volumn 11, Issue , 2010, Pages 713-741

  Shi, Jianing ^a   Yin, Wotao ^b   Osher, Stanley ^c   Sajda, Paul ^a  

^a Fu Foundation School of Engineering and Applied Science   (United States)

^b Rice University   (United States)

^c UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

1 regularization; Fixed point continuation; Large scale; Logistic regression; Supervised learning

Indexed keywords

BENCHMARK DATA; COMPUTATIONALLY EFFICIENT; FEATURE SELECTION; FIXED POINT CONTINUATION; FIXED POINTS; GEOMETRIC RATES; GLOBAL CONVERGENCE; HIGH DIMENSIONS; HYBRID ALGORITHMS; INTERIOR POINT; LARGE-SCALE LOGISTICS; LINE SEARCHES; LINEAR RATE; LOGISTIC REGRESSIONS; LOSS OF ACCURACY; MACHINE-LEARNING; MATRIX VECTOR MULTIPLICATION; MEMORY EFFICIENT; NEURAL SIGNAL PROCESSING; NEWTON'S METHODS; NUMERICAL COMPARISON; OBJECTIVE FUNCTIONS; OPTIMIZATION TECHNIQUES; SECOND PHASE; UCI MACHINE LEARNING REPOSITORY;

BIOINFORMATICS; CLASSIFIERS; COMPUTER VISION; CONVERGENCE OF NUMERICAL METHODS; LOGISTICS; NEWTON-RAPHSON METHOD; OPTIMIZATION; REGRESSION ANALYSIS; SIGNAL PROCESSING; SUPERVISED LEARNING;

LEARNING ALGORITHMS;

EID: 77949503812     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (63)

1. C.M. Bishop. Pattern Recognition and Machine Learning. Springer, 2007.
2. S. Boyd, S.-J. Kim, L. Vandenberghe, and A. Hassibi. A tutorial on geometric programming. Optimization and Engineering, 8(1):67-127, 2007.
3. M. Burger, G. Gilboa, S. Osher, and J. Xu. Nonlinear inverse scale space methods. Communications in Mathematical Sciences, 4:179-212, 2006.
4. E.J. Candés and T. Tao. Near optimal signal recovery from random projections: Universal encoding strategies? IEEE Trans. Inform. Theory, 52(2):5406-5425, 2006.
5. E.J. Candés, J. Romberg, and T. Tao. Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inform. Theory, 52(2):489-509, 2006.
6. S. Chen, D.L. Donoho, and M.A. Saunders. Atomic decomposition by basis pursuit. SIAM J. Scientific Computing, 20:33-61, 1998.
7. J.F. Claerbout and F. Muir. Robust modeling with erratic data. Geophysics, 38(5):826-844, 1973.
8. A. d'Aspremont, L. El Ghaoui, M. Jordan, and G. Lanckriet. A direct formulation for sparse pca using semidefinite programming. In Advances in Neural Information Processing Systems, pages 41-48. MIT Press, 2005.
9. D.L. Donoho. Compressed sensing. IEEE Trans. Inform. Theory, 52:1289-1306, 2006.
10. D.L. Donoho and M. Elad. Optimally sparse representations in general nonorthogonal dictionaries by ℓ1 minimization. Proc. Nat'l Academy of Science, 100(5):2197-2202, 2003.
11. D.L. Donoho and X. Huo. Uncertainty principles and ideal atomic decomposition. IEEE. Trans. Inform. Theory, 48(9):2845-2862, 2001.
12. D.L. Donoho and B.F. Logan. Signal recovery and the large sieve. SIAM J. Appl. Math., 52(2): 577-591, 1992.
13. D.L. Donoho and P.B. Stark. Uncertainty principle and signal recovery. SIAM J. Appl. Math., 49(3): 906-931, 1989.
14. D.L. Donoho, I. Johnstone, G. Kerkyacharian, and D. Picard. Wavelet shrinkage: Asymptopia? J. Roy. Stat. Soc. B, 57(2):301-337, 1995.
15. B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani. Least angle regression. Annals of Statistics, 32(2):407-499, 2004.
16. S. Eyheramendy, A. Genkin, W. Ju, D. Lewis, and D. Madigan. Sparse bayesian classifiers for text categorization. Technical report, J. Intelligence Community Research and Development, 2003.
17. M. Figueiredo. Adaptive sparseness for supervised learning. IEEE Trans. Pattern Analysis and Machine Intelligence, 25:1150-1159, 2003.
18. M. Figueiredo and A. Jain. Bayesian learning of sparse classifiers. In IEEE Conf. Computer Vision and Pattern Recognition, pages 35-41, 2001.
19. M. Figueiredo, R. Nowak, and S.Wright. Gradient projection for sparse reconstruction: application to compressed sensing and other inverse problems. IEEE J. Selected Topics in Signal Processing: Special Issue on Convex Optimization Methods for Signal Processing, 1(4):586-598, 2007.
20. A. Genkin, D.D. Lewis, and D. Madigan. Large-scale bayesian logistic regression for text categorization. Technometrics, 49(3):291-304, 2007.
21. A.D. Gerson, L.C. Parra, and P. Sajda. Cortical origins of response time variability during rapid discrimination of visual objects. Neuroimage, 28(2):342-353, 2005.
22. A. Ghosh and S. Boyd. Growing well-connected graphs. In 45th IEEE Conference on Decision and Control, pages 6605-6611, 2006.
23. J. Goodman. Exponential priors for maximum entropy models. In Proceedings of the Annual Meetings of the Association for Computational Linguistics, 2004.
24. E. Hale, W. Yin, and Y. Zhang. Fixed-point continuation for ℓ1-minimization: methodology and convergence. SIAM J. Optimization, 19(3):1107-1130, 2008.
25. A. Hassibi, J. How, and S. Boyd. Low-authority controller design via convex optimization. AIAA Journal of Guidance, Control and Dynamics, 22(6):862-872, 1999.
26. K. Koh, S.-J. Kim, and S. Boyd. An interior-point method for large-scale ℓ1-regularized logistic regression. J. Machine Learning Research, 8:1519-1555, 2007.
27. B. Krishnapuram and A. Hartemink. Sparse multinomial logistic regression: fast algorithms and generalization bounds. IEEE Trans. Pattern Analysis and Machine Intelligence, 27(6):957-968, 2005.
28. B. Krishnapuram, L. Carin, and M. Figueiredo. Sparse multinomial logistic regression: fast algorithms and generalization bounds. IEEE Trans. Pattern Analysis and Machine Intelligence, 27(6): 957-968, 2005.
29. S. Lee, H. Lee, P. Abbeel, and A. Ng. Efficient ℓ1-regularized logistic regression. In 21th National Conference on Artificial Intelligence (AAAI), 2006.
30. D.D. Lewis, Y. Yang, T.G. Rose, and F. Li. Rcv1: A new benchmark collection for text categorization research. J. Machine Learning Research, 5:361-397, 2004.
31. J.G. Liao and K.V. Chin. Logistic regression for disease classification using microarray data: model selection in a large p and small n case. Bioinformatics, 23(15):1945-1951, 2007.
32. N. Littlestone. Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm. Machine Learning, 2:285-318, 1988.
33. M. Lobo, M. Fazel, and S. Boyd. Portfolio optimization with linear and fixed transaction costs. Annals of Operations Research, 152(1):376-394, 2007.
34. J. Lokhorst. The lasso and generalised linear models. Technical report, Honors Project, Department of Statistics, University of Adelaide, South Australia, Australia, 1999.
35. D. Madigan, A. Genkin, D. Lewis, and D Fradkin. Bayesian multinomial logistic regression for author identification. In Maxent Conference, pages 509-516, 2005.
36. B.K. Natarajan. Sparse approximate solutions to linear system. SIAMJ. Computing, 24(2):227-234, 1995.
37. A. Ng. Feature selection, ℓ1 vs ℓ2 regularization, and rotational invariance. In International Conference on Machine Learning (ICML), pages 78-85. ACM Press, New York, 2004.
38. A. Ng. On feature selection: Learning with exponentially many irrelevant features as training examples. In International Conference on Machine Learning (ICML), pages 404-412, 1998.
39. S. Osher, Y. Mao, B. Dong, and W. Yin. Fast linearized bregman iteration for compressive sensing and sparse denoising. Communications in Mathematical Sciences, 8(1):93-111, 2010.
40. M.Y. Park and T. Hastie. ℓ1 regularized path algorithm for generalized linear models. J. R. Statist. Soc. B, 69:659-677, 2007.
41. L.C. Parra, C. Spence, and P. Sajda. Higher-order statistical properties arising from the nonstationarity of natural signals. In Advances in Neural Information Processing Systems, volume 13, pages 786-792, 2001.
42. L.C. Parra, C.D. Spence, A.D. Gerson, and P. Sajda. Recipes for the linear analysis of EEG. Neuroimage, 28(2):326-341, 2005.
43. S. Perkins and J. Theiler. Online feature selection using grafting. In Proceedings of the Twenty-First International Conference on Machine Learning (ICML), pages 592-599. ACM Press, 2003.
44. M.G. Philiastides and P. Sajda. Temporal characterization of the neural correlates of perceptual decision making in the human brain. Cereb Cortex, 16(4):509-518, 2006.
45. B. Polyak. Introduction to Optimization. Optimization Software, 1987.
46. V. Roth. The generalized lasso. IEEE Tran. Neural Networks, 15(1):16-28, 2004.
47. L. Rudin, S. Osher, and E. Fatemi. Nonlinear total variation based noise removal algorithms. Physica D, 60(1-4):259-268, 1992.
48. M. Schmidt, G. Fung, and R. Rosales. Fast optimization methods for l1 regularization: a comparative study and two new approaches. In European Conference on Machine Learning (ECML), pages 286-297, 2007.
49. J. Shi and S. Osher. A nonlinear inverse scale space method for a convex multiplicative noise model. SIAM J. Imaging Sciences, 1(3):294-321, 2008.
50. J. Shi, J. Wielaard, R.T. Smith, and P. Sajda. Perceptual decision making investigated via sparse decoding of a spiking neuron model of V1. In 4th International IEEE/EMBS Conference on Neural Engineering, pages 558-561, 2009.
51. N.Z. Shor. Minimization Methods for Non-differentiable functions. Springer Series in Computational Mathematics. Springer, 1985.
52. H.L. Taylor, S.C. Banks, and J.F. McCoy. Deconvolution with the ℓ1 norm. Geophysics, 44(1): 39-52, 1979.
53. R. Tibshirani. Regression shrinkage and selection via the lasso. J. Roy. Stat. Soc. B, 58(1):267-288, 1996.
54. Y. Tsuruoka, J. McNaught, J. Tsujii, and S. Ananiadou. Learning string similarity measures for gene/protein name dictionary look-up using logistic regression. Bioinformatics, 23(20):2768-2774, 2007.
55. L. Vandenberghe, S. Boyd, and A. El Gamal. Optimal wire and transistor sizing for circuits with non-tree topology. In IEEE/ACM International Conference on Computer Aided Design, pages 252-259, 1997.
56. L. Vandenberghe, S. Boyd, and A. El Gamal. Optimizing dominant time constant in RC circuits. IEEE Trans. Computer-Aided Design, 17(2):110-125, 1998.
57. V. Vapnik. Estimation of Dependences Based on Empirical Data. Springer-Verlag, 1982.
58. V. Vapnik. Statistical Learning Theory. John Wiley & Sons, 1988.
59. Z. Wen, W. Yin, D. Goldfarb, and Y. Zhang. A fast algorithm for sparse reconstruction based on shrinkage, subspace optimization and continuation. Technical report, Rice University CAAM TR09-01, 2009.
60. W. Yin, S. Osher, J. Darbon, and D. Goldfarb. Bregman iterative algorithm for ℓ1-minimization with applications to compressed sensing. SIAM J. Imaging Science, 1(1):143-168, 2008.
61. P. Zhao and B. Yu. On model selection consistency of lasso. J. Machine Learning Research, 7: 2541-2567, 2007.
62. J. Zhu, S. Rosset, T. Hastie, and R. Tibshirani. 1-norm support vector machines. In Advances in Neural Information Processing Systems, volume 16, pages 49-56. MIT Press, 2004.
63. H. Zou, T. Hastie, and R. Tibshirani. Sparse principle component analysis. J. Computational and Graphical Statistics, 15(2):262-286, 2006.