Iterative Hessian sketch: Fast and accurate solution approximation for constrained least-squares

Journal of Machine Learning Research

Volumn 17, Issue , 2016, Pages

  Pilanci, Mert ^a   Wainwright, Martin J ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Convex optimization; Information theory; Lasso; Low rank approximation; Random projection

Indexed keywords

APPROXIMATION THEORY; CONVEX OPTIMIZATION; INFORMATION THEORY; ITERATIVE METHODS; PROBLEM SOLVING;

CONSTRAINED LEAST SQUARES; LASSO; LEAST SQUARES PROBLEMS; LOW RANK APPROXIMATIONS; NUMBER OF ITERATIONS; QUADRATIC OBJECTIVE FUNCTIONS; RANDOM PROJECTIONS; STATISTICAL COMPLEXITY;

LEAST SQUARES APPROXIMATIONS;

EID: 84974641899     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (50)

1. N. Ailon and B. Chazelle. Approximate nearest neighbors and the fast Johnson-Lindenstrauss transform. In Proceedings of the 38th Annual ACM Symposium on Theory of Computing, pages 557-563. ACM, 2006.
2. Y. Amit, M. Fink, N. Srebro, and S. Ullman. Uncovering shared structures in multiclass classification. In Proceedings of the 24th International Conference on Machine Learning, ICML '07, pages 17-24, New York, NY, USA, 2007. ACM. ISBN 978-1-59593-793-3. doi: 10.1145/1273496.1273499. URL http://doi.acm.org/10.1145/1273496.1273499.
3. A. Argyriou, T. Evgeniou, and M. Pontil. Convex multi-task feature learning. Machine Learning, 73(3):243-272, 2008. ISSN 0885-6125. doi: 10.1007/s10994-007-5040-8. URL http://dx.doi.org/10.1007/s10994-007-5040-8.
4. H. Avron, P. Maymounkov, and S. Toledo. Blendenpik: Supercharging lapack's least-squares solver. SIAM Journal on Scientific Computing, 32(3):1217-1236, 2010.
5. F. Bach. Consistency of trace norm minimization. Journal of Machine Learning Research, 9: 1019-1048, June 2008.
6. F. Bach, R. Jenatton, J. Mairal, G. Obozinski, et al. Convex optimization with sparsity-inducing norms. Optimization for Machine Learning, pages 19-53, 2011.
7. P. L. Bartlett, O. Bousquet, and S. Mendelson. Local Rademacher complexities. Annals of Statistics, 33(4):1497-1537, 2005.
8. A. Beck and M. Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences, 2(1):183-202, 2009.
9. L. Birgé. Estimating a density under order restrictions: Non-asymptotic minimax risk. Annals of Statistics, 15(3):995-1012, March 1987.
10. J. Bourgain, S. Dirksen, and J. Nelson. Toward a unified theory of sparse dimensionality reduction in euclidean space. Geometric and Functional Analysis, 25(4), 2015.
11. C. Boutsidis and P. Drineas. Random projections for the nonnegative least-squares problem. Linear Algebra and its Applications, 431(5-7):760-771, 2009.
12. F. Bunea, Y. She, and M. Wegkamp. Optimal selection of reduced rank estimators of high-dimensional matrices. Annals of Statistics, 39(2):1282-1309, 2011.
13. E. J. Candes and T. Tao. Decoding by linear programming. IEEE Trans. Info Theory, 51 (12):4203-4215, December 2005.
14. S. Chen, D. L. Donoho, and M. A. Saunders. Atomic decomposition by basis pursuit. SIAM J. Sci. Computing, 20(1):33-61, 1998.
15. K. L. Clarkson, P. Drineas, M. Magdon-Ismail, M. W. Majoney, X. Meng, and D. P. Woodruff. The fast cauchy transform and faster robust linear regression. In Proceedings of the Twenty-Fourth Annual ACM-SIAM Symposium on Discrete Algorithms, pages 466-477. SIAM, 2013.
16. T.M. Cover and J.A. Thomas. Elements of Information Theory. John Wiley and Sons, New York, 1991.
17. P. Drineas and M. W. Mahoney. Effective resistances, statistical leverage, and applications to linear equation solving. arXiv preprint arXiv:1005.3097, 2010.
18. P. Drineas, M. W. Mahoney, S. Muthukrishnan, and T. Sarlos. Faster least squares approximation. Numer. Math, 117(2):219-249, 2011.
19. P. Drineas, M. Magdon-Ismail, M. W. Mahoney, and D. P. Woodruff. Fast approximation of matrix coherence and statistical leverage. Journal of Machine Learning Research, 13(1):3475-3506, 2012.
20. B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani. Least angle regression. Annals of Statistics, 32(2):407-499, 2004.
21. M. Fazel. Matrix Rank Minimization with Applications. PhD thesis, Stanford, 2002. Available online: http://faculty.washington.edu/mfazel/thesis-final.pdf.
22. L. El Ghaoui, V. Viallon, and T. Rabbani. Safe feature elimination for the lasso. Submitted, April, 2011.
23. Y. Gordon, A. E. Litvak, S. Mendelson, and A. Pajor. Gaussian averages of interpolated bodies and applications to approximate reconstruction. Journal of Approximation Theory, 149:59-73, 2007.
24. P. Huber. Robust regression: Asymptotics, conjectures and Monte Carlo. Annals of Statistics, 1:799-821, 2001.
25. D. M. Kane and J. Nelson. Sparser Johnson-Lindenstrauss transforms. Journal of the ACM, 61(1), 2014.
26. M. Ledoux and M. Talagrand. Probability in Banach Spaces: Isoperimetry and Processes. Springer-Verlag, New York, NY, 1991.
27. P. Loh and M. J. Wainwright. High-dimensional regression with noisy and missing data: Provable guarantees with non-convexity. Annals of Statistics, 40(3):1637-1664, September 2012.
28. M.J. Lyons, S. Akamatsu, M. Kamachi, and J. Gyoba. Coding facial expressions with gabor wavelets. In Proc. Int'l Conf. Automatic Face and Gesture Recognition, pages 200-205, 1998.
29. M. W. Mahoney. Randomized algorithms for matrices and data. Foundations and Trends in Machine Learning in Machine Learning, 3(2), 2011.
30. H. M. Markowitz. Portfolio Selection. Wiley, New York, 1959.
31. S. Negahban and M. J. Wainwright. Estimation of (near) low-rank matrices with noise and high-dimensional scaling. Annals of Statistics, 39(2):1069-1097, 2011.
32. S. Negahban and M. J. Wainwright. Restricted strong convexity and (weighted) matrix completion: Optimal bounds with noise. Journal of Machine Learning Research, 13:1665-1697, May 2012.
33. V. Ojansivu and J. Heikkil. Blur insensitive texture classification using local phase quantization. In Proc. Image and Signal Processing (ICISP 2008), pages 236-243, 2008.
34. M. R. Osborne, B. Presnell, and B. A. Turlach. On the Lasso and its dual. Journal of Computational and Graphical Statistics, 9(2):319-337, 2000.
35. M. Pilanci and M. J. Wainwright. Randomized sketches of convex programs with sharp guarantees. IEEE Trans. Info. Theory, 9(61):5096-5115, September 2015a.
36. M. Pilanci and M. J. Wainwright. Newton sketch: A linear-time optimization algorithm with linear-quadratic convergence. Technical report, UC Berkeley, 2015b. URL http://arxiv.org/pdf/1505.02250.pdf.
37. M. Pilanci, L. El Ghaoui, and V. Chandrasekaran. Recovery of sparse probability measures via convex programming. In Advances in Neural Information Processing Systems, pages 2420-2428, 2012.
38. q-balls. IEEE Trans. Information Theory, 57(10):6976-6994, October 2011.
39. B. Recht, M. Fazel, and P. Parrilo. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Review, 52(3):471-501, 2010.
40. V. Rokhlin and M. Tygert. A fast randomized algorithm for overdetermined linear least-squares regression. Proceedings of the National Academy of Sciences, 105(36):13212-13217, 2008.
41. T. Sarlos. Improved approximation algorithms for large matrices via random projections. In Foundations of Computer Science, 2006. FOCS'06. 47th Annual IEEE Symposium on, pages 143-152. IEEE, 2006.
42. N. Srebro, N. Alon, and T. S. Jaakkola. Generalization error bounds for collaborative prediction with low-rank matrices. In Neural Information Processing Systems (NIPS), Vancouver, Canada, December 2005.
43. R. Tibshirani. Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society, Series B, 58(1):267-288, 1996.
44. S. Vempala. The Random Projection Method. Discrete Mathematics and Theoretical Computer Science. American Mathematical Society, Providence, RI, 2004.
45. R. Vershynin. Introduction to the non-asymptotic analysis of random matrices. Compressed Sensing: Theory and Applications, 2012.
46. T. T. Wu and K. Lange. Coordinate descent algorithms for Lasso penalized regression. Annals of Applied Statistics, 2(1):224-244, 2008.
47. Y. Yang, M. Pilanci, and M. J. Wainwright. Randomized sketches for kernels: Fast and optimal non-parametric regression. Technical report, UC Berkeley, 2015. URL http://arxiv.org/pdf/1501.06195.pdf.
48. B. Yu. Assouad, Fano and Le Cam. In Festschrift for Lucien Le Cam, pages 423-435. Springer-Verlag, Berlin, 1997.
49. M. Yuan and Y. Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society B, 1(68):49, 2006.
50. M. Yuan, A. Ekici, Z. Lu, and R. Monteiro. Dimension reduction and coefficient estimation in multivariate linear regression. Journal Of The Royal Statistical Society Series B, 69(3): 329-346, 2007.