A statistical perspective on algorithmic leveraging

Journal of Machine Learning Research

Volumn 16, Issue , 2015, Pages 861-911

  Ping, Ma ^a   Mahoney, Michael W ^b   Yu, Bin ^b  

^a UNIVERSITY OF GEORGIA   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Least squares; Leverage scores; Linear regression; Randomized algorithm; Subsampling

Indexed keywords

ALGORITHMS; APPROXIMATION ALGORITHMS; COMPUTATION THEORY; COMPUTATIONAL EFFICIENCY; IMPORTANCE SAMPLING; LINEAR REGRESSION; MATRIX ALGEBRA; NUMERICAL METHODS; REGRESSION ANALYSIS; SAMPLING; STATISTICS;

HIGH-QUALITY IMPLEMENTATION; LEAST ABSOLUTE DEVIATIONS; LEAST SQUARE; LEVERAGE SCORES; LINEAR REGRESSION MODELS; LOW-RANK MATRIX APPROXIMATIONS; RANDOMIZED ALGORITHMS; SUBSAMPLING;

LEAST SQUARES APPROXIMATIONS;

EID: 84938311437     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (46)

1. N. Ailon and B. Chazelle. Faster dimension reduction. Communications of the ACM, 53 (2):97-104, 2010.
2. T. W. Anderson and J. B. Taylor. Strong consistency of least squares estimates in normal linear regression. Annals of Statistics, 4(4):788-790, 1976.
3. H. Avron, P. Maymounkov, and S. Toledo. Blendenpik: Supercharging LAPACK's least-squares solver. SIAM Journal on Scientific Computing, 32:1217-1236, 2010.
4. P. J. Bickel, F. Gotze, and W. R. van Zwet. Resampling fewer than n observations: gains, losses, and remedies for losses. Statistica Sinica, 7:1-31, 1997.
5. S. Chatterjee and A. S. Hadi. Influential observations, high leverage points, and outliers in linear regression. Statistical Science, 1(3):379-393, 1986.
6. K. L. Clarkson and D. P. Woodruff. Low rank approximation and regression in input sparsity time. In Proceedings of the 45th Annual ACM Symposium on Theory of Computing, pages 81-90, 2013.
7. K. L. Clarkson, P. Drineas, M. Magdon-Ismail, M. W. Mahoney, X. Meng, and D. P. Woodruff. The Fast Cauchy Transform and faster robust linear regression. In Proceedings of the 24th Annual ACM-SIAM Symposium on Discrete Algorithms, pages 466-477, 2013.
8. N. Cloonan, A.R. Forrest, G. Kolle, B.B. Gardiner, G.J. Faulkner, M.K. Brown, D.F. Taylor, A.L. Steptoe, S. Wani, G. Bethel, A.J. Robertson, A.C. Perkins, S.J. Bruce, C.C. Lee, S.S. Ranade, H.E. Peckham, J.M. Manning, K.J. McKernan, and S.M. Grimmond. Stem cell transcriptome profiling via massive-scale mRNA sequencing. Nature Methods, 5(7): 613-619, 2008.
9. N. Cressie. Statistics for Spatial Data. Wiley, New York, 1991.
10. D. Dalpiaz, X. He, and P. Ma. Bias correction in RNA-Seq short-read counts using penalized regression. Statistics in Biosciences, 5(1):88-99, 2013.
11. P. Dhillon, Y. Lu, D. P. Foster, and L. Ungar. New subsampling algorithms for fast least squares regression. In Advances in Neural Information Processing Systems, pages 360-368, 2013.
12. P. Drineas, M. W. Mahoney, and S. Muthukrishnan. Sampling algorithms for ℓ2 regression and applications. In Proceedings of the 17th Annual ACM-SIAM Symposium on Discrete Algorithms, pages 1127-1136, 2006.
13. P. Drineas, M. W. Mahoney, S. Muthukrishnan, and T. Sarlós. Faster least squares approximation. Numerische Mathematik, 117(2):219-249, 2010.
14. P. Drineas, M. Magdon-Ismail, M. W. Mahoney, and D. P. Woodruff. Fast approximation of matrix coherence and statistical leverage. Journal of Machine Learning Research, 13: 3475-3506, 2012.
15. B. Efron. Bootstrap methods: another look at the jackknife. Annals of Statistics, 7(1): 1-26, 1979.
16. B. Efron and G. Gong. A leisurely look at the bootstrap, the jackknife, and cross-validation. American Statistician, 37(1):36-48, 1983.
17. W. Fu and K. Knight. Asymptotics for lasso-type estimators. Annals of Statistics, 28: 1356-1378, 2000.
18. A. Gittens and M. W. Mahoney. Revisiting the Nyström method for improved large-scale machine learning. Technical report, 2013. Preprint: arXiv:1303.1849.
19. G. H. Golub and C. F. Van Loan. Matrix Computations. Johns Hopkins University Press, Baltimore, 1996.
20. D. A. Harville. Matrix Algebra from a Statistician's Perspective. Springer-Verlag, New York, 1997.
21. D. V. Hinkley. Jackknifing in unbalanced situations. Technometrics, 19(3):285-292, 1977.
22. D. C. Hoaglin and R. E. Welsch. The hat matrix in regression and ANOVA. American Statistician, 32(1):17-22, 1978.
23. D. Hsu, S. M. Kakade, and T. Zhang. Random design analysis of ridge regression. Foundations of Computational Mathematics, 14(3):569-600, 2014.
24. L. Jaeckel. The infinitesimal jackknife. Bell Laboratories Memorandum, MM:72-1215-11, 1972.
25. A. Kleiner, A. Talwalkar, P. Sarkar, and M. Jordan. The big data bootstrap. In Proceedings of the 29th International Conference on Machine Learning, 2012.
26. T. L. Lai, H. Robbins, and C. Z. Wei. Strong consistency of least squares estimates in multiple regression. Proceedings of National Academy of Sciences, 75(7):3034-3036, 1978.
27. J. Li, H. Jiang, and W. H. Wong. Modeling non-uniformity in short-read rates in RNA-seq data. Genome Biology, 11:R50, 2010.
28. J. S. Liu, R. Chen, and W. H. Wong. Rejection control and sequential importance sampling. Journal of the American Statistical Association, 93(443):1022-1031, 1998.
29. M. W. Mahoney. Randomized Algorithms for Matrices and Data. Foundations and Trends in Machine Learning. NOW Publishers, Boston, 2011. Also available at: arXiv:1104.5557.
30. M. W. Mahoney and P. Drineas. CUR matrix decompositions for improved data analysis. Proceedings of National Academy of Sciences, 106:697-702, 2009.
31. X. Meng and M. W. Mahoney. Low-distortion subspace embeddings in input-sparsity time and applications to robust linear regression. In Proceedings of the 45th Annual ACM Symposium on Theory of Computing, pages 91-100, 2013.
32. X. Meng, M. A. Saunders, and M. W. Mahoney. LSRN: A parallel iterative solver for strongly over- or under-determined systems. SIAM Journal on Scientific Computing, 36 (2):C95-C118, 2014.
33. R. G. Miller. An unbalanced jackknife. Annals of Statistics, 2(5):880-891, 1974a.
34. R. G. Miller. The jackknife-a review. Biometrika, 61(1):1-15, 1974b.
35. T. Nielsen, R. B. West, S. C. Linn, O. Alter, M. A. Knowling, J. O'Connell, S. Zhu, M. Fero, G. Sherlock, J. R. Pollack, P. O. Brown, D. Botstein, and M. van de Rijn. Molecular characterisation of soft tissue tumours: a gene expression study. Lancet, 359(9314):1301-1307, 2002.
36. P. Paschou, E. Ziv, E. G. Burchard, S. Choudhry, W. Rodriguez-Cintron, M. W. Mahoney, and P. Drineas. PCA-correlated SNPs for structure identification in worldwide human populations. PLoS Genetics, 3:1672-1686, 2007.
37. P. Paschou, J. Lewis, A. Javed, and P. Drineas. Ancestry informative markers for fine-scale individual assignment to worldwide populations. Journal of Medical Genetics, page doi:10.1136/jmg.2010.078212, 2010.
38. D. N. Politis, J. P. Romano, and M. Wolf. Subsampling. Springer-Verlag, New York, 1999.
39. D. B. Rubin. The Bayesian bootstrap. Annals of Statistics, 9(1):130-134, 1981.
40. R. Serfling. Asymptotic relative efficiency in estimation. In Miodrag Lovric, editor, International Encyclopedia of Statistical Sciences, pages 68-72. Springer, 2010.
41. J. Shao. On Resampling Methods for Variance Estimation and Related Topics. PhD thesis, University of Wisconsin at Madison, 1987.
42. J. Shao and D. Tu. The Jackknife and Bootstrap. Springer-Verlag, New York, 1995.
43. P. F. Velleman and R. E. Welsch. Efficient computing of regression diagnostics. American Statistician, 35(4):234-242, 1981.
44. S. Weisberg. Applied Linear Regression. Wiley, New York, 2005.
45. C. F. J. Wu. Jackknife, bootstrap and other resampling methods in regression analysis. Annals of Statistics, 14(4):1261-1295, 1986.
46. J. Yang, X. Meng, and M. W. Mahoney. Quantile regression for large-scale applications. Technical report, 2013. Preprint: arXiv:1305.0087.