Random projections for Bayesian regression

Statistics and Computing

Volumn 27, Issue 1, 2017, Pages 79-101

  Geppert, Leo N ^a   Ickstadt, Katja ^a   Munteanu, Alexander ^a   Quedenfeld, Jens ^a   Sohler, Christian ^a  

^a TU DORTMUND UNIVERSITY   (Germany)

Author keywords

Bayesian regression; Data reduction; Posterior approximation; Random projections

Indexed keywords

EID: 84947608290     PISSN: 09603174     EISSN: 15731375     Source Type: Journal    
DOI: 10.1007/s11222-015-9608-z     Document Type: Article

References (69)

1. Ailon, N., Liberty, E.: Fast dimension reduction using Rademacher series on dual BCH codes. Discret. Comput. Geom. 42(4), 615–630 (2009)
2. Alon, N., Matias, Y., Szegedy, M.: The space complexity of approximating the frequency moments. J. Comput. Syst. Sci. 58(1), 137–147 (1999)
3. Balakrishnan, S., Madigan, D.: A one-pass sequential Monte Carlo method for Bayesian analysis of massive datasets. Bayesian Anal. 1(2), 345–361 (2006)
4. Banerjee, A., Dunson, D.B., Tokdar, S.T.: Efficient Gaussian process regression for large datasets. Biometrika 100(1), 75–89 (2013)
5. Baraniuk, R., Davenport, M., Devore, R., Wakin, M.: A simple proof of the restricted isometry property for random matrices. Constr. Approx. 28(3), 253–263 (2008)
6. Bardenet, R., Doucet, A., Holmes, C.C.: Towards scaling up Markov Chain Monte Carlo an adaptive subsampling approach. In: Proc. of ICML, pp. 405–413 (2014)
7. Batson, J.D., Spielman, D.A., Srivastava, N.: Twice-Ramanujan sparsifiers. SIAM J. Comput. 41(6), 1704–1721 (2012)
8. Beaumont, M.A., Zhang, W., Balding, D.J.: Approximate Bayesian computation in population genetics. Genetics 162(4), 2025–2035 (2002)
9. Benson, A.R., Gleich, D.F., Demmel, J.: Direct QR factorizations for tall-and-skinny matrices in MapReduce architectures. In: Proceedings of IEEE International Conference on Big Data, pp. 264–272 (2013)
10. Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Berlin (2006)
11. Boutsidis, C., Gittens, A.: Improved matrix algorithms via the subsampled randomized hadamard transform. SIAM J. Matrix Anal. Appl. 34(3), 1301–1340 (2013)
12. Boutsidis, C., Magdon-Ismail, M.: Faster SVD-truncated regularized least-squares. In: IEEE symposium information theory, pp. 1321–1325 (2014)
13. Boutsidis, C., Zouzias, A., Drineas, P.: Random projections for k -means clustering. In: Proceedings of NIPS, pp. 298–306 (2010)
14. Boutsidis, C., Drineas, P., Magdon-Ismail, M.: Near-optimal coresets for least-squares regression. IEEE Trans. Inf. Theory 59(10), 6880–6892 (2013)
15. Candès, E.J., Romberg, J.K., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52(2), 489–509 (2006)
16. Clarkson, K.L., Woodruff, D.P.: Numerical linear algebra in the streaming model. In: Proceedings of STOC, pp. 205–214 (2009)
17. Clarkson, K.L., Woodruff, D.P.: Low rank approximation and regression in input sparsity time. In: Proceedings of STOC, pp. 81–90 (2013)
18. Clarkson, K.L., Woodruff, D.P.: Sketching for M-estimators: A unified approach to robust regression. In: Proceedings of SODA, pp. 921–939 (2015)
19. Clarkson, K.L., Drineas, P., Magdon-Ismail, M., Mahoney, M.W., Meng, X., Woodruff, D.P.: The fast Cauchy transform and faster robust linear regression. In: Proceedings of SODA, pp. 466–477 (2013)
20. Cohen, M.B., Elder, S., Musco, C., Musco, C., Persu, M.: Dimensionality reduction for k -means clustering and low rank approximation. In: Proceedings of STOC (2015)
21. Constantine, P.G., Gleich, D.F.: Tall and skinny QR factorizations in MapReduce architectures. In: Proceedings of International Workshop on MapReduce and Application, ACM, pp. 43–50 (2011)
22. Csillery, K., Blum, M., Gaggiotti, O., Francois, O.: Approximate Bayesian computation (ABC) in practice. Trends Ecol. Evol. 25(7), 410–418 (2010)
23. p regression. SIAM J. Comput. 38(5), 2060–2078 (2009)
24. Demmel, J., Grigori, L., Hoemmen, M., Langou, J.: Communication-optimal parallel and sequential QR and LU factorizations. SIAM J. Sci. Comput. 34(1), A206–A239 (2012)
25. Dietzfelbinger, M., Hagerup, T., Katajainen, J., Penttonen, M.: A reliable randomized algorithm for the closest-pair problem. J. Algorithms 25(1), 19–51 (1997)
26. Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory 52(2), 1289–1306 (2006)
27. 2 regression and applications. In: Proceedings of SODA, pp. 1127–1136 (2006)
28. Drineas, P., Mahoney, M.W., Muthukrishnan, S., Sarlós, T.: Faster least squares approximation. Numer. Math. 117(2), 219–249 (2011)
29. DuMouchel, W., Volinsky, C., Johnson, T., Cortes, C., Pregibon, D.: Squashing flat files flatter. In: Proceedings of KDD, pp. 6–15 (1999)
30. Fanaee-T, H., Gama, J.: Event labeling combining ensemble detectors and background knowledge. Prog. in AI 2(2–3), 113–127 (2014)
31. Feldman, D., Schmidt, M., Sohler, C.: Turning big data into tiny data: constant-size coresets for k-means, PCA and projective clustering. In: Proceedings of SODA, pp. 1434–1453 (2013)
32. Gelman, A.: Prior distributions for variance parameters in hierarchical models (comment on article by Browne and Draper). Bayesian Anal. 1(3), 515–534 (2006)
33. Gelman, A., Carlin, J.B., Stern, H.S., Dunson, D.B., Vehtari, A., Rubin, D.B.: Bayesian Data Analysis. Texts in Statistical Science, 3rd edn. Chapman & Hall/CRC, London (2014)
34. Geppert, L.N., Ickstadt, K., Munteanu, A., Quedenfeld, J., Sohler, C.: RaProR: Random Projections for Bayesian Linear Regression, R-package, Version 1.0 (2015) http://ls2-www.cs.uni-dortmund.de/projekte/RaProR/
35. Givens, C.R., Shortt, R.M.: A class of Wasserstein metrics for probability distributions. Mich. Math. J. 31(2), 231–240 (1984)
36. Golub, G.H.: Numerical methods for solving linear least squares problems. Numer. Math. 7(3), 206–216 (1965)
37. Golub, G.H., van Loan, C.F.: Matrix computations, 4th edn. Johns Hopkins University Press, Baltimore (2013)
38. Guhaniyogi, R., Dunson, D.B.: Bayesian compressed regression. J. Amer. Stat. Assoc. published online (2014) doi:10.1080/01621459.2014.969425
39. Halko, N., Martinsson, P.-G., Tropp, J.A.: Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions. SIAM Rev. 53(2), 217–288 (2011)
40. Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning: Data mining, Inference, and Prediction, 2nd edn. Springer, New York (2009)
41. Hoffman, M.D., Gelman, A.: The No-U-Turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. J. Mach. Learn. Res. 15, 1593–1623 (2014)
42. Horn, R., Johnson, C.: Matrix Analysis. Cambridge University Press, Cambridge (1990)
43. Ji, S., Carin, L.: Bayesian compressive sensing and projection optimization. In: Proceedings of ICML, pp. 377–384 (2007)
44. Jolliffe, I.: Principal Component Analysis, 2nd edn. Springer, Berlin (2002)
45. Kannan, R., Vempala, S.: Spectral algorithms. Found. Trends Theor. Comput. Sci. 4(3–4), 157–288 (2009)
46. Kannan, R., Vempala, S., Woodruff, D.P.: Principal component analysis and higher correlations for distributed data. In: Proceedings of COLT, pp. 1040–1057 (2014)
47. Kerber, M., Raghvendra, S.: Approximation and streaming algorithms for projective clustering via random projections. CoRR abs/1407.2063 (2014)
48. Lawson, C.L., Hanson, R.J.: Solving Least Squares Problems. Classics in Applied Mathematics. SIAM, Philadelphia, PA (1995)
49. Lichman, M.: UCI machine learning repository. http://archive.ics.uci.edu/ml (2013)
50. Ma, P., Mahoney, M.W., Yu, B.: A statistical perspective on algorithmic leveraging. In: Proceedings of ICML, pp. 91–99 (2014)
51. Madigan, D., Raghavan, N., DuMouchel, W., Nason, M., Posse, C., Ridgeway, G.: Likelihood-based data squashing: a modeling approach to instance construction. Data Min. Knowl. Discov. 6(2), 173–190 (2002)
52. Martins, T., Simpson, D., Lindgren, F., Rue, H.: Bayesian computing with INLA: new features. Comput. Stat. Data Anal. 67, 68–83 (2013)
53. Muthukrishnan, S.: Data streams: algorithms and applications. Found. Trends Theor. Comput. Sci. 1(2) 1–126 (2005)
54. Nelson, J., Nguyen, H.L.: OSNAP: Faster numerical linear algebra algorithms via sparser subspace embeddings. In: Proceedings of FOCS, pp. 117–126 (2013a)
55. Nelson, J., Nguyen, H.L.: Sparsity lower bounds for dimensionality reducing maps. In: Proceedings of STOC, pp. 101–110 (2013b)
56. Nelson, J., Nguyên, H.L.: Lower bounds for oblivious subspace embeddings. In: Proceedings of ICALP, Part I, pp. 883–894 (2014)
57. Paul, S., Boutsidis, C., Magdon-Ismail, M., Drineas, P.: Random projections for linear support vector machines. ACM Trans. Knowl. Discov. Data 8(4), 22 (2014)
58. Quiroz, M., Villani, M., Kohn, R.: Speeding up MCMC by efficient data subsampling. stat.ME abs/1404.4178 (2015)
59. R Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria (2014) http://www.R-project.org
60. Raskutti, G., Mahoney, M.: Statistical and algorithmic perspectives on randomized sketching for ordinary least-squares. In: Proceedings of ICML, pp. 617–625 (2015)
61. Rue, H., Martino, S., Chopin, N.: Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations (with discussion). J. R. Stat. Soc. B 71, 319–392 (2009)
62. Rusu, F., Dobra, A.: Pseudo-random number generation for sketch-based estimations. ACM Trans. Database Syst. 32(2), 11 (2007)
63. Sarlós, T.: Improved approximation algorithms for large matrices via random projections. In: Proceedings of FOCS, pp. 143–152 (2006)
64. Stan Development Team: Stan: A C++ Library for Probability and Sampling, Version 2.3. (2013) http://mc-stan.org/
65. Venkatasubramanian, S., Wang, Q.: The Johnson-Lindenstrauss transform: an empirical study. In: Proceedings of ALENEX, pp. 164–173 (2011)
66. Villani, C.: Optimal transport: Old and new. Grundlehren der mathematischen Wissenschaften. Springer, Berlin (2009)
67. Welling, M., Teh, Y.W., Andrieu, C., Kominiarczuk, J., Meeds, T., Shahbaba, B., Vollmer, S.: Bayesian inference & Big Data: a snapshot from a workshop. ISBA Bull. 21(4), 8–11 (2014)
68. p -regression using exponential random variables. In: Proceedings of COLT, pp. 546–567 (2013)
69. Yang, J., Meng, X., Mahoney, M.W.: Implementing randomized matrix algorithms in parallel and distributed environments. CoRR abs/1502.03032 (2015)