Modeling and optimization for big data analytics: (Statistical) learning tools for our era of data deluge

IEEE Signal Processing Magazine

Volumn 31, Issue 5, 2014, Pages 18-31

  Slavakis, Konstantinos ^a   Giannakis, Georgios B ^b   Mateos, Gonzalo ^c  

^a UNIVERSITY OF MINNESOTA   (United States)

^b Digital Technology Center   (United States)

^c Carnegie Mellon University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CLUSTERING ALGORITHMS; DATA VISUALIZATION; DIGITAL STORAGE; MEDICAL IMAGING; OPTIMIZATION; PRINCIPAL COMPONENT ANALYSIS; SIGNAL PROCESSING; TENSORS;

HIGH DIMENSIONAL DATA; KERNEL-BASED LEARNING; MODELING AND OPTIMIZATION; NETWORK VISUALIZATION; OPTIMIZATION ALGORITHMS; SCALABLE ARCHITECTURES; SCIENCE AND ENGINEERING; STATISTICAL LEARNING;

BIG DATA;

EID: 85032752433     PISSN: 10535888     EISSN: None     Source Type: Journal    
DOI: 10.1109/MSP.2014.2327238     Document Type: Article

References (80)

1. E. Acar, D. M. Dunlavy, T. G. Kolda, and M. Mørup, "Scalable tensor factorizations for incomplete data," Chemomet. Intell. Lab. Syst., vol. 106, no. 1, pp. 41-56, 2011.
2. M. Aharon, M. Elad, and A. Bruckstein, "K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation," IEEE Trans. Signal Processing, vol. 54, no. 11, pp. 4311-4322, Nov. 2006. (Pubitemid 44637761)
3. B. Baingana and G. B. Giannakis, "Embedding graphs under centrality constraints for network visualization," submitted for publication. arXiv:1401.4408.
4. L. Balzano, R. Nowak, and B. Recht, "Online identification and tracking of subspaces from highly incomplete information," in Proc. Allerton Conf. Communication, Control, and Computing, Monticello, IL, 2010, pp. 704-711.
5. H. H. Bauschke and P. L. Combettes, Convex Analysis and Monotone Operator Theory in Hilbert Spaces. New York: Springer, 2011.
6. J. A. Bazerque and G. B. Giannakis, "Nonparametric basis pursuit via sparse kernel-based learning," IEEE Signal Process. Mag., vol. 30, no. 4, pp. 112-125, July 2013.
7. J. A. Bazerque, G. Mateos, and G. B. Giannakis, "Rank regularization in Bayesian inference for tensor completion and extrapolation," IEEE Trans. Signal Processing, vol. 61, no. 22, pp. 5689-5703, Nov. 2013.
8. T. Bengtsson, P. Bickel, and B. Li, "Curse-of-dimensionality revisited: Collapse of the particle filter in very large scale systems," in Probability and Statistics: Essays in Honor of David A. Freedman. Beachwood, OH: IMS, 2008, vol. 2, pp. 316-334.
9. D. P. Bertsekas and J. N. Tsitsiklis, Parallel and Distributed Computation: Numerical Methods. Belmont, MA: Athena Scientific, 1999.
10. S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, "Distributed optimization and statistical learning via the alternating direction method of multipliers," Found. Trends Machine Learn., vol. 3, no. 1, pp. 1-122, 2011.
11. E. Candès and M. B. Wakin, "An introduction to compressive sampling," IEEE Signal Process. Mag., vol. 25, no. 2, pp. 21-30, 2008.
12. E. J. Candès, X. Li, Y. Ma, and J. Wright, "Robust principal component analysis?," J. ACM, vol. 58, no. 1, pp. 1-37, 2011.
13. E. J. Candes and Y. Plan, "Matrix completion with noise," Proc. IEEE, vol. 98, no. 6, pp. 925-936, June 2009.
14. V. Chandrasekaran, S. Sanghavi, P. R. Parrilo, and A. S. Willsky, "Ranksparsity incoherence for matrix decomposition," SIAM J. Optim., vol. 21, no. 2, pp. 572-596, 2011.
15. Q. Chenlu and N. Vaswani, "Recursive sparse recovery in large but correlated noise," in Proc. Allerton Conf. Communication, Control, and Computing, Sept. 2011, pp. 752-759.
16. Y. Chi, Y. C. Eldar, and R. Calderbank, "PETRELS: Parallel subspace estimation and tracking using recursive least squares from partial observations," IEEE Trans. Signal Processing, vol. 61, no. 23, pp. 5947-5959, 2013.
17. K. L. Clarkson and D. P. Woodruff, "Low rank approximation and regression in input sparsity time," in Proc. Symp. Theory Computing, June 1-4, 2013, pp. 81-90. arXiv:1207.6365v4.
18. K. Cukier. (2010). Data, data everywhere. The Economist. [Online]. Available: http://www.economist.com/node/15557443
19. J. Dean and S. Ghemawat, "MapReduce: Simplified data processing on large clusters," in Proc. Symp. Operating System Design and Implementation, San Francisco, CA, 2004, vol. 6, p. 10.
20. P. Drineas and M. W. Mahoney, "A randomized algorithm for a tensor-based generalization of the SVD," Linear Algeb. Appl., vol. 420, no. 2-3, pp. 553-571, 2007. (Pubitemid 44762244)
21. J. Feng, H. Xu, and S. Yan, "Online robust PCA via stochastic optimization," in Proc. Advances in Neural Information Processing Systems, Lake Tahoe, NV, Dec. 2013, pp. 404-412.
22. P. Forero, A. Cano, and G. B. Giannakis, "Consensus-based distributed support vector machines," J. Mach. Learn. Res., vol. 11, pp. 1663-1707, May 2010.
23. P. Forero, K. Rajawat, and G. B. Giannakis, "Prediction of partially observed dynamical processes over networks via dictionary learning," IEEE Trans. Signal Processing, to be published.
24. [Online]. Available: http://www.osirix-viewer.com/datasets/
25. H. Gao, J. Cai, Z. Shen, and H. Zhao, "Robust principal component analysisbased four-dimensional computed tomography," Phys. Med. Biol., vol. 56, no. 1, pp. 3181-3198, 2011.
26. G. B. Giannakis, V. Kekatos, N. Gatsis, S. J. Kim, H. Zhu, and B. Wollenberg, "Monitoring and optimization for power grids: A signal processing perspective," IEEE Signal Process. Mag., vol. 30, no. 5, pp. 107-128, Sept. 2013.
27. L. Harrison and A. Lu, "The future of security visualization: Lessons from network visualization," IEEE Netw., vol. 26, pp. 6-11, Dec. 2012.
28. T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed. New York: Springer, 2009.
29. J. He, L. Balzano, and A. Szlam, "Incremental gradient on the Grassmannian for online foreground and background separation in subsampled video," in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Providence, RI, June 2012, pp. 1568-1575.
30. [Online]. Available: http://www.internet2.edu/observatory/
31. M. I. Jordan, "On statistics, computation and scalability," Bernoulli, vol. 19, no. 4, pp. 1378-1390, 2013.
32. S. A. Kassam and H. V. Poor, "Robust techniques for signal processing: A survey," Proc. IEEE, vol. 73, no. 3, pp. 433-481, Mar. 1985. (Pubitemid 15110686)
33. S.-J. Kim and G. B. Giannakis, "Optimal resource allocation for MIMO ad hoc cognitive radio networks," IEEE Trans. Info. Theory, vol. 57, no. 5, pp. 3117-3131, May 2011.
34. A. Kleiner, A. Talwalkar, P. Sarkar, and M. I. Jordan, "A scalable bootstrap for massive data," J. Royal Statist. Soc.: Ser. B, to be published. [Online]. Available: http://dx.doi.org/10.1111/rssb.12050
35. E. D. Kolaczyk, Statistical Analysis of Network Data: Methods and Models. New York: Springer, 2009.
36. T. G. Kolda and B. W. Bader, "Tensor decompositions and applications," SIAM Rev., vol. 51, no. 3, pp. 455-500, 2009.
37. J. B. Kruskal, "Three-way arrays: Rank and uniqueness of trilinear decompositions, with applications to arithmetic complexity and statistics," Linear Algeb. Appl., vol. 18, no. 2, pp. 95-138, 1977.
38. A. Lakhina, M. Crovella, and C. Diot, "Diagnosing network-wide traffic anomalies," in Proc. SIGCOMM, Aug. 2004, pp. 201-206.
39. D. Lee and H. S. Seung, "Learning the parts of objects by non-negative matrix factorization," Nature, vol. 401, pp. 788-791, Oct. 1999.
40. J. Leskovec, J. Kleinberg, and C. Faloutsos, "Graph evolution: Densification and shrinking diameters," ACM Trans. Knowl. Discov. Data, vol. 1, no. 1, Mar. 2007.
41. J. Liu, P. Musialski, P. Wonka, and J. Ye, "Tensor completion for estimating missing values in visual data," IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, pp. 208-220, Jan. 2013.
42. Y. Low, J. Gonzalez, A. Kyrola, D. Bickson, C. Guestrin, and J. Hellerstein, "GraphLab: A new framework for parallel machine learning," in Proc. 26th Conf. Uncertainty in Artificial Intelligence, Catalina Island: CA, 2010.
43. Y. Ma, P. Niyogi, G. Sapiro, and R. Vidal, "Dimensionality reduction via subspace and submanifold learning [From the Guest Editors]," IEEE Signal Process. Mag., vol. 28, no. 2, pp. 14-126, Mar. 2011.
44. L. Mackey, A. Talwalkar, and M. I. Jordan, "Distributed matrix completion and robust factorization," submitted for publication. arXiv:1107.0789v7.
45. M. W. Mahoney, "Randomized algorithms for matrices and data," Found. Trends Machine Learn., vol. 3, no. 2, pp. 123-224, 2011.
46. J. Mairal, F. Bach, J. Ponce, and G. Sapiro, "Online learning for matrix factorization and sparse coding," J. Machine Learn. Res., vol. 11, pp. 19-60, Mar. 2010.
47. M. Mardani, G. Mateos, and G. B. Giannakis, "Decentralized sparsity-regularized rank minimization: Algorithms and applications," IEEE Trans. Signal Processing, vol. 61, no. 11, pp. 5374-5388, Nov. 2013.
48. M. Mardani, G. Mateos, and G. B. Giannakis, "Dynamic anomalography: Tracking network anomalies via sparsity and low rank," IEEE J. Sel. Topics Signal Process., vol. 8, pp. 50-66, Feb. 2013.
49. M. Mardani, G. Mateos, and G. B. Giannakis, "Recovery of low-rank plus compressed sparse matrices with application to unveiling traffic anomalies," IEEE Trans. Info. Theory, vol. 59, no. 8, pp. 5186-5205, Aug. 2013.
50. M. Mardani, G. Mateos, and G. B. Giannakis, "Subspace learning and imputation for streaming big data matrices and tensors," IEEE Trans. Signal Processing, submitted for publication.
51. G. Mateos, J. A. Bazerque, and G. B. Giannakis, "Distributed sparse linear regression," IEEE Trans. Signal Processing, vol. 58, no. 10, pp. 5262-5276, Oct. 2010.
52. G. Mateos and G. B. Giannakis, "Robust PCA as bilinear decomposition with outlier-sparsity regularization," IEEE Trans. Signal Processing, vol. 60, no. 10, pp. 5176-5190, Oct. 2012.
53. B. K. Natarajan, "Sparse approximate solutions to linear systems," SIAM J. Comput., vol. 24, no. 2, pp. 227-234, Apr. 1995.
54. Y. Nesterov, "A method for solving the convex programming problem with convergence rate O(1/k2), Dokl. Akad. Nauk SSSR, vol. 269, no. 3, pp. 543-547, 1983.
55. Office of Science and Technology Policy. (2012). Big data research and development initiative. Executive Office of the President. [Online]. Available: http://www.whitehouse.gov/sites/default/files/microsites/ostp/ big-data-press-release-final-2.pdf
56. B. A. Olshausen and D. J. Field, "Sparse coding with an overcomplete basis set: A strategy employed by V1?" Vision Res., vol. 37, no. 23, pp. 3311-3325, 1997. (Pubitemid 27493805)
57. E. E. Papalexakis, U. Kang, C. Faloutsos, N. D. Sidiropoulos, and A. Harpale, "Large scale tensor decompositions: Algorithmic developments and applications," IEEE Data Eng. Bull., vol. 36, no. 3, pp. 59-66, Sept. 2013.
58. H. Raja and W. U. Bajwa, "Cloud K-SVD: Computing data-adaptive representations in the cloud," in Proc. Allerton Conf. Communication, Control, and Computing, Oct. 2013, pp. 1474-1481.
59. M. Razaviyayn, M. Hong, and Z.-Q. Luo, "A unified convergence analysis of block successive minimization methods for nonsmooth optimization," SIAM J. Optim., vol. 23, no. 2, pp. 1126-1153, 2013.
60. H. Robbins and S. Monro, "A stochastic approximation method," Ann. Math. Statist., vol. 22, pp. 327-495, Sept. 1951.
61. L. K. Saul and S. T. Roweis, "Think globally, fit locally: Unsupervised learning of low dimensional manifolds," J. Mach. Learn. Res., vol. 4, pp. 119-155, Dec. 2003.
62. I. D. Schizas, A. Ribeiro, and G. B. Giannakis, "Consensus in ad hoc WSNs with noisy links-Part I: Distributed estimation of deterministic signals," IEEE Trans. Signal Processing, vol. 56, no. 1, pp. 350-364, Jan. 2008.
63. B. Schölkopf and A. J. Smola, Learning with Kernels. Cambridge, MA: MIT Press, 2001.
64. G. Scutari, F. Facchinei, P. Song, D. P. Palomar, and J.-S. Pang, "Decomposition by partial linearization: Parallel optimization of multi-agent systems," IEEE Trans. Signal Processing, vol. 62, no. 3, pp. 641-656.
65. S. Shalev-Shwartz, "Online learning and online convex optimization," Found. Trends Mach. Learn., vol. 4, no. 2, pp. 107-194, 2012.
66. M. Signoretto, R. V. Plas, B. D. Moor, and J. A. K. Suykens, "Tensor versus matrix completion: A comparison with application to spectral data," IEEE Signal Process. Lett., vol. 18, pp. 403-406, July 2011.
67. K. Skretting and K. Engan, "Recursive least squares dictionary learning algorithm," IEEE Trans. Signal Processing, vol. 58, no. 4, pp. 2121-2130, Apr. 2010.
68. K. Slavakis and G. B. Giannakis, "Online dictionary learning from big data using accelerated stochastic approximation algorithms," in Proc. ICASSP, Florence, Italy, 2014, pp. 16-20.
69. V. Solo and X. Kong, Adaptive Signal Processing Algorithms: Stability and Performance. Englewood Cliffs, NJ: Prentice Hall, 1995.
70. M. Soltanolkotabi and E. J. Candès, "A geometric analysis of subspace clustering with outliers," Ann. Statist., vol. 40, no. 4, pp. 2195-2238, Dec. 2012.
71. P. Sprechmann, A. M. Bronstein, and G. Sapiro, "Real-time online singing voice separation from monaural recordings using robust low-rank modeling," in Proc. Conf. Int. Society for Music Information Retrieval, Oct. 2012, pp. 67-72.
72. N. Srebro and A. Shraibman, "Rank, trace-norm and max-norm," in Learning Theory. Berlin/Heidelberg: Germany: Springer, 2005, pp. 545-560.
73. N. Städler, D. J. Stekhoven, and P. Bühlmann, "Pattern alternating maximization algorithm for missing data in large p small n problems," J. Mach. Learn. Res., to be published. arXiv:1005.0366v3.
74. J. M. F. ten Berge and N. D. Sidiropoulos, "On uniqueness in CANDECOMP/PARAFAC," Psychometrika, vol. 67, no. 3, pp. 399-409, 2002.
75. S. Theodoridis, K. Slavakis, and I. Yamada, "Adaptive learning in a world of projections: A unifying framework for linear and nonlinear classification and regression tasks," IEEE Signal Process. Mag., vol. 28, no. 1, pp. 97-123, Jan. 2011.
76. G. Tomasi and R. Bro, "PARAFAC and missing values," Chemom. Intell. Lab. Syst., vol. 75, no. 2, pp. 163-180, 2005. (Pubitemid 40164340)
77. P. Tseng, "Convergence of block coordinate decent method for nondifferentiable minimization," J. Optim. Theory Appl., vol. 109, pp. 475-494, June 2001.
78. R. Vidal, "Subspace clustering," IEEE Signal Process. Mag., vol. 28, no. 2, pp. 52-68, Mar. 2011.
79. B. Widrow and J. M. E. Hoff, "Adaptive switching circuits," IRE WESCON Conv. Rec., vol. 4, pp. 96-104, Aug. 1960.
80. M. Yamagishi and I. Yamada, "Over-relaxation of the fast iterative shrinkagethresholding algorithm with variable stepsize," Inverse Probl., vol. 27, no. 10, p. 105008, 2011.