Algorithms for nonnegative matrix and tensor factorizations: A unified view based on block coordinate descent framework

Journal of Global Optimization

Volumn 58, Issue 2, 2014, Pages 285-319

  Kim, Jingu ^a   He, Yunlong ^b   Park, Haesun ^c  

^a NOKIA RESEARCH CENTER   (United States)

^b GEORGIA INSTITUTE OF TECHNOLOGY   (United States)

^c Georgia Institute of Technology   (United States)

Author keywords

Block coordinate descent; Low rank approximation; Nonnegative matrix factorization; Nonnegative tensor factorization

Indexed keywords

EID: 84895057098     PISSN: 09255001     EISSN: 15732916     Source Type: Journal    
DOI: 10.1007/s10898-013-0035-4     Document Type: Article

References (88)

1. Acar, E.; Yener, B.: Unsupervised multiway data analysis: a literature survey. IEEE Trans. Knowl. Data Eng. 21(1), 6-20 (2009)
2. Bellavia, S.; Macconi, M.; Morini, B.: An interior point newton-like method for non-negative least-squares problems with degenerate solution. Numer. Linear Algebra Appl. 13(10), 825-846 (2006)
3. Berman, A.; Plemmons, R.J.: Nonnegative matrices in the mathematical sciences. Society for Industrial and Applied Mathematics, Philadelphia (1994)
4. Berry, M.; Browne, M.; Langville, A.; Pauca, V.; Plemmons, R.: Algorithms and applications for approximate nonnegative matrix factorization. Comput. Stat. Data Anal. 52(1), 155-173 (2007)
5. Bertsekas, D.P.: Nonlinear programming. Athena Scientific (1999)
6. Biggs, M.; Ghodsi, A.; Vavasis, S.: Nonnegative matrix factorization via rank-one downdate. In: Proceedings of the 25th International Conference on, Machine Learning, pp. 64-71 (2008)
7. Birgin, E.; Martínez, J.; Raydan, M.: Nonmonotone spectral projected gradient methods on convex sets. SIAM J. Optim. 10(4), 1196-1211 (2000)
8. Björck, A: Numerical Methods for Least Squares Problems. Society for Industrial and Applied Mathematics, Philadelphia (1996)
9. Bro, R.; De Jong, S.: A fast non-negativity-constrained least squares algorithm. J. Chemom. 11, 393-401 (1997)
10. Brunet, J.; Tamayo, P.; Golub, T.; Mesirov, J.: Metagenes and molecular pattern discovery using matrix factorization. Proc. Natal. Acad. Sci. 101(12), 4164-4169 (2004)
11. Bucak, S.; Gunsel, B.: Video content representation by incremental non-negative matrix factorization. In: Proceedings of the 2007 IEEE International Conference on Image Processing (ICIP), vol. 2, pp. II-113-II-116 (2007)
12. Cai, D.; He, X.; Han, J.; Huang, T.: Graph regularized nonnegative matrix factorization for data representation. IEEE Trans. Pattern Anal. Mach. Intell. 33(8), 1548-1560 (2011)
13. Cao, B.; Shen, D.; Sun, J.T.; Wang, X.; Yang, Q.; Chen, Z.: Detect and track latent factors with online nonnegative matrix factorization. In: Proceedings of the 20th International Joint Conference on Artifical, Intelligence, pp. 2689-2694 (2007)
14. Carroll, J.D.; Chang, J.J.: Analysis of individual differences in multidimensional scaling via an n-way generalization of "eckart-young" decomposition. Psychometrika 35(3), 283-319 (1970)
15. Chen, D.; Plemmons, R.J.: Nonnegativity constraints in numerical analysis. In: Proceedings of the Symposium on the Birth of Numerical Analysis, Leuven Belgium, pp. 109-140 (2009)
16. Chen, S.S.; Donoho, D.L.; Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM Rev. 43(1), 129-159 (2001)
17. Chu, M.; Plemmons, R.: Nonnegative matrix factorization and applications. IMAGE: Bull. Int. Linear Algebra Soc. 34, 2-7 (2005)
18. Chu, M.T.; Lin, M.M.: Low-dimensional polytope approximation and its applications to nonnegative matrix factorization. SIAM J. Sci. Comput. 30(3), 1131-1155 (2008)
19. Cichocki, A.; Phan, A.H.: Fast local algorithms for large scale nonnegative matrix and tensor factorizations. IEICE Trans. Fundam. Electron. Commun. Comput. Sci. E92-A(3), 708-721 (2009)
20. Cichocki, A.; Zdunek, R.; Amari, S.I.: Hierarchical ALS algorithms for nonnegative matrix and 3d tensor factorization. In: Lecture Notes in Computer Science, vol. 4666, pp. 169-176. Springer (2007)
21. Cichocki, A.; Zdunek, R.; Choi, S.; Plemmons, R.; Amari, S.-I.: Nonnegative tensor factorization using alpha and beta divergencies. In: Proceedings of the 32nd International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Honolulu, April 2007, vol. 3, pp. III-1393-III-1396 (2007)
22. Cichocki, A.; Zdunek, R.; Phan, A.H.; Amari, S.I.: Nonnegative Matrix and Tensor Factorizations: Applications to Exploratory Multi-Way Data Analysis and Blind Source Separation. Wiley, West Sussex (2009)
23. Devarajan, K.: Nonnegative matrix factorization: an analytical and interpretive tool in computational biology. PLoS Comput. Biol. 4(7), e1000,029 (2008)
24. Dhillon, I.; Sra, S.: Generalized nonnegative matrix approximations with bregman divergences. In: Advances in Neural Information Processing Systems 18, pp. 283-290. MIT Press (2006)
25. Ding, C.; Li, T.; Jordan, M.: Convex and semi-nonnegative matrix factorizations. IEEE Trans. Pattern Anal. Mach. Intell. 32(1), 45-559 (2010)
26. Ding, C.; Li, T.; Peng, W.; Park, H.: Orthogonal nonnegative matrix tri-factorizations for clustering. In: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 126-135 (2006)
27. Donoho, D.; Stodden, V.: When does non-negative matrix factorization give a correct decomposition into parts? In: Advances in Neural Information Processing Systems 16. MIT Press (2004)
28. Drake, B.; Kim, J.; Mallick, M.; Park, H.: Supervised Raman spectra estimation based on nonnegative rank deficient least squares. In: Proceedings of the 13th International Conference on Information Fusion, Edinburgh, UK (2010)
29. Févotte, C.; Bertin, N.; Durrieu, J.: Nonnegative matrix factorization with the Itakura-Saito divergence: With application to music analysis. Neural Comput. 21(3), 793-830 (2009)
30. Figueiredo, M.A.T.; Nowak, R.D.; Wright, S.J.: Gradient projection for sparse reconstruction: application to compressed sensing and other inverse problems. IEEE J. Sel. Top. Signal Process. 1(4), 586-597 (2007)
31. Franc, V.; Hlavac, V.; Navara, M.: Sequential coordinate-wise algorithm for the non-negative least squares problem. In: Proceedings of the 11th International Conference on Computer Analysis of Images and Patterns, pp. 407-414 (2005)
32. Friedlander, M.P.; Hatz, K.: Computing nonnegative tensor factorizations. Comput. Optim. Appl. 23(4), 631-647 (2008)
33. Frigyesi, A.; Höglund, M.: Non-negative matrix factorization for the analysis of complex gene expression data: identification of clinically relevant tumor subtypes. Cancer Inform. 6, 275-292 (2008)
34. Gillis, N.: Nonnegative matrix factorization complexity, algorithms and applications. Ph.D. thesis, Université catholique de Louvain (2011)
35. Gillis, N.; Glineur, F.: Nonnegative factorization and the maximum edge biclique problem. CORE Discussion Paper 2008/64, Universite catholique de Louvain (2008)
36. Gillis, N.; Glineur, F.: Using underapproximations for sparse nonnegative matrix factorization. Pattern Recognit. 43(4), 1676-1687 (2010)
37. Gillis, N.; Glineur, F.: Accelerated multiplicative updates and hierarchical als algorithms for nonnegative matrix factorization. Neural Comput. 24(4), 1085-1105 (2012)
38. Gillis, N.; Glineur, F.: A multilevel approach for nonnegative matrix factorization. J. Comput. Appl. Math. 236, 1708-1723 (2012)
39. Golub, G.; Van Loan, C.: Matrix Computations. Johns Hopkins University Press, Baltimore (1996)
40. Gonzalez, E.F.; Zhang, Y.: Accelerating the lee-seung algorithm for non-negative matrix factorization. Department of Computational and Applied Mathematics, Rice University, Technical report (2005)
41. Grippo, L.; Sciandrone, M.: On the convergence of the block nonlinear gauss-seidel method under convex constraints. Oper. Res. Lett. 26(3), 127-136 (2000)
42. Han, L.; Neumann, M.; Prasad, U.: Alternating projected Barzilai-Borwein methods for nonnegative matrix factorization. Electron. Trans. Numer. Anal. 36, 54-82 (2009)
43. Harshman, R.A.: Foundations of the parafac procedure: models and conditions for an "explanatory" multi-modal factor analysis. In: UCLA Working Papers in Phonetics, vol. 16, pp. 1-84 (1970)
44. Ho, N.D.: Nonnegative matrix factorization algorithms and applications. Ph.D. thesis, Univ. Catholique de Louvain (2008)
45. Horn, R.A.; Johnson, C.R.: Matrix Analysis. Cambridge University Press, Cambridge (1990)
46. Horst, R.; Pardalos, P.; Van Thoai, N.: Introduction to Global Optimization. Kluwer, Berlin (2000)
47. Hoyer, P.O.: Non-negative matrix factorization with sparseness constraints. J. Mach. Learn. Res. 5, 1457-1469 (2004)
48. Hsieh, C.J.; Dhillon, I.S.: Fast coordinate descent methods with variable selection for non-negative matrix factorization. In: Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1064-1072 (2011)
49. Hutchins, L.; Murphy, S.; Singh, P.; Graber, J.: Position-dependent motif characterization using non-negative matrix factorization. Bioinformatics 24(23), 2684-2690 (2008)
50. Júdice, J.J.; Pires, F.M.: A block principal pivoting algorithm for large-scale strictly monotone linear complementarity problems. Comput. Oper. Res. 21(5), 587-596 (1994)
51. Kim, D.; Sra, S.; Dhillon, I.S.: Fast Newton-type methods for the least squares nonnegative matrix approximation problem. In: Proceedings of the 2007 SIAM International Conference on Data Mining, pp. 343-354 (2007)
52. Kim, H.; Park, H.: Sparse non-negative matrix factorizations via alternating non-negativity-constrained least squares for microarray data analysis. Bioinformatics 23(12), 1495-1502 (2007)
53. Kim, H.; Park, H.: Nonnegative matrix factorization based on alternating nonnegativity constrained least squares and active set method. SIAM J. Matrix Anal. Appl. 30(2), 713-730 (2008)
54. Kim, H.; Park, H.; Eldén, L.: Non-negative tensor factorization based on alternating large-scale non-negativity-constrained least squares. In: Proceedings of IEEE 7th International Conference on Bioinformatics and, Bioengineering (BIBE07), vol. 2, pp. 1147-1151 (2007)
55. Kim, J.: Nonnegative Matrix and Tensor Factorizations, Least Squares Problems, and Applications. Ph.D. Thesis, Georgia Institute of Technology (2011)
56. Kim, J.; Monteiro, R.D.; Park, H.: Group Sparsity in Nonnegative Matrix Factorization. In: Proceedings of the 2012 SIAM International Conference on Data Mining, pp 851-862 (2012)
57. Kim, J.; Park, H.: Sparse nonnegative matrix factorization for clustering. Technical report, Georgia Institute of Technology GT-CSE-08-01 (2008)
58. Kim, J.; Park, H.: Toward faster nonnegative matrix factorization: a new algorithm and comparisons. In: Proceedings of the 8th IEEE International Conference on Data Mining (ICDM), pp. 353-362 (2008)
59. Kim, J.; Park, H.: Fast nonnegative matrix factorization: an active-set-like method and comparisons. SIAM J. Sci. Comput. 33(6), 3261-3281 (2011)
60. Kim, J.; Park, H.: Fast nonnegative tensor factorization with an active-set-like method. In: High-Performance Scientific Computing: Algorithms and Applications, pp. 311-326. Springer (2012)
61. Kolda, T.G.; Bader, B.W.: Tensor decompositions and applications. SIAM Rev. 51(3), 455-500 (2009)
62. Korattikara, A.; Boyles, L.; Welling, M.; Kim, J.; Park, H.: Statistical optimization of non-negative matrix factorization. In: Proceedings of the 14th International Conference on Artificial Intelligence and Statistics (AISTATS), JMLR: W &CP, vol. 15, pp. 128-136 (2011)
63. Kuang, D.; Ding, C.; Park, H.: Symmetric nonnegative matrix factorization for graph clustering. In: Proceedings of 2012 SIAM International Conference on Data Mining, pp. 106-117 (2012)
64. Lawson, C.L.; Hanson, R.J.: Solving Least Squares Problems. Prentice Hall, New Jersey (1974)
65. LeCun, Y.; Bottou, L.; Bengio, Y.; Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278-2324 (1998)
66. Lee, D.D.; Seung, H.S.: Learning the parts of objects by non-negative matrix factorization. Nature 401(6755), 788-791 (1999)
67. Lee, D.D.; Seung, H.S.: Algorithms for non-negative matrix factorization. In: Advances in Neural Information Processing Systems 13, pp. 556-562. MIT Press (2001)
68. Li, L.; Lebanon, G.; Park, H.: Fast bregman divergence nmf using taylor expansion and coordinate descent. In: Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 307-315 (2012)
69. Li, S.Z.; Hou, X.; Zhang, H.; Cheng, Q.: Learning spatially localized, parts-based representation. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and, Pattern Recognition, vol. 1, pp. I-207-I-212 (2001)
70. Lin, C.: On the convergence of multiplicative update algorithms for nonnegative matrix factorization. IEEE Trans. Neural Netw. 18(6), 1589-1596 (2007)
71. Lin, C.J.: Projected gradient methods for nonnegative matrix factorization. Neural Comput. 19(10), 2756-2779 (2007)
72. Lin, M.M.; Chu, M.T.: On the nonnegative rank of euclidean distance matrices. Linear Algebra Appl. 433(3), 681-689 (2010)
73. Merritt, M.; Zhang, Y.: Interior-point gradient method for large-scale totally nonnegative least squares problems. J. optim. Theory Appl. 126(1), 191-202 (2005)
74. Paatero, P.; Tapper, U.: Positive matrix factorization: a non-negative factor model with optimal utilization of error estimates of data values. Environmetrics 5(1), 111-126 (1994)
75. Park, H.; Kim, H.: One-sided non-negative matrix factorization and non-negative centroid dimension reduction for text classification. In: Proceedings of the 2006 Text Mining Workshop in the Tenth SIAM International Conference on Data Mining (2006)
76. Pauca, V.P.; Piper, J.; Plemmons, R.J.: Nonnegative matrix factorization for spectral data analysis. Linear Algebra Appl. 416(1), 29-47 (2006)
77. Pauca, V.P.; Shahnaz, F.; Berry, M.W.; Plemmons, R.J.: Text mining using non-negative matrix factorizations. In: Proceedings of the 2004 SIAM International Conference on Data Mining, pp. 452-456 (2004)
78. Schmidt, M.N.; Winther, O.; Hansen, L.K.: Bayesian non-negative matrix factorization. In: Proceedings of the 2009 International Conference on Independent Component Analysis and Signal Separation, Lecture Notes in Computer Science (LNCS), vol. 5441, pp. 540-547. Springer (2009)
79. Sra, S.: Block-iterative algorithms for non-negative matrix approximation. In: Proceedings of the 8th IEEE International Conference on Data Mining, pp. 1037-1042 (2008)
80. Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B (Methodological) 58(1), 267-288 (1996)
81. Van Benthem, M.H.; Keenan, M.R.: Fast algorithm for the solution of large-scale non-negativity-constrained least squares problems. J. Chemom. 18, 441-450 (2004)
82. Vavasis, S.A.: On the complexity of nonnegative matrix factorization. SIAM J. Optim. 20(3), 1364-1377 (2009)
83. Weinberger, K.; Saul, L.: Distance metric learning for large margin nearest neighbor classification. J. Mach. Learn. Res. 10, 207-244 (2009)
84. Welling, M.; Weber, M.: Positive tensor factorization. Pattern Recogn. Lett. 22(12), 1255-1261 (2001)
85. Xu, W.; Liu, X.; Gong, Y.: Document clustering based on non-negative matrix factorization. In: Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Informaion Retrieval, pp. 267-273 (2003)
86. Zdunek, R.; Cichocki, A.: Non-negative matrix factorization with quasi-newton optimization. In: Proceedings of the Eighth International Conference on Artificial Intelligence and, Soft Computing, pp. 870-879 (2006)
87. Zdunek, R.; Cichocki, A.: Fast nonnegative matrix factorization algorithms using projected gradient approaches for large-scale problems. Comput. Intell. Neurosci. 2008, 939567 (2008)
88. Zhong, M.; Girolami, M.: Reversible jump MCMC for non-negative matrix factorization. In: Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics (AISTATS), JMLR: W &CP, vol. 5, pp. 663-670 (2009)