The Why and How of Nonnegative Matrix Factorization

Regularization, Optimization, Kernels, and Support Vector Machines

Volumn , Issue , 2014, Pages 257-291

  Gillis, Nicolas ^a  

^a UNIVERSITY OF MONS   (Belgium)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85142053618     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1201/b17558-15     Document Type: Chapter

References (113)

1. A. Ambikapathi, T.-H. Chan, C.-Y. Chi, and K. Keizer. Hyperspectral data geometry based estimation of number of endmembers using p-norm based pure pixel identification. IEEE Trans. on Geoscience and Remote Sensing, 51(5):2753–2769, 2013.
2. U.M.C. Araújo, B.T.C. Saldanha, R.K.H. Galvão, T. Yoneyama, H.C. Chame, and V. Visani. The successive projections algorithm for variable selection in spectroscopic multicomponent analysis. Chemometrics and Intelligent Laboratory Systems, 57(2):65–73, 2001.
3. S. Arora, R. Ge, Y. Halpern, D. Mimno, A. Moitra, D. Sontag, Y. Wu, and M. Zhu. A practical algorithm for topic modeling with provable guarantees. In Int. Conf. on Machine Learning (ICML ’13), volume 28, pages 280–288. 2013.
4. S. Arora, R. Ge, R. Kannan, and A. Moitra. Computing a nonnegative matrix factorization – provably. In Proc. of the 44th Symp. on Theory of Computing (STOC ’12), pages 145–162, 2012.
5. R. Badeau, N. Bertin, and E. Vincent. Stability analysis of multiplicative update algorithms and application to nonnegative matrix factorization. IEEE Trans. on Neural Networks, 21(12):1869–1881, 2010.
6. M.W. Berry, M. Browne, A. Langville, V.P. Pauca, and R.J. Plemmons. Algorithms and Applications for Approximate Nonnegative Matrix Factorization. Computational Statistics & Data Analysis, 52:155–173, 2007.
7. J.M. Bioucas-Dias and J.M.P. Nascimento. Estimation of signal subspace on hyperspectral data. In Remote Sensing, page 59820L. International Society for Optics and Photonics, 2005.
8. J.M. Bioucas-Dias, A. Plaza, N. Dobigeon, M. Parente, Q. Du, P. Gader, and J. Chanussot. Hyperspectral unmixing overview: Geometrical, statistical, and sparse regression-based approaches. IEEE J. of Selected Topics in Applied Earth Observations and Remote Sensing, 5(2):354–379, 2012.
9. V. Bittorf, B. Recht, E. Ré, and J.A. Tropp. Factoring nonnegative matrices with linear programs. In Advances in Neural Information Processing Systems (NIPS ’12), pages 1223–1231, 2012.
10. D.M. Blei. Probabilistic topic models. Communications of the ACM, 55(4):77–84, 2012.
11. C. Boutsidis and E. Gallopoulos. SVD based initialization: A head start for nonnegative matrix factorization. Pattern Recognition, 41:1350– 1362, 2008.
12. C. Boutsidis, M.W. Mahoney, and P. Drineas. An improved approximation algorithm for the column subset selection problem. In Proc. of the 20th Annual ACM-SIAM Symp. on Discrete Algorithms (SODA ’09), pages 968–977, 2009.
13. G. Braun, S. Fiorini, S. Pokutta, and D. Steurer. Approximation limits of linear programs (beyond hierarchies). In Proc. of the 53rd Annual IEEE Symp. on Foundations of Computer Science (FOCS’ 12), pages 480–489, 2012.
14. R. Bro. Multi-Way Analysis in the Food Industry: Models, Algorithms, and Applications. PhD thesis, University of Copenhagen, 1998. http://curis.ku.dk/ws/files/13035961/Rasmus_Bro.pdf.
15. P. Businger and G.H. Golub. Linear least squares solutions by householder transformations. Numerische Mathematik, 7:269–276, 1965.
16. D. Cai, X. He, J. Han, and T.S. Huang. Graph regularized nonnegative matrix factorization for data representation. IEEE Trans. on Pattern Analysis and Machine Intelligence, 33(8):1548–1560, 2011.
17. E. Carlini and F. Rapallo. Probability matrices, non-negative rank, and parameterization of mixture models. Linear Algebra and its Applications, 433:424–432, 2010.
18. G. Casalino, N. Del Buono, and C. Mencar. Subtractive clustering for seeding non-negative matrix factorizations. Information Sciences, (257):369–387, 2013.
19. A. Çivril and M. Magdon-Ismail. On selecting a maximum volume submatrix of a matrix and related problems. Theoretical Computer Science, 410(47-49):4801–4811, 2009.
20. A. Çivril and M. Magdon-Ismail. Exponential inapproximability of selecting a maximum volume sub-matrix. Algorithmica, 65(1):159–176, 2013.
21. T.-H. Chan, W.-K. Ma, A. Ambikapathi, and C.-Y. Chi. A simplex volume maximization framework for hyperspectral endmember extraction. IEEE Trans. on Geoscience and Remote Sensing, 49(11):4177– 4193, 2011.
22. T.-H. Chan, W.-K. Ma, C.-Y. Chi, and Y. Wang. A convex analysis framework for blind separation of non-negative sources. IEEE Trans. on Signal Processing, 56(10):5120–5134, 2008.
23. E.C. Chi and T.G. Kolda. On tensors, sparsity, and nonnegative factorizations. SIAM J. on Matrix Analysis and Applications, 33(4):1272– 1299, 2012.
24. S. Choi. Algorithms for orthogonal nonnegative matrix factorization. In Proc. of the Int. Joint Conf. on Neural Networks, pages 1828–1832, 2008.
25. A. Cichocki and A.H. Phan. Fast local algorithms for large scale Nonnegative Matrix and Tensor Factorizations. IEICE Trans. on Fundamentals of Electronics, Vol. E92-A No.3:708–721, 2009.
26. A. Cichocki, R. Zdunek, and S.-I. Amari. Non-negative Matrix Factorization with Quasi-Newton Optimization. In Lecture Notes in Artificial Intelligence, Springer, volume 4029, pages 870–879, 2006.
27. A. Cichocki, R. Zdunek, and S.-I. Amari. Hierarchical ALS Algorithms for Nonnegative Matrix and 3D Tensor Factorization. In Lecture Notes in Computer Science, Vol. 4666, Springer, pages 169–176, 2007.
28. A. Cichocki, R. Zdunek, A.H. Phan, and S.-I. Amari. Nonnegative Matrix and Tensor Factorizations: Applications to Exploratory Multi-way Data Analysis and Blind Source Separation. Wiley-Blackwell, 2009.
29. P. Comon. Independent component analysis: A new concept? Signal Processing, 36:287–314, 1994.
30. M. Conforti, G. Cornuéjols, and G. Zambelli. Extended formulations in combinatorial optimization. 4OR: A Quarterly Journal of Operations Research, 10(1):1–48, 2010.
31. G. Das and D.A. Joseph. The Complexity of Minimum Convex Nested Polyhedra. In Proc. of the 2nd Canadian Conf. on Computational Geometry, pages 296–301, 1990.
32. A. d’Aspremont, L. El Ghaoui, M.I. Jordan, and G.R.G. Lanckriet. A Direct Formulation for Sparse PCA Using Semidefinite Programming. SIAM Review, 49(3):434–448, 2007.
33. M.E. Daube-Witherspoon and G. Muehllehner. An iterative image space reconstruction algorithm suitable for volume ECT. IEEE Trans. on Medical Imaging, 5:61–66, 1986.
34. K. Devarajan. Nonnegative Matrix Factorization: An Analytical and Interpretive Tool in Computational Biology. PLoS Computational Biology, 4(7):e1000029, 2008.
35. C. Ding, X. He, and H.D. Simon. On the Equivalence of Nonnegative Matrix Factorization and Spectral Clustering. In SIAM Int. Conf. Data Mining (SDM’05), pages 606–610, 2005.
36. C. Ding, T. Li, and M.I. Jordan. Convex and semi-nonnegative matrix factorizations. IEEE Trans. on Pattern Analysis and Machine Intelligence, 32(1):45–55, 2010.
37. C. Ding, T. Li, and W. Peng. On the equivalence between non-negative matrix factorization and probabilistic latent semantic indexing. Computational Statistics & Data Analysis, 52(8):3913–3927, 2008.
38. C. Ding, T. Li, W. Peng, and H. Park. Orthogonal nonnegative matrix t-factorizations for clustering. In Proc. of the 12th ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, pages 126–135, 2006.
39. W. Ding, M.H. Rohban, P. Ishwar, and V. Saligrama. Topic discovery through data dependent and random projections. In Int. Conf. on Machine Learning (ICML ’13), volume 28, pages 471–479. 2013.
40. E. Elhamifar, G. Sapiro, and R. Vidal. See all by looking at a few: Sparse modeling for finding representative objects. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR ’12), 2012.
41. E. Esser, M. Moller, S. Osher, G. Sapiro, and J. Xin. A convex model for nonnegative matrix factorization and dimensionality reduction on physical space. IEEE Trans. on Image Processing, 21(7):3239–3252, 2012.
42. C. Févotte, N. Bertin, and J.-L. Durrieu. Nonnegative matrix factorization with the Itakura-Saito divergence: With application to music analysis. Neural Computation, 21(3):793–830, 2009.
43. S. Fiorini, V. Kaibel, K. Pashkovich, and D.O. Theis. Combinatorial bounds on nonnegative rank and extended formulations. Discrete Mathematics, 313(1):67–83, 2013.
44. X. Fu, W.-K. Ma, T.-H. Chan, J.M. Bioucas-Dias, and M.-D. Iordache. Greedy algorithms for pure pixels identification in hyperspectral unmixing: A multiple-measurement vector viewpoint. In Proc. of 21st European Signal Processing Conf. (EUSIPCO ’13), 2013.
45. E. Gaussier and C. Goutte. Relation between PLSA and NMF and implications. In Proc. of the 28th Annual Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 601–602, 2005.
46. R. Ge. Provable Algorithms for Machine Learning Problems. PhD thesis, Princeton University, 2013. http://dataspace.princeton.edu/jspui/bitstream/88435/dsp019k41zd62n/1/Ge_princeton_0181D_10819.pdf.
47. N. Gillis. Nonnegative Matrix Factorization: Complexity, Algorithms and Applications. PhD thesis, Université catholique de Louvain, 2011. https://sites.google.com/site/nicolasgillis/.
48. N. Gillis. Sparse and unique nonnegative matrix factorization through data preprocessing. Journal of Machine Learning Research, 13(Nov):3349–3386, 2012.
49. N. Gillis. Robustness analysis of Hottopixx, a linear programming model for factoring nonnegative matrices. SIAM J. on Matrix Analysis and Applications, 34(3):1189–1212, 2013.
50. N. Gillis. Successive nonnegative projection algorithm for robust nonnegative blind source separation. arXiv:1310.7529, 2013.
51. N. Gillis and F. Glineur. Using underapproximations for sparse nonnegative matrix factorization. Pattern Recognition, 43(4):1676–1687, 2010.
52. N. Gillis and F. Glineur. Accelerated multiplicative updates and hierarchical ALS algorithms for nonnegative matrix factorization. Neural Computation, 24(4):1085–1105, 2012.
53. N. Gillis and F. Glineur. On the geometric interpretation of the nonnegative rank. Linear Algebra and its Applications, 437(11):2685–2712, 2012.
54. N. Gillis and R. Luce. Robust near-separable nonnegative matrix factorization using linear optimization. Journal of Machine Learning Research, 2014. to appear.
55. N. Gillis and S.A. Vavasis. Fast and robust recursive algorithms for separable nonnegative matrix factorization. IEEE Trans. Pattern Anal. Mach. Intell., 2013. doi:10.1109/TPAMI.2013.226.
56. N. Gillis and S.A. Vavasis. Semidefinite programming based preconditioning for more robust near-separable nonnegative matrix factorization. arXiv:1310.2273, 2013.
57. G.H. Golub and C.F. Van Loan. Matrix Computation, 3rd Edition. The Johns Hopkins University Press Baltimore, 1996.
58. J. Gouveia, P.A. Parrilo, and R.R. Thomas. Approximate cone factorizations and lifts of polytopes. arXiv:1308.2162, 2013.
59. L. Grippo and M. Sciandrone. On the convergence of the block nonlinear Gauss-Seidel method under convex constraints. Operations Research Letters, 26:127–136, 2000.
60. N. Guan, D. Tao, Z. Luo, and B. Yuan. NeNMF: an optimal gradient method for nonnegative matrix factorization. IEEE Trans. on Signal Processing, 60(6):2882–2898, 2012.
61. D. Guillamet and J. Vitrià. Non-negative matrix factorization for face recognition. In Lecture Notes in Artificial Intelligence, pages 336–344. Springer, 2002.
62. J. Han, L. Han, M. Neumann, and U. Prasad. On the rate of convergence of the image space reconstruction algorithm. Operators and Matrices, 3(1):41–58, 2009.
63. N.-D. Ho. Nonnegative Matrix Factorization - Algorithms and Applications. PhD thesis, Université Catholique de Louvain, 2008.
64. P.O. Hoyer. Nonnegative matrix factorization with sparseness constraints. Journal of Machine Learning Research, 5:1457–1469, 2004.
65. C.-J. Hsieh and I.S. Dhillon. Fast coordinate descent methods with variable selection for non-negative matrix factorization. In Proc. of the 17th ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, pages 1064–1072, 2011
66. K. Huang, N.D. Sidiropoulos, and A. Swami. Non-negative matrix factorization revisited: Uniqueness and algorithm for symmetric decomposition. IEEE Trans. on Signal Processing, 62(1):211–224, 2014.
67. M.-D. Iordache, J.M. Bioucas-Dias, and A. Plaza. Total variation spatial regularization for sparse hyperspectral unmixing. IEEE Trans. on Geoscience and Remote Sensing, 50(11):4484–4502, 2012.
68. S. Jia and Y. Qian. Constrained nonnegative matrix factorization for hyperspectral unmixing. IEEE Trans. on Geoscience and Remote Sensing, 47(1):161–173, 2009.
69. V. Kaibel. Extended Formulations in Combinatorial Optimization. Optima, 85:2–7, 2011.
70. B. Kanagal and V. Sindhwani. Rank selection in low-rank matrix approximations. In Advances in Neural Information Processing Systems (NIPS ’10), 2010.
71. Q. Ke and T. Kanade. Robust L1 norm factorization in the presence of outliers and missing data by alternative convex programming. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR ’05), pages 739–746, 2005.
72. H. Kim and H. Park. Sparse non-negative matrix factorizations via alternating non-negativity-constrained least squares for microarray data analysis. Bioinformatics, 23(12):1495–1502, 2007.
73. H. Kim and H. Park. Non-negative Matrix Factorization Based on Alternating Non-negativity Constrained Least Squares and Active Set Method. SIAM J. on Matrix Analysis and Applications, 30(2):713–730, 2008.
74. J. Kim, Y. He, and H. Park. Algorithms for nonnegative matrix and tensor factorizations: A unified view based on block coordinate descent framework. Journal of Global Optimization, 2013. doi:10.1007/s10898- 013-0035-4.
75. J. Kim and H. Park. Fast nonnegative matrix factorization: An activeset- like method and comparisons. SIAM J. on Scientific Computing, 33(6):3261–3281, 2011.
76. A. Kumar and V. Sindhwani. Near-separable non-negative matrix factorization with `1- and Bregman loss functions. arXiv:1312.7167, 2013.
77. A. Kumar, V. Sindhwani, and P. Kambadur. Fast conical hull algorithms for near-separable non-negative matrix factorization. In Int. Conf. on Machine Learning (ICML ’13), volume 28, pages 231–239. 2013.
78. C.L. Lawson and R.J. Hanson. Solving Least Squares Problems. Prentice- Hall, 1974.
79. D.D. Lee and H.S. Seung. Learning the Parts of Objects by Nonnegative Matrix Factorization. Nature, 401:788–791, 1999.
80. D.D. Lee and H.S. Seung. Algorithms for Non-negative Matrix Factorization. In Advances in Neural Information Processing (NIPS ’01), 13, 2001.
81. T. Lee and A. Shraibman. Lower bounds in communication complexity. Now Publishers Inc., 2009.
82. A. Lefèvre. Dictionary learning methods for single-channel source separations. PhD thesis, Ecole Normale Supérieure de Cachan, 2012.
83. L. Li, G. Lebanon, and H. Park. Fast Bregman divergence NMF using Taylor expansion and coordinate descent. In Proc. of the 18th ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, pages 307–315, 2012.
84. L. Li and Y.-J. Zhang. FastNMF: highly efficient monotonic fixed-point nonnegative matrix factorization algorithm with good applicability. J. Electron. Imaging, Vol. 18(033004), 2009.
85. T. Li, Y. Zhang, and V. Sindhwani. A non-negative matrix trifactorization approach to sentiment classification with lexical prior knowledge. In Association of Computational Lingustics, pages 244–252, 2009.
86. Y. Li, D.M. Sima, S. Van Cauter, A.R. Croitor Sava, U. Himmelreich, Y. Pi, and S. Van Huffel. Hierarchical non-negative matrix factorization (hNMF): a tissue pattern differentiation method for glioblastoma multiforme diagnosis using MRSI. NMR in Biomedicine, 26(3):307–319, 2013.
87. C.-J. Lin. On the convergence of multiplicative update algorithms for nonnegative matrix factorization. IEEE Trans. on Neural Networks, 18(6):1589–1596, 2007.
88. C.-J. Lin. Projected Gradient Methods for Nonnegative Matrix Factorization. Neural Computation, 19:2756–2779, 2007.
89. J. Liu, J. Liu, P. Wonka, and J. Ye. Sparse non-negative tensor factorization using columnwise coordinate descent. Pattern Recognition, 45(1):649–656, 2012.
90. W.-K. Ma, J.M. Bioucas-Dias, T.-H. Chan, N. Gillis, P. Gader, A. Plaza, A. Ambikapathi, and C.-Y. Chi. A Signal Processing Perspective on Hyperspectral Unmixing. IEEE Signal Processing Magazine, 31(1):67– 81, 2014.
91. M.W. Mahoney and P. Drineas. CUR matrix decompositions for improved data analysis. Proc. of the National Academy of Sciences, 106(3): 697–702, 2009.
92. P. Melville and V. Sindhwani. Recommender systems. Encyclopedia of machine learning, 1:829–838, 2010.
93. L. Miao and H. Qi. Endmember extraction from highly mixed data using minimum volume constrained nonnegative matrix factorization. IEEE Trans. on Geoscience and Remote Sensing, 45(3):765–777, 2007.
94. A. Moitra. An almost optimal algorithm for computing nonnegative rank. In Proc. of the 24th Annual ACM-SIAM Symp. on Discrete Algorithms (SODA ’13), pages 1454–1464, 2013.
95. W. Naanaa and J.-M. Nuzillard. Blind source separation of positive and partially correlated data. Signal Processing, 85(9):1711–1722, 2005.
96. J.M.P. Nascimento and J.M. Bioucas-Dias. Vertex component analysis: a fast algorithm to unmix hyperspectral data. IEEE Trans. on Geoscience and Remote Sensing, 43(4):898–910, 2005.
97. P. Paatero and U. Tapper. Positive matrix factorization: a non-negative factor model with optimal utilization of error estimates of data values. Environmetrics, 5:111–126, 1994.
98. J. Rapin, J. Bobin, A. Larue, and J.-L. Starck. Sparse and non-negative BSS for noisy data. IEEE Trans. on Signal Processing, 61(22):5620–5632, 2013.
99. H. Ren and C.-I. Chang. Automatic spectral target recognition in hyperspectral imagery. IEEE Trans. on Aerospace and Electronic Systems, 39(4):1232–1249, 2003.
100. R. Sandler and M. Lindenbaum. Nonnegative matrix factorization with earth mover’s distance metric. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR ’09), pages 1873–1880, 2009.
101. F. Shahnaz, M.W. Berry, V.P. Pauca, and R.J. Plemmons. Document clustering using nonnegative matrix factorization. Information Processing and Management, 42:373–386, 2006.
102. P. Smaragdis, C. Févotte, G.J. Mysore, N. Mohammadiha, and M. Hoffman. A Unified View of Static and Dynamic Source Separation Using Non-Negative Factorizations. IEEE Signal Processing Magazine, 2014.
103. N. Takahashi and R. Hibi. Global convergence of modified multiplicative updates for nonnegative matrix factorization. Computational Optimization and Applications, 2013. doi:10.1007/s10589-013-9593-0.
104. V.Y.F. Tan and C. Févotte. Automatic relevance determination in nonnegative matrix factorization. In Signal Processing with Adaptive Sparse Structured Representations (SPARS ’09), 2009.
105. S.A. Vavasis. On the complexity of nonnegative matrix factorization. SIAM J. on Optimization, 20(3):1364–1377, 2009.
106. F. Wang, T. Li, X. Wang, S. Zhu, and C. Ding. Community discovery using nonnegative matrix factorization. Data Min. Knowl. Disc., 22(3):493–521, 2011.
107. S. Wild, J. Curry, and A. Dougherty. Improving non-negative matrix factorizations through structured initialization. Pattern Recognition, 37(11):2217–2232, 2004.
108. M. Winter. N-FINDR: an algorithm for fast autonomous spectral endmember determination in hyperspectral data. In Proc. SPIE Conf. on Imaging Spectrometry V, pages 266–275, 1999.
109. Y. Xue, C.S. Tong, Y. Chen, and W.-S. Chen. Clustering-based initialization for non-negative matrix factorization. Applied Mathematics and Computation, 205(2):525–536, 2008.
110. Z. Yang and E. Oja. Linear and nonlinear projective nonnegative matrix factorization. IEEE Trans. on Neural Networks, 21(5):734–749, 2010.
111. M. Yannakakis. Expressing Combinatorial Optimization Problems by Linear Programs. Journal of Computer and System Sciences, 43(3):441–466, 1991.
112. R. Zdunek. Initialization of nonnegative matrix factorization with vertices of convex polytope. In Artificial Intelligence and Soft Computing, volume 7267 of Lecture Notes in Computer Science, pages 448–455. Springer Berlin Heidelberg, 2012.
113. S. Zhong and J. Ghosh. Generative model-based document clustering: a comparative study. Knowledge and Information Systems, 8(3):374–384, 2005.