Unified development of multiplicative algorithms for linear and quadratic nonnegative matrix factorization

IEEE Transactions on Neural Networks

Volumn 22, Issue 12 PART 1, 2011, Pages 1878-1891

  Yang, Zhirong ^a   Oja, Erkki ^a  

^a AALTO UNIVERSITY   (Finland)

Author keywords

Divergence; matrix factorization; multiplicative; nonnegative; optimization

Indexed keywords

AUXILIARY FUNCTIONS; CONVENTIONAL METHODS; DISSIMILARITY MEASURES; DIVERGENCE; GENERAL APPROACH; KARUSH KUHN TUCKERS; MATRIX; MATRIX FACTORIZATIONS; MULTIPLICATIVE; MULTIPLICATIVE UPDATES; NONNEGATIVE; NONNEGATIVE MATRIX FACTORIZATION; NONSEPARABLE; NUMERICAL EXPERIMENTS; OBJECTIVE FUNCTIONS; OPTIMALITY; SECOND ORDERS;

ALGORITHMS; FACTORIZATION; OPTIMIZATION;

MATRIX ALGEBRA;

ALGORITHM; ARTICLE; COMPUTER SIMULATION; STATISTICAL MODEL;

ALGORITHMS; COMPUTER SIMULATION; LINEAR MODELS;

EID: 83855163513     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2011.2170094     Document Type: Article

References (47)

1. D. D. Lee and H. S. Seung, "Learning the parts of objects by non-negative matrix factorization," Nature, vol. 401, pp. 788-791, Oct. 1999.
2. D. D. Lee and H. S. Seung, "Algorithms for non-negative matrix factorization," in Proc. Adv. Neural Inf. Process. Syst., vol. 13. Apr. 2001, pp. 556-562.
3. P. O. Hoyer, "Non-negative matrix factorization with sparseness constraints," J. Mach. Learn. Res., vol. 5, pp. 1457-1469, Nov. 2004.
4. R. Kompass, "A generalized divergence measure for nonnegative matrix factorization," Neural Comput., vol. 19, no. 3, pp. 780-791, Mar. 2007.
5. I. S. Dhillon and S. Sra, "Generalized nonnegative matrix approximations with Bregman divergences," in Proc. Adv. Neural Inf. Process. Syst., vol. 18. 2006, pp. 283-290.
6. A. Cichocki, H. Lee, Y.-D. Kim, and S. Choi, "Non-negative matrix factorization with α-divergence," Pattern Recognit. Lett., vol. 29, pp. 1433-1440, Feb. 2008.
7. S. Choi, "Algorithms for orthogonal nonnegative matrix factorization," in Proc. IEEE Int. Joint Conf. Neural Netw., Hong Kong, Jun. 2008, pp. 1828-1832.
8. C. Févotte, N. Bertin, and J.-L. Durrieu, "Nonnegative matrix factorization with the Itakura-Saito divergence: With application to music analysis," Neural Comput., vol. 21, no. 3, pp. 793-830, Mar. 2009.
9. C. Ding, T. Li, and M. I. Jordan, "Convex and semi-nonnegative matrix factorizations," IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 1, pp. 45-55, Jan. 2010.
10. C. Ding, T. Li, W. Peng, and H. Park, "Orthogonal nonnegative matrix t-factorizations for clustering," in Proc. 12th ACM SIGKDD Int. Conf. Knowl. Disc. Data Min., 2006, pp. 126-135. (Pubitemid 44535510)
11. S. Behnke, "Discovering hierarchical speech features using convolu-tional non-negative matrix factorization," in Proc. Int. Joint Conf. Neural Netw., vol. 4. 2003, pp. 2758-2763.
12. H. Kim and H. Park, "Sparse non-negative matrix factorizations via alternating non-negativity-constrained least squares for microarray data analysis," Bioinformatics, vol. 23, no. 12, pp. 1495-1502, May 2007. (Pubitemid 47244474)
13. A. Cichocki and R. Zdunek, "Multilayer nonnegative matrix factorization using projected gradient approaches," Int. J. Neural Syst., vol. 17, no. 6, pp. 431-446, 2007.
14. A. Cichocki, R. Zdunek, A.-H. Phan, and S. Amari, Nonnegative Matrix and Tensor Factorizations: Applications to Exploratory Multi-Way Data Analysis. New York: Wiley, 2009.
15. C.-J. Lin, "Projected gradient methods for nonnegative matrix factorization," Neural Comput., vol. 19, no. 10, pp. 2756-2779, Oct. 2007.
16. R. Zdunek and A. Cichocki, "Nonnegative matrix factorization with quadratic programming," Neurocomputing, vol. 71, nos. 10-12, pp. 2309-2320, Jun. 2008.
17. D. Kim, S. Sra, and I. S. Dhillon, "Fast projection-based methods for the least squares nonnegative matrix approximation problem," Stat. Anal. Data Min., vol. 1, no. 1, pp. 38-51, Feb. 2008.
18. Z. Yang and J. Laaksonen, "Multiplicative updates for non-negative projections," Neurocomputing, vol. 71, nos. 1-3, pp. 363-373, Dec. 2007. (Pubitemid 350028651)
19. H. Fujisawa and S. Eguchi, "Robust parameter estimation with a small bias against heavy contamination," J. Multivar. Anal., vol. 99, no. 9, pp. 2053-2081, Oct. 2008.
20. C. Févotte and J. Idier, "Algorithms for nonnegative matrix factorization with the β-divergence," Neural Comput., vol. 23, no. 9, pp. 2421-2456, Sep. 2011.
21. A. Cichocki, R. Zdunek, and S. Amari, "Csiszár's divergences for non-negative matrix factorization: Family of new algorithms," in Proc. Int. Conf. Independent Comp. Anal. Blind Signal Separat., 2006, pp. 32-39. (Pubitemid 43998396)
22. H. Lee, J. Yoo, and S. Choi, "Semi-supervised nonnegative matrix factorization," IEEE Signal Process. Lett., vol. 17, no. 1, pp. 4-7, Jan. 2010.
23. D. R. Hunter and K. Lange, "A tutorial on MM algorithms," Amer. Stat., vol. 58, no. 1, pp. 30-37, Feb. 2004. (Pubitemid 38260327)
24. L. T. H. An and P. D. Tao, "The DC (difference of convex functions) programming and DCA revisited with DC models of real world nonconvex optimization problems," Ann. Oper. Res., vol. 133, nos. 1-4, pp. 23-46, 2005. (Pubitemid 40389738)
25. F. Sha, Y. Lin, L. K. Saul, and D. D. Lee, "Multiplicative updates for nonnegative quadratic programming," Neural Comput., vol. 19, no. 8, pp. 2004-2031, Aug. 2007.
26. M. Nakano, H. Kameoka, J. L. Roux, Y. Kitano, N. Ono, and S. Sagayama, "Convergence-guaranteed multiplicative algorithms for nonnegative matrix factorization with β-divergence," in Proc. IEEE Int. Workshop Mach. Learn. Signal Process., Kittilä, Finland, Aug.-Sep. 2010, pp. 283-288.
27. M. Spivak, Calculus. Houston, TX: Perish, 1994.
28. J. Kivinen and M. K. Warmuth, "Exponentiated gradient versus gradient descent for linear predictors," Inf. Comput., vol. 132, no. 1, pp. 1-63, Jan. 1997. (Pubitemid 127454001)
29. R. Salakhutdinov and S. Roweis, "Adaptive overrelaxed bound optimization methods," in Proc. 20th Int. Conf. Mach. Learn., Washington D.C., 2003, pp. 664-671.
30. Z. Yuan and E. Oja, "Projective nonnegative matrix factorization for image compression and feature extraction," in Proc. 14th Scandinavian Conf. Image Anal., Joensuu, Finland, Jun. 2005, pp. 333-342. (Pubitemid 41422676)
31. Z. Yang, Z. Yuan, and J. Laaksonen, "Projective non-negative matrix factorization with applications to facial image processing," Int. J. Pattern Recognit. Artif. Intell., vol. 21, no. 8, pp. 1353-1362, Dec. 2007.
32. Z. Yang and E. Oja, "Linear and nonlinear projective nonnegative matrix factorization," IEEE Trans. Neural Netw., vol. 21, no. 5, pp. 734-749, May 2010.
33. C. Ding, T. Li, and M. I. Jordan, "Nonnegative matrix factorization for combinatorial optimization: Spectral clustering, graph matching, and clique finding," in Proc. 8th IEEE Int. Conf. Data Min., Pisa, Italy, Dec. 2008, pp. 183-192.
34. B. Lakshminarayanan and R. Raich, "Non-negative matrix factorization for parameter estimation in hidden Markov models," in Proc. IEEE Int. Workshop Mach. Learn. Signal Process., Kittilä, Finland, Aug.-Sep. 2010, pp. 89-94.
35. C.-J. Lin, "On the convergence of multiplicative update algorithms for nonnegative matrix factorization," IEEE Trans. Neural Netw., vol. 18, no. 6, pp. 1589-1596, Nov. 2007. (Pubitemid 350148414)
36. D. Donoho and V. Stodden, "When does non-negative matrix factorization give a correct decomposition into parts?" in Proc. Adv. Neural Inf. Process. Syst. 16, 2003, pp. 1141-1148.
37. F. Samaria and A. Harter, "Parameterisation of a stochastic model for human face identification," in Proc. 2nd IEEE Workshop Appl. Comput. Vi s., Sarasota, FL, Dec. 1994, pp. 138-142.
38. E. F. Gonzalez and Y. Zhang, "Accelerating the Lee-Seung algorithm for nonnegative matrix factorization," Dept. Comput. Appl. Math., Rice Univ., Houston, TX, Tech. Rep. TR05-02, Mar. 2005.
39. R. Badeau, N. Bertin, and E. Vincent, "Stability analysis of multiplicative update algorithms and application to nonnegative matrix factorization," IEEE Trans. Neural Netw., vol. 21, no. 12, pp. 1869-1881, Dec. 2010.
40. Z. Zheng, J. Yang, and Y. Zhu, "Initialization enhancer for non-negative matrix factorization," Eng. Appl. Artif. Intell., vol. 20, no. 1, pp. 101-110, Feb. 2007. (Pubitemid 44692462)
41. Y. Kim and S. Choi, "A method of initialization for nonnegative matrix factorization," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., Honolulu, HI, Apr. 2007, pp. 537-540.
42. L. Eldén, Matrix Methods in Data Mining and Pattern Recognition. Philadelphia, PA: SIAM, 2007.
43. A. Cichocki and S.-I. Amari, "Families of alpha-beta-and gamma-divergences: Flexible and robust measures of similarities," Entropy, vol. 12, no. 6, pp. 1532-1568, 2010.
44. A. Cichocki, S. Cruces, and S.-I. Amari, "Generalized alpha-beta divergences and their application to robust nonnegative matrix factorization," Entropy, vol. 13, no. 1, pp. 134-170, 2011.
45. H. Choi, S. Choi, A. Katake, and Y. Choe, "Learning α-integration with partially-labeled data," in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., Dallas, TX, Mar. 2010, pp. 2058-2061.
46. M. Mollah, N. Sultana, and M. Minami, "Robust extraction of local structures by the minimum β-divergence method," Neural Netw., vol. 23, no. 2, pp. 226-238, Mar. 2010.
47. F. Sha, L. K. Saul, and D. D. Lee, "Multiplicative updates for large margin classifiers," in Proc. 16th Annu. Conf. Comput. Learn. Theory, 2003, pp. 188-202. (Pubitemid 37053204)