Newton-based optimization for Kullback-Leibler nonnegative tensor factorizations

Optimization Methods and Software

Volumn 30, Issue 5, 2015, Pages 1002-1029

  Hansen, Samantha ^a   Plantenga, Todd ^b   Kolda, Tamara G ^b  

^a Northwestern University   (United States)

^b SANDIA NATIONAL LABORATORIES   (United States)

Author keywords

Kullback Leibler; multilinear algebra; nonlinear optimization; poisson; tensor factorization

Indexed keywords

FACTORIZATION; NEWTON-RAPHSON METHOD; NONLINEAR PROGRAMMING;

KULLBACK-LEIBLER; MULTI-LINEAR ALGEBRAS; NON-LINEAR OPTIMIZATION; POISSON; TENSOR FACTORIZATION;

TENSORS;

EID: 84940448141     PISSN: 10556788     EISSN: 10294937     Source Type: Journal    
DOI: 10.1080/10556788.2015.1009977     Document Type: Article

References (44)

1. E. Acar, D.M. Dunlavy, and, T.G. Kolda., A scalable optimization approach for fitting canonical tensor decompositions, J. Chemom. 25 (2011), pp. 67-86. doi: 10.1002/cem.1335
2. G. Andrew, and, J. Gao.,. Scalable Training of-Regularized Log-Linear Models, Proceedings of the 24th International Conference on Machine Learning, Corvallis, OR, ACM, 2007, pp. 33-40. Available at http://dl.acm.org/citation.cfm?id=1273501.
3. B.W. Bader, M.W. Berry, and, M. Browne.,. Discussion tracking in Enron email using PARAFAC, in Survey of Text Mining: Clustering, Classification, and Retrieval, M.W. Berry and M. Castellanos, eds., 2nd ed., Springer, New York, 2007, pp. 147-162 (Chapter 8).
4. B.W. Bader, T.G. Kolda, et al.,. MATLAB Tensor Toolbox Version 2.5, preprint (2012). Available at http://www.sandia.gov/~tgkolda/TensorToolbox/http://www.sandia.gov/∼tgkolda/TensorToolbox/.
5. D. Bertsekas., On the Goldstein-Levitin-Polyak gradient projection method, IEEE Trans. Autom. Control 21 (1976), pp. 174-184. doi: 10.1109/TAC.1976.1101194
6. D.P. Bertsekas., Projected Newton methods for optimization problems with simple constraints, SIAM J. Control Optim. 20 (1982), pp. 221-246. doi: 10.1137/0320018
7. R. Bro.,. Multi-way analysis in the food industry: Models, algorithms, and applications, Ph.D. thesis, Universiteit van Amsterdam, 1998.
8. R. Bro, and, S.D. Jong., A fast non-negativity-constrained least squares algorithm, J. Chemom. 11 (1997), pp. 393-401. doi: 10.1002/(SICI)1099-128X(199709/10)11:5<393:AID-CEM483>3.0.CO;2-L
9. R.H. Byrd, P. Lu, J. Nocedal, and, C. Zhu., A limited memory algorithm for bound constrained optimization, SIAM J. Sci. Comput. 16 (1995), pp. 1190-1208. doi: 10.1137/0916069
10. J.D. Carroll, and, J.J. Chang., Analysis of individual differences in multidimensional scaling via an n-way generalization of Eckart-Young decomposition, Psychometrika 35 (1970), pp. 283-319. doi: 10.1007/BF02310791
11. E.C. Chi, and, T.G. Kolda., On tensors, sparsity, and nonnegative factorizations, SIAM J. Matrix Anal. Appl. 33 (2012), pp. 1272-1299. doi: 10.1137/110859063
12. A. Cichocki, and, A.-H. Phan., Fast local algorithms for large scale nonnegative matrix and tensor factorizations, IEICE Trans. Fundam. Electron. Commun. Comput. Sci. 92 (2009), pp. 708-721. doi: 10.1587/transfun.E92.A.708
13. A.R. Conn, N.I.M. Gould, and, Ph.L. Toint., Global convergence of a class of trust region algorithms for optimization with simple bounds, SIAM J. Numer. Anal. 25 (1988), pp. 433-460. doi: 10.1137/0725029
14. D.M. Dunlavy, T.G. Kolda, and, E. Acar., Temporal link prediction using tensor and matrix factorizations, ACM Trans. Knowl. Discovery Data 5 (2011), pp. 1-27. doi: 10.1145/1921632.1921636
15. M.P. Friedlander, and, K. Hatz., Computing nonnegative tensor factorizations, Comput. Optim. Appl. 23 (2008), pp. 631-647.
16. E. Gonzalez, and, Y. Zhang.,. Accelerating the Lee-Seung algorithm for non-negative matrix factorization, Tech. Rep. TR-05-02, Department of Computational and Applied Mathematics, Rice University, Houston, TX, 2005.
17. R.A. Harshman.,. Foundations of the PARAFAC procedure: Models and conditions for an explanatory multi-modal factor analysis, UCLA Working Papers in Phonetics, 16, 1970. Available at http://publish.uwo.ca/~harshman/wpppfac0.pdf http://publish.uwo.ca/∼harshman/wpppfac0.pdf.
18. C.-J. Hsieh, and, I.S. Dhillon.,. Fast Coordinate Descent Methods with Variable Selection for Non-negative Matrix Factorization, Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Diego, CA, ACM, 2011, pp. 1064-1072. Available at http://dl.acm.org/citation.cfm?id=2020577.
19. H. Kim, and, H. Park., Nonnegative matrix factorization based on alternating nonnegativity constrained least squares and active set method, SIAM J. Matrix Anal. Appl. 30 (2008), pp. 713-730. doi: 10.1137/07069239X
20. J. Kim, and, H. Park.,. Fast nonnegative tensor factorization with an active-set-like method, in High-Performance Scientific Computing, Algorithms and Applications, M.W. Berry, K.A. Gallivan, E. Gallopoulos, A. Grama, B. Philippe, Y. Saad, and F. Saied, eds., Springer-Verlag, London, 2012, pp. 311-326.
21. D. Kim, S. Sra, and, I.S. Dhillon., Fast projection-based methods for the least squares non-negative matrix approximation problem, Stat. Anal. Data Min. 1 (2008), pp. 38-51. doi: 10.1002/sam.104
22. D. Kim, S. Sra, and, I.S. Dhillon., Tackling box-constrained optimization via a new projected quasi-Newton approach, SIAM J. Sci. Comput. 32 (2010), pp. 3548-3563. doi: 10.1137/08073812X
23. T.G. Kolda, and, B.W. Bader., Tensor decompositions and applications, SIAM Rev. 51 (2009), pp. 455-500. doi: 10.1137/07070111X
24. D.D. Lee, and, H.S. Seung., Learning the parts of objects by non-negative matrix factorization, Nature 401 (1999), pp. 788-791. doi: 10.1038/44565
25. D.D. Lee, and, H.S. Seung., Algorithms for non-negative matrix factorization, Adv. Neural Inform. Process. Syst. 13 (2001), pp. 556-562.
26. M. Ley.,. The DBLP Computer Science Bibliography: Evolution, Research Issues, Perspectives, Proceedings of the 9th International Symposium on String Processing and Information Retrieval, Vol. 2476, 2002, pp. 1-10. Available at http://www.informatik.uni-trier.de/~ley/db/.
27. C.J. Lin., Projected gradient methods for nonnegative matrix factorization, Neural Comput. 19 (2007), pp. 2756-2779. doi: 10.1162/neco.2007.19.10.2756
28. J. Liu, J. Liu, P. Wonka, and, J. Ye., Sparse non-negative tensor factorization using columnwise coordinate descent, Pattern Recognit. Lett. 45 (2012), pp. 649-656. doi: 10.1016/j.patcog.2011.05.015
29. J. Nocedal., Updating quasi-Newton matrices with limited storage, Math. Comput. 35 (1980), pp. 773-782. doi: 10.1090/S0025-5718-1980-0572855-7
30. J. Nocedal and S.J. Wright, Numerical Optimization, 2nd ed., Springer, New York, 2006.
31. P. Paatero., A weighted non-negative least squares algorithm for three-way PARAFAC factor analysis, Chemom. Intell. Lab. Syst. 38 (1997), pp. 223-242. doi: 10.1016/S0169-7439(97)00031-2
32. P. Paatero, and, U. Tapper., Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values, Environmetrics 5 (1994), pp. 111-126. doi: 10.1002/env.3170050203
33. A.-H. Phan, A. Cichocki, R. Zdunek, and, T. Dinh.,. Novel alternating least squares algorithm for nonnegative matrix and tensor factorizations, in Neural Information Processing. Theory and Algorithms, K.W. Wong, B.S.U. Mendis, and A. Bouzerdoum, eds., Lecture Notes in Computer Science, Vol. 6443, Springer, Berlin, 2010, pp. 262-269.
34. A.-H. Phan, P. Tichavsky, and, A. Cichocki.,. Fast Damped Gauss-Newton Algorithm for Sparse and Nonnegative Tensor Factorization, in 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Prague, 2011, pp. 1988-1991.
35. M. Schmidt, D. Kim, and, S. Sra.,. Projected Newton-type methods in machine learning, in Optimization for Machine Learning, S. Sra, S. Nowozin, and S.J. Wright, eds., MIT Press, Cambridge, MA, 2011, pp. 305-330.
36. J. Sun, D. Tao, and, C. Faloutsos.,. Beyond Streams and Graphs: Dynamic Tensor Analysis, in KDD '06, Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Philadelphia, PA, ACM, 2006, pp. 374-383. Available at http://dl.acm.org/citation.cfm?id=1150445.
37. G. Tomasi, and, R. Bro., A comparison of algorithms for fitting the PARAFAC model, Comput. Stat. Data Anal. 50 (2006), pp. 1700-1734. doi: 10.1016/j.csda.2004.11.013
38. S. Vavasis., On the complexity of nonnegative matrix factorization, SIAM J. Optim. 20 (2009), pp. 1364-1377. doi: 10.1137/070709967
39. M. Welling, and, M. Weber., Positive tensor factorization, Pattern Recognit. Lett. 22 (2001), pp. 1255-1261. doi: 10.1016/S0167-8655(01)00070-8
40. Y. Xu, and, W. Yin.,. A block coordinate descent method for multi-convex optimization with applications to nonnegative tensor factorization and completion, Tech. Rep. 12-15, CAAM Rice University, Houston, TX, 2012.
41. S. Zafeiriou, and, M. Petrou., Nonnegative tensor factorization as an alternative Csiszár-Tusnády procedure: Algorithms, convergence, probabilistic interpretations and novel probabilistic tensor latent variable analysis algorithms, Data Min. Knowl. Discovery 22 (2011), pp. 419-466. doi: 10.1007/s10618-010-0196-4
42. R. Zdunek, and, A. Cichocki.,. Non-negative Matrix Factorization with Quasi-Newton Optimization, in Eighth International Conference on Artificial Intelligence and Soft Computing, ICAISC, Springer, Zakopane, 2006, pp. 870-879.
43. R. Zdunek, and, A. Cichocki., Nonnegative matrix factorization with constrained second-order optimization, Signal Process. 87 (2007), pp. 1904-1916. doi: 10.1016/j.sigpro.2007.01.024
44. Y. Zheng, and, Q. Zhang., Damped Newton based iterative non-negative matrix factorization for intelligent wood defects detection, J. Softw. 5 (2010), pp. 899-906. doi: 10.4304/jsw.5.8.899-906