Optimization-based algorithms for tensor decompositions: Canonical polyadic decomposition, decomposition in rank-(Lr, Lr, 1) terms, and a new generalization

SIAM Journal on Optimization

Volumn 23, Issue 2, 2013, Pages 695-720

  Sorber, Laurent ^a   Van Barel, Marc ^a   De Lathauwer, Lieven ^a  

^a UNIVERSITY OF LEUVEN   (Belgium)

Author keywords

Algorithms; Block term decomposition; Canonical polyadic decomposition; Multilinear algebra; Optimization; Tensor decompositions

Indexed keywords

ALTERNATING LEAST SQUARES; BLOCK TERM DECOMPOSITIONS; CANONICAL POLYADIC DECOMPOSITIONS; MULTI-LINEAR ALGEBRAS; NONLINEAR LEAST SQUARES METHODS; OPTIMIZATION-BASED ALGORITHM; TENSOR DECOMPOSITION; UNCONSTRAINED OPTIMIZATION;

ALGEBRA; NUMERICAL METHODS; OPTIMIZATION; SIGNAL PROCESSING; TENSORS;

ALGORITHMS;

EID: 84880828991     PISSN: 10526234     EISSN: None     Source Type: Journal    
DOI: 10.1137/120868323     Document Type: Article

References (68)

1. T. J. Abatzoglou, J. M. Mendel, and G. A. Harada, The constrained total least squares technique and its applications to harmonic superresolution, IEEE Trans. Signal Process., 39 (1991), pp. 1070-1087.
2. E. Acar, T. G. Kolda, and D. M. Dunlavy, An Optimization Approach for Fitting Canonical Tensor Decompositions, Technical report, Sandia National Laboratories, Albuquerque, NM and Livermore, CA, 2009.
3. J. Albersmeyer and M. Diehl, The lifted Newton method and its application in optimization, SIAM J. Optim., 20 (2010), pp. 1655-1684.
4. B. W. Bader and T. G. Kolda, Efficient MATLAB computations with sparse and factored tensors, SIAM J. Sci. Comput., 30 (2006), pp. 205-231.
5. R. Bro, Multi-Way Analysis in the Food Industry: Models, Algorithms, and Applications, Ph.D. thesis, University of Amsterdam, Amsterdam, Netherlands, 1998.
6. R. H. Byrd, R. B. Schnabel, and G. A. Schultz, Approximate solution of the trust region problem by minimization over two-dimensional subspaces, Math. Program., 40 (1988), pp. 247-263.
7. J. D. Carroll and J.-J. Chang, Analysis of individual differences in multidimensional scaling via an n-way generalization of "Eckart-Young" decomposition, Psychometrika, 35 (1970), pp. 283-319.
8. A. Cichocki, R. Zdunek, A. H. Phan, and S.-I. Amari, Nonnegative Matrix and Tensor Factorizations: Applications to Exploratory Multi-Way Data Analysis and Blind Source Separation, Wiley, New York, 2009.
9. P. Comon and Chr. Jutten, Handbook of Blind Source Separation: Independent Component Analysis and Applications, Academic Press, New York, 2010.
10. P. Comon, X. Luciani, and A. L. F. de Almeida, Tensor decompositions, alternating least squares and other tales, J. Chem., 23 (2009), pp. 393-405.
11. L. De Lathauwer, A link between the canonical decomposition in multilinear algebra and simultaneous matrix diagonalization, SIAM J. Matrix Anal. Appl., 28 (2006), pp. 642-666.
12. L. De Lathauwer, Decompositions of a higher-order tensor in block terms - Part I: Lemmas for partitioned matrices, SIAM J. Matrix Anal. Appl., 30 (2008), pp. 1022-1032.
13. L. De Lathauwer, Decompositions of a higher-order tensor in block terms - Part II: Definitions and uniqueness, SIAM J. Matrix Anal. Appl., 30 (2008), pp. 1033-1066.
14. r, 1) terms, SIAM J. Matrix Anal. Appl., 32 (2011), pp. 1451-1474.
15. L. De Lathauwer, A short introduction to tensor-based methods for factor analysis and blind source separation, in Proceedings of the 7th International Symposium on Image and Signal Processing and Analysis (ISPA 2011), 2011, pp. 558-563.
16. L. De Lathauwer, Block component analysis, a new concept for source separation and factor analysis, in Latent Variable Analysis and Signal Separation, Lecture Notes in Comput. Sci. 7191, Springer, Berlin, 2012, pp. 1-8.
17. L. De Lathauwer, P. Comon, and N. Mastronardi, Special issue on tensor decompositions and applications, SIAM J. Matrix Anal. Appl., 30 (2008), pp. 1-1279.
18. L. De Lathauwer and A. de Baynast, Blind deconvolution of DS-CDMA signals by means of decomposition in rank-(1, L,L) terms, IEEE Trans. Signal Process., 56 (2008), pp. 1562-1571.
19. L. De Lathauwer, B. L. R. De Moor, and J. Vandewalle, A multilinear singular value decomposition, SIAM J. Matrix Anal. Appl., 21 (2000), pp. 1253-1278.
20. L. De Lathauwer and D. Nion, Decompositions of a higher-order tensor in block terms - Part III: Alternating least squares algorithms, SIAM J. Matrix Anal. Appl., 30 (2008), pp. 1067-1083.
21. V. de Silva and L.-H. Lim, Tensor rank and the ill-posedness of the best low-rank approximation problem, SIAM J. Matrix Anal. Appl., 30 (2008), pp. 1084-1127.
22. J. E. Dennis, Jr. and H. H. W. Mei, Two new unconstrained optimization algorithms which use function and gradient values, J. Optim. Theory Appl., 28 (1979), pp. 453-482.
23. E. D. Dolan and J. J. Moré, Benchmarking optimization software with performance profiles, Math. Program., 91 (2002), pp. 201-213.
24. I. Domanov and L. De Lathauwer, On the Uniqueness of the Canonical Polyadic Decomposition - Part I: Basic Results and Uniqueness of One Factor Matrix, Internal report 12-72, ESAT-SISTA, KU Leuven, 2012.
25. I. Domanov and L. De Lathauwer, On the Uniqueness of the Canonical Polyadic Decomposition - Part II: Overall Uniqueness, Internal report 12-90, ESAT-SISTA, KU Leuven, 2012.
26. W. W. Hager and H. Zhang, A new conjugate gradient method with guaranteed descent and an efficient line search, SIAM J. Optim., 16 (2006), pp. 170-192.
27. R. A. Harshman, Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi-modal factor analysis, UCLA Working Papers in Phonetics, 16 (1970), pp. 84-84.
28. C. Hayashi and F. Hayashi, A new algorithm to solve PARAFAC-model, Behaviormetrika, 9 (1982), pp. 49-60.
29. F. L. Hitchcock, The expression of a tensor or a polyadic as a sum of products, J. Math. Phys., 6 (1927), pp. 164-189.
30. F. L. Hitchcock, Multiple invariants and generalized rank of a p-way matrix or tensor, J. Math. Phys., 7 (1927), pp. 39-79.
31. T. Jiang and N. D. Sidiropoulos, Kruskal's permutation lemma and the identification of CANDECOMP/PARAFAC and bilinear models with constant modulus constraints, IEEE Trans. Signal Process., 52 (2004), pp. 2625-2636.
32. C. G. Khatri and C. R. Rao, Solutions to some functional equations and their applications to characterization of probability distributions, Sankhyā Ser. A, 30 (1968), pp. 167-180.
33. T. G. Kolda and B. W. Bader, Tensor decompositions and applications, SIAM Rev., 51 (2009), pp. 455-500.
34. W. P. Krijnen, T. K. Dijkstra, and A. Stegeman, On the non-existence of optimal solutions and the occurrence of "degeneracy" in the Candecomp/Parafac model, Psychometrika, 73 (2008), pp. 431-439.
35. P. M. Kroonenberg, Applied Multiway Data Analysis, Wiley, New York, 2008.
36. J. B. Kruskal, Three-way arrays: Rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics, Linear Algebra Appl., 18 (1977), pp. 95-138.
37. J. B. Kruskal, Rank, Decomposition, and Uniqueness for 3-way and N-way Arrays, Elsevier Science Publishers B.V., Amsterdam, 1989, pp. 7-18.
38. J. B. Kruskal, R. A. Harshman, and M. E. Lundy, How 3-MFA data can cause degenerate PARAFAC solutions, among other relationships, in Multiway Data Analysis, R. Coppi and S. Bolasco, eds., North-Holland, Amsterdam, 1989, pp. 115-121.
39. B. C. Mitchell and D. S. Burdick, Slowly converging PARAFAC sequences: Swamps and two-factor degeneracies, J. Chem., 8 (1994), pp. 155-168.
40. J. J. Moré and D. J. Thuente, Line search algorithms with guaranteed sufficient decrease, ACM Trans. Math. Software, 20 (1994), pp. 286-307.
41. C. Navasca, L. De Lathauwer, and S. Kindermann, Swamp reducing technique for tensor decomposition, in Proceedings of the 16th European Signal Processing Conference (EUSIPCO 2008), Lausanne, Switzerland, 2008.
42. D. Nion and L. De Lathauwer, An enhanced line search scheme for complex-valued tensor decompositions. Application in DS-CDMA, Signal Process., 88 (2008), pp. 749-755.
43. J. Nocedal and S. J. Wright, Numerical Optimization, Springer Ser. Oper. Res., 2nd ed., Springer, New York, 2006.
44. P. Paatero, A weighted non-negative least squares algorithm for three-way "PARAFAC" factor analysis, Chemometr. Intell. Lab. Syst., 38 (1997), pp. 223-242.
45. P. Paatero, The multilinear engine: A table-driven, least squares program for solving multilinear problems, including the n-way parallel factor analysis model, J. Comput. Graph. Statist., 8 (1999), pp. 854-888.
46. P. Paatero, Construction and analysis of degenerate PARAFAC models, J. Chem., 14 (2000), pp. 285-299.
47. A. H. Phan, P. Tichavsk'y, and A. Cichocki, Low complexity damped Gauss-Newton algorithms for CANDECOMP/PARAFAC, Comp. Research Repos., arXiv:1205.2584, 2012.
48. A. H. Phan, P. Tichavsk'y, and A. Cichocki, On fast computation of gradients for CANDECOMP/PARAFAC algorithms, Comp. Research Repos., arXiv:1204.1586, 2012.
49. M. J. D. Powell, A hybrid method for nonlinear equations, in Numerical Methods for Nonlinear Algebraic Equations, P. Robinowitz, ed., Gordon and Breach Science, London, 1970, pp. 87-144.
50. M. J. D. Powell, A new algorithm for unconstrained optimization, in Nonlinear Programming, J. B. Rosen, O. L. Mangasarian, and K. Ritter, eds., Academic Press, New York, 1970, pp. 31-66.
51. M. Rajih, P. Comon, and R. A. Harshman, Enhanced line search: A novel method to accelerate PARAFAC, SIAM J. Matrix Anal. Appl., 30 (2008), pp. 1128-1147.
52. C. R. Rao and M. B. Rao, Matrix Algebra and its Applications to Statistics and Econometrics, World Scientific, River Edge, NJ, 1998.
53. R. Remmert, Theory of Complex Functions, Springer-Verlag, New York, 1991.
54. N. D. Sidiropoulos and R. Bro, On the uniqueness of multilinear decomposition of N-way arrays, J. Chem., 14 (2000), pp. 229-239.
55. N. D. Sidiropoulos, R. Bro, and G. B. Giannakis, Parallel factor analysis in sensor array processing, IEEE Trans. Signal Process., 48 (2000), pp. 2377-2388.
56. N. D. Sidiropoulos and G. Z. Dimić, Blind multiuser detection in W-CDMA systems with large delay spread, IEEE Signal Process. Lett., 8 (2001), pp. 87-89.
57. N. D. Sidiropoulos, G. B. Giannakis, and R. Bro, Blind PARAFAC receivers for DS-CDMA systems, IEEE Trans. Signal Process., 48 (2000), pp. 810-823.
58. A. Smilde, R. Bro, and P. Geladi, Multi-Way Analysis with Applications in the Chemical Sciences, Wiley, New York, 2004.
59. L. Sorber, M. Van Barel, and L. De Lathauwer, Unconstrained optimization of real functions in complex variables, SIAM J. Optim., 22 (2012), pp. 879-898.
60. A. Stegeman, Degeneracy in Candecomp/Parafac explained for p×p×2 arrays of rank p+1 or higher, Psychometrika, 71 (2006), pp. 483-501.
61. A. Stegeman, Low-rank approximation of generic p × q × 2 arrays and diverging components in the Candecomp/Parafac model, SIAM J. Matrix Anal. Appl., 30 (2008), pp. 988-1007.
62. T. Steihaug, The conjugate gradient method and trust regions in large scale optimization, SIAM J. Numer. Anal., 20 (1983), pp. 626-637.
63. G. Tomasi, Practical and Computational Aspects in Chemometric Data Analysis, Ph.D. thesis, The Royal Veterinary and Agricultural University, Frederiksberg, Denmark, 2006.
64. G. Tomasi and R. Bro, A comparison of algorithms for fitting the PARAFAC model, Comput. Statist. Data Anal., 50 (2006), pp. 1700-1734.
65. L. R. Tucker, The extension of factor analysis to three-dimensional matrices, in Contributions to Mathematical Psychology, H. Gulliksen and N. Frederiksen, eds., Holt, Rinehart and Winston, New York, 1964, pp. 110-127.
66. L. R. Tucker, Some mathematical notes on three-mode factor analysis, Psychometrika, 31 (1966), pp. 279-311.
67. A. van den Bos, Complex gradient and Hessian, IEE Proc. Vis., Image & Signal Process., 141 (1994), pp. 380-383.
68. P. Wolfe, Convergence conditions for ascent methods, SIAM Rev., 11 (1969), pp. 226-235.