A majorization-minimization approach to the sparse generalized eigenvalue problem

Machine Learning

Volumn 85, Issue 1-2, 2011, Pages 3-39

  Sriperumbudur, Bharath K ^a   Torres, David A ^b   Lanckriet, Gert R G ^a,b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Canonical correlation analysis; Cross language document retrieval; D.c. program; Fisher discriminant analysis; Generalized eigenvalue problem; Majorization minimization; Music annotation; Principal component analysis; Sparsity; Zangwill's theory of global convergence

Indexed keywords

CANONICAL CORRELATION ANALYSIS; D.C. PROGRAM; DOCUMENT RETRIEVAL; FISHER DISCRIMINANT ANALYSIS; GENERALIZED EIGENVALUE PROBLEMS; GLOBAL CONVERGENCE; MAJORIZATION-MINIMIZATION; MUSIC ANNOTATION; PRINCIPAL COMPONENTS; SPARSITY;

CONVERGENCE OF NUMERICAL METHODS; DATA REDUCTION; DISCRIMINANT ANALYSIS; EIGENVALUES AND EIGENFUNCTIONS; FISHER INFORMATION MATRIX; LEARNING ALGORITHMS; OPTIMIZATION; PRINCIPAL COMPONENT ANALYSIS;

PROBLEM SOLVING;

EID: 80053050395     PISSN: 08856125     EISSN: 15730565     Source Type: Journal    
DOI: 10.1007/s10994-010-5226-3     Document Type: Article

References (64)

1. U. Alon N. Barkai D. A. Notterman K. Gish S. Ybarra D. Mack A. J. Levine 1999 Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon cancer tissues Cell Biology 96 6745 6750 (Pubitemid 29274954)
2. Bioucas-Dias, J., Figueiredo, M., & Oliveira, J. (2006). Total-variation image deconvolution: a majorization-minimization approach. In: Proc. IEEE international conference on acoustics, speech, and signal processing, Toulouse, France.
3. D. Böhning B. G. Lindsay 1988 Monotonicity of quadratic- approximation algorithms Annals of the Institute of Statistical Mathematics 40 4 641 663 996690 0723.65150 10.1007/BF00049423
4. Bonnans, J. F., Gilbert, J. C., Lemaréchal, C., & Sagastizábal, C. A. (2006). Numerical optimization: theoretical and practical aspects. Berlin: Springer.
5. Boyd, S. P., & Vandenberghe, L. (2004). Convex optimization. Cambridge: Cambridge University Press.
6. Bradley, P. S., & Mangasarian, O. L. (1998). Feature selection via concave minimization and support vector machines. In Proc. 15th international conf. on machine learning (pp. 82-90). San Francisco: Kaufmann.
7. J. Cadima I. Jolliffe 1995 Loadings and correlations in the interpretation of principal components Applied Statistics 22 203 214 1342655 10.1080/757584614
8. 1 minimization The Journal of Fourier Analysis and Applications 14 877 905 2461611
9. d'Aspremont, A., El Ghaoui, L., Jordan, M. I., & Lanckriet, G. R. G. (2005). A direct formulation for sparse PCA using semidefinite programming. In L. K. Saul, Y. Weiss, & L. Bottou (Eds.), Advances in neural information processing systems (Vol. 17, pp. 41-48). Cambridge: MIT Press.
10. A. d'Aspremont L. El Ghaoui M. I. Jordan G. R. G. Lanckriet 2007 A direct formulation for sparse PCA using semidefinite programming SIAM Review 49 3 434 448 2353806 1128.90050 10.1137/050645506 (Pubitemid 47377693)
11. A. d'Aspremont F. R. Bach L. El Ghaoui 2008 Optimal solutions for sparse principal component analysis Journal of Machine Learning Research 9 1269 1294
12. I. Daubechies M. Defrise C. D. Mol 2004 An iterative thresholding algorithm for linear inverse problems with a sparsity constraint Communications on Pure and Applied Mathematics 57 1413 1457 2077704 1077.65055 10.1002/cpa.20042 (Pubitemid 39427442)
13. de Leeuw, J. (1977). Applications of convex analysis to multidimensional scaling. In J. R. Barra, F. Brodeau, G. Romier, & B. V. Cutsem (Eds.), Recent advantages in statistics (pp. 133-146). Amsterdam: North Holland.
14. A. P. Dempster N. M. Laird D. B. Rubin 1977 Maximum likelihood from incomplete data via the EM algorithm Journal of the Royal Statistical Society, Series B 39 1 38 501537 0364.62022
15. B. Efron T. Hastie I. Johnstone R. Tibshirani 2004 Least angle regression Annals of Statistics 32 2 407 499 2060166 1091.62054 10.1214/009053604000000067 (Pubitemid 41250302)
16. Fazel, M., Hindi, H., & Boyd, S. (2003). Log-det heuristic for matrix rank minimization with applications to Hankel and Euclidean distance matrices. In: Proc. American control conference, Denver, Colorado.
17. M. A. T. Figueiredo J. M. Bioucas-Dias R. D. Nowak 2007 Majorization-minimization algorithms for wavelet-based image restoration IEEE Transactions on Image Processing 16 2980 2991 2472805 10.1109/TIP.2007.909318 (Pubitemid 350201231)
18. Germann, U. (2001) Aligned Hansards of the 36th parliament of Canada. http://www.isi.edu/natural-language/download/hansard/.
19. T. R. Golub D. K. Slonim P. Tamayo C. Huard M. Gaasenbeek J. Mesirov H. Coller M. K. Loh J. R. Downing M. A. Caligiuri C. D. Bloomfield E. Lander 1999 Molecular classification of cancer: class discovery and class prediction by gene expression monitoring Science 286 531 537 10.1126/science.286.5439.531
20. W. J. Heiser 1987 Correspondence analysis with least absolute residuals Computational Statistics and Data Analysis 5 337 356 0624.62052 10.1016/0167-9473(87)90057-0
21. R. Horst N. V. Thoai 1999 D.c. programming: overview Journal of Optimization Theory and Applications 103 1 43 1715016 10.1023/A:1021765131316
22. H. Hotelling 1933 Analysis of a complex of statistical variables into principal components Journal of Educational Psychology 24 417 441 10.1037/h0071325
23. H. Hotelling 1936 Relations between two sets of variates Biometrika 28 321 377 0015.40705
24. Huber, P. J. (1981). Robust statistics. New York: Wiley.
25. D. R. Hunter K. Lange 2004 A tutorial on MM algorithms The American Statistician 58 30 37 2055509 10.1198/0003130042836 (Pubitemid 38260327)
26. D. R. Hunter R. Li 2005 Variable selection using MM algorithms The Annals of Statistics 33 1617 1642 2166557 1078.62028 10.1214/009053605000000200 (Pubitemid 41423982)
27. J. Jeffers 1967 Two case studies in the application of principal components Applied Statistics 16 225 236 10.2307/2985919
28. Jolliffe, I. (1986). Principal component analysis. New York: Springer.
29. I. T. Jolliffe N. T. Trendafilov M. Uddin 2003 A modified principal component technique based on the LASSO Journal of Computational and Graphical Statistics 12 531 547 2002634 10.1198/1061860032148 (Pubitemid 37239916)
30. M. Journée Y. Nesterov P. Richtárik R. Sepulchre 2010 Generalized power method for sparse principal component analysis Journal of Machine Learning Research 11 517 553
31. K. Lange D. R. Hunter I. Yang 2000 Optimization transfer using surrogate objective functions with discussion Journal of Computational and Graphical Statistics 9 1 1 59 1819865 10.2307/1390605
32. Lee, D. D., & Seung, H. S. (2001). Algorithms for non-negative matrix factorization. In T. Leen, T. Dietterich, & V. Tresp (Eds.), Advances in neural information processing systems (Vol. 13, pp. 556-562). Cambridge: MIT Press.
33. Lemaréchal, C., & Oustry, F. (1999). Semidefinite relaxations and Lagrangian duality with application to combinatorial optimization (Tech. Rep. RR3710). INRIA.
34. Littman, M. L., Dumais, S. T., & Landauer, T. K. (1998). Automatic cross-language information retrieval using latent semantic indexing. In G. Grefenstette (Ed.), Cross-language information retrieval (pp. 51-62). Norwell: Kluwer Academic.
35. Mackey, L. (2009). Deflation methods for sparse PCA. In D. Koller, D. Schuurmans, Y. Bengio, & L. Bottou (Eds.), Advances in neural information processing systems (Vol. 21, pp. 1017-1024). Cambridge: MIT Press.
36. G. McCabe 1984 Principal variables Technometrics 26 137 144 742924 0548.62037 10.2307/1268108
37. Mckinney, M. F. (2003). Features for audio and music classification. In Proc. of the international symposium on music information retrieval (pp. 151-158).
38. X. L. Meng 2000 Discussion on "optimization transfer using surrogate objective functions" Journal of Computational and Graphical Statistics 9 1 35 43 10.2307/1390609
39. Mika, S., Rätsch, G., & Müller, K. R. (2001). A mathematical programming approach to the kernel Fisher algorithm. In T. Leen, T. Dietterich, & V. Tresp (Eds.), Advances in neural information processing systems (Vol. 13). Cambridge: MIT Press.
40. Minoux, M. (1986). Mathematical programming: theory and algorithms. New York: Wiley.
41. Moghaddam, B., Weiss, Y., & Avidan, S. (2007a). Generalized spectral bounds for sparse LDA. In: Proc. of international conference on machine learning.
42. Moghaddam, B., Weiss, Y., & Avidan, S. (2007b). Spectral bounds for sparse PCA: exact and greedy algorithms. In B. Schölkopf, J. Platt, & T. Hoffman (Eds.), Advances in neural information processing systems (Vol. 19). Cambridge: MIT Press.
43. Y. Nesterov 2005 Smooth minimization of non-smooth functions Mathematical Programming, Series A 103 127 152 2166537 1079.90102 10.1007/s10107-004-0552-5 (Pubitemid 40537674)
44. Ortega, J. M., & Rheinboldt, W. C. (1970). Iterative solution of nonlinear equations in several variables. New York: Academic Press.
45. S. Ramaswamy P. Tamayo R. Rifkin S. Mukherjee C. Yeang M. Angelo C. Ladd M. Reich E. Latulippe J. Mesirov T. Poggio W. Gerald M. Loda E. Lander T. Golub 2001 Multiclass cancer diagnosis using tumor gene expression signature Proceedings of the National Academy of Sciences 98 15,149 15,154 10.1073/pnas.211566398 (Pubitemid 34013984)
46. Rockafellar, R. T. (1970). Convex analysis. Princeton: Princeton University Press.
47. Shawe-Taylor, J., & Cristianini, N. (2004). Kernel methods for pattern analysis. Cambridge: Cambridge University Press.
48. Sriperumbudur, B. K., & Lanckriet, G. R. G. (2009). On the convergence of the concave-convex procedure. In Y. Bengio, D. Schuurmans, J. Lafferty, C. K. I. Williams, & A. Culotta (Eds.), Advances in neural information processing systems (Vol. 22, pp. 1759-1767). Cambridge: MIT Press.
49. Sriperumbudur, B. K., Torres, D. A., & Lanckriet, G. R. G. (2007). Sparse eigen methods by d.c. programming. In: Proc. of the 24th annual international conference on machine learning.
50. Sriperumbudur, B. K., Torres, D. A., & Lanckriet, G. R. G. (2009). A d.c. programming approach to the sparse generalized eigenvalue problem. In: Optimization for machine learning workshop, NIPS.
51. Strang, G. (1986). Introduction to applied mathematics. Wellesley: Cambridge Press.
52. Suykens, J. A. K., Gestel, T. V., Brabanter, J. D., Moor, B. D., & Vandewalle, J. (2002). Least squares support vector machines. Singapore: World Scientific.
53. R. Tibshirani 1996 Regression shrinkage and selection via the LASSO Journal of Royal Statistical Society, Series B 58 1 267 288 1379242 0850.62538
54. M. E. Tipping 2001 Sparse Bayesian learning and the relevance vector machine Journal of Machine Learning Research 1 211 244 1875838 0997.68109 (Pubitemid 33687203)
55. Torres, D., Turnbull, D., Barrington, L., & Lanckriet, G. R. G. (2007a). Identifying words that are musically meaningful. In: Proc. of international symposium on music information and retrieval.
56. Torres, D. A., Turnbull, D., Sriperumbudur, B. K., Barrington, L., & Lanckriet, G. R. G. (2007b). Finding musically meaningful words using sparse CCA. In: Music brain & cognition workshop, NIPS.
57. D. Turnbull L. Barrington D. Torres G. R. G. Lanckriet 2008 Semantic annotation and retrieval of music and sound effects IEEE Transactions on Audio, Speech and Language Processing 16 467 476 10.1109/TASL.2007.913750
58. L. Vandenberghe S. Boyd 1996 Semidefinite programming SIAM Review 38 49 95 1379041 0845.65023 10.1137/1038003 (Pubitemid 126610912)
59. Vinokourov, A., Shawe-Taylor, J., & Cristianini, N. (2003). Inferring a semantic representation of text via cross-language correlation analysis. In S. Becker, S. Thrun, & K. Obermayer (Eds.), Advances in neural information processing systems (Vol. 15, pp. 1473-1480). Cambridge: MIT Press.
60. J. Weston A. Elisseeff B. Schölkopf M. Tipping 2003 Use of the zero-norm with linear models and kernel methods Journal of Machine Learning Research 3 1439 1461 1102.68605
61. A. L. Yuille A. Rangarajan 2003 The concave-convex procedure Neural Computation 15 915 936 1022.68112 10.1162/08997660360581958 (Pubitemid 37049812)
62. Zangwill, W. I. (1969). Nonlinear programming: a unified approach. Englewood Cliffs: Prentice-Hall.
63. H. Zou T. Hastie 2005 Regularization and variable selection via the elastic net Journal of the Royal Statistical Society, Series B 67 301 320 2137327 1069.62054 10.1111/j.1467-9868.2005.00503.x (Pubitemid 40465877)
64. H. Zou T. Hastie R. Tibshirani 2006 Sparse principal component analysis Journal of Computational and Graphical Statistics 15 265 286 2252527 10.1198/106186006X113430 (Pubitemid 43935978)