Alternating direction method of multipliers for penalized zero-variance discriminant analysis

Computational Optimization and Applications

Volumn 64, Issue 3, 2016, Pages 725-754

  Ames, Brendan P W ^a   Hong, Mingyi ^b  

^a UNIVERSITY OF ALABAMA   (United States)

^b Iowa State University   (United States)

Author keywords

Alternating direction method of multipliers; Classification; Dimension reduction; Feature selection; Linear discriminant analysis; Nonconvex optimization

Indexed keywords

CLUSTERING ALGORITHMS; COVARIANCE MATRIX; DATA MINING; DISCRIMINANT ANALYSIS; EIGENVALUES AND EIGENFUNCTIONS; FEATURE EXTRACTION; HARMONIC ANALYSIS; OPTIMIZATION; PRINCIPAL COMPONENT ANALYSIS; VECTOR SPACES;

ALTERNATING DIRECTION METHOD OF MULTIPLIERS; DIMENSION REDUCTION; GENERALIZED EIGENVALUE PROBLEMS; LINEAR DISCRIMINANT ANALYSIS; NONCONVEX OPTIMIZATION; SPARSE PRINCIPAL COMPONENT ANALYSIS; TIME SERIES CLASSIFICATIONS; WITHIN-CLASS COVARIANCE;

CLASSIFICATION (OF INFORMATION);

EID: 84957951366     PISSN: 09266003     EISSN: 15732894     Source Type: Journal    
DOI: 10.1007/s10589-016-9828-y     Document Type: Article

References (53)

1. Allen, G.I.: Sparse and functional principal components analysis. arXiv:1309.2895 (2013)
2. Ames, B., Hong, M.: SZVD: ADMM for sparse zero-variance discriminant analysis. http://bpames.people.ua.edu/software.html (2014)
3. Bach, F., Jenatton, R., Mairal, J., Obozinski, G.: Structured sparsity through convex optimization. Stat. Sci. 27(4), 450–468 (2012)
4. Baik, J., Silverstein, J.W.: Eigenvalues of large sample covariance matrices of spiked population models. J. Multivar. Anal. 97(6), 1382–1408 (2006)
5. Bickel, P., Levina, E.: Some theory for Fisher’s linear discriminant function, naive Bayes’, and some alternatives when there are many more variables than observations. Bernoulli 10(6), 989–1010 (2004)
6. Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends ® Mach. Learn. 3(1), 1–122 (2011)
7. Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004)
8. Boyd, S., Vandenberghe, L.: Subgradients. Lecture Notes for EE364b, Stanford University, Winter 2006-07 (2008). http://see.stanford.edu/materials/lsocoee364b/01-subgradients_notes.pdf
9. Briandet, R., Kemsley, E.K., Wilson, R.H.: Discrimination of Arabica and Robusta in instant coffee by Fourier transform infrared spectroscopy and chemometrics. J. Agric. Food Chem. 44(1), 170–174 (1996)
10. Cai, T., Liu, W.: A direct estimation approach to sparse linear discriminant analysis. J. Am. Stat. Assoc. 106(496), 1566–1577 (2011)
11. Candès, E.J.: Compressive sampling. In: Proceedings oh the International Congress of Mathematicians: Madrid, August 22–30, 2006: invited lectures, pages pp. 1433–1452 (2006)
12. Clemmensen, L.: On discriminant analysis techniques and correlation structures in high dimensions. Technical Report-2013. Technical University of Denmark (2013)
13. Clemmensen, L.: contributions by Max Kuhn. sparseLDA: Sparse Discriminant Analysis. R package version 0.1-6 (2012)
14. Clemmensen, L., Hastie, T., Witten, D., Ersbøll, B.: Sparse discriminant analysis. Technometrics, 53(4), 406–413 (2011)
15. d’Aspremont, A., Bach, F., El Ghaoui, L.: Optimal solutions for sparse principal component analysis. J. Mach. Learn. Res. 9, 1269–1294 (2008)
16. d’Aspremont, A., Bach, F., El Ghaoui, L.: Approximation bounds for sparse principal component analysis. Math. Program. 148(1–2), 89–110 (2014)
17. d’Aspremont, A., El Ghaoui, L., Jordan, M.I., Lanckriet, G.R.: A direct formulation for sparse PCA using semidefinite programming. SIAM Rev. 49(3), 434–448 (2007)
18. Dudoit, S., Fridlyand, J., Speed, T.P.: Comparison of discrimination methods for the classification of tumors using gene expression data. J. Am. Stat. Assoc. 97(457), 77–87 (2002)
19. Tebbens, J.Duintjer, Schlesinger, P.: Improving implementation of linear discriminant analysis for the high dimension/small sample size problem. Comput. Stat. Data Anal. 52(1), 423–437 (2007)
20. Eckstein, J., Bertsekas, D.P.: On the Douglas-Rachford splitting method and the proximal point algorithm for maximal monotone operators. Math. Program. 55(1–3), 293–318 (1992)
21. Fisher, R.A.: The use of multiple measurements in taxonomic problems. Ann. Eugen. 7(2), 179–188 (1936)
22. Friedman, J.H.: Regularized discriminant analysis. J. Am. Stat. Assoc. 84(405), 165–175 (1989)
23. Golub, G.H., Van Loan, C.F.: Matrix Computations. Johns Hopkins University Press, Baltimore (1996)
24. Guo, Y., Hastie, T., Tibshirani, R.: Regularized linear discriminant analysis and its application in microarrays. Biostatistics 8(1), 86–100 (2007)
25. Hastie, T., Buja, A., Tibshirani, R.: Penalized discriminant analysis. Ann. Stat. 73–102 (1995)
26. Hastie, T., Tibshirani, R., Friedman, J.J.H.: The Elements of Statistical Learning. Springer, New York (2013)
27. Hong, M., Luo, Z.-Q.: On the linear convergence of the alternating direction method of multipliers. arXiv:1208.3922, preprint (2012)
28. Hong, M., Luo, Z.-Q., Razaviyayn, M.: Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems. arXiv:1410.1390, preprint (2012)
29. Johnstone, I.M., Lu, A.Y.: Sparse principal components analysis. Unpublished manuscript (2004)
30. Jolliffe, I.T., Trendafilov, N.T., Uddin, M.: A modified principal component technique based on the lasso. J. Comput. Graph. Stat. 12(3), 531–547 (2003)
31. Journée, M., Nesterov, Y., Richtárik, P., Sepulchre, R.: Generalized power method for sparse principal component analysis. J. Mach. Learn. Res. 11, 517–553 (2010)
32. Keogh, E., Xi, X., Wei, L., Ratanamahatana, C.A.: The UCR time series classification/clustering homepage (2006)
33. Krzanowski, W., Jonathan, P., McCarthy, W., Thomas, M.: Discriminant analysis with singular covariance matrices: methods and applications to spectroscopic data. Appl. Stat. 101–115 (1995)
34. Kutyniok, G.: Compressed sensing: Theory and applications. CoRR, arXiv:1203.3815 (2012)
35. Luss, R., Teboulle, M.: Convex approximations to sparse PCA via lagrangian duality. Oper. Res. Lett. 39(1), 57–61 (2011)
36. Ma, S.: Alternating direction method of multipliers for sparse principal component analysis. J. Oper. Res. Soc. China 1(2), 253–274 (2013)
37. Papailiopoulos, D.S., Dimakis, A.G., Korokythakis, S.: Sparse PCA through low-rank approximations. arXiv:1303.0551, preprint (2013)
38. Paul, D.: Asymptotics of sample eigenstructure for a large dimensional spiked covariance model. Statistica Sinica 17(4), 1617 (2007)
39. Richtárik, P., Takáč, M., Ahipaşaoğlu, S.D.: Alternating maximization: Unifying framework for 8 sparse PCA formulations and efficient parallel codes. arXiv:1212.4137, preprint (2012)
40. Shao, J., Wang, Y., Deng, X., Wang, S.: Sparse linear discriminant analysis by thresholding for high dimensional data. Ann. Stat. 39(2), 1241–1265 (2011)
41. Tapp, H.S., Defernez, M., Kemsley, E.K.: Ftir spectroscopy and multivariate analysis can distinguish the geographic origin of extra virgin olive oils. J. Agric. Food Chem. 51(21), 6110–6115 (2003)
42. Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B 58(1), 267–288 (1996)
43. Tibshirani, R., Hastie, T., Narasimhan, B., Chu, G.: Class prediction by nearest shrunken centroids, with applications to dna microarrays. Stat. Sci. 104–117 (2003)
44. Welling, M.: Fisher linear discriminant analysis. Department of Computer Science, University of Toronto (2005)
45. Witten, D.: penalizedLDA: Penalized classification using Fisher’s linear discriminant, (2011). R package version 1.0
46. Witten, D.M., Tibshirani, R.: Penalized classification using fisher’s linear discriminant. J. R. Stat. Soc. Ser. B 73(5), 753–772 (2011)
47. Witten, D.M., Tibshirani, R., Hastie, T.: A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis. Biostatistics 10(3), 515–534 (2009)
48. Xu, P., Brock, G.N., Parrish, R.S.: Modified linear discriminant analysis approaches for classification of high-dimensional microarray data. Comput. Stat. Data Anal. 53(5), 1674–1687 (2009)
49. Yuan, M., Lin, Y.: Model selection and estimation in regression with grouped variables. J. R. Stat. Soc. Ser. B 68(1), 49–67 (2006)
50. Yuan, X.-T., Zhang, T.: Truncated power method for sparse eigenvalue problems. J. Mach. Learn. Res. 14(1), 899–925 (2013)
51. Zhang, Y., d’Aspremont, A., Ghaoui, L.E.I.: Sparse PCA: Convex relaxations, algorithms and applications. Handbook on Semidefinite, Conic and Polynomial Optimization, pp. 915–940. Springer, Berlin (2012)
52. Zou, H., Hastie, T.: Regularization and variable selection via the elastic net. J. R. Stat. Soc. Ser. B 67(2), 301–320 (2005)
53. Zou, H., Hastie, T., Tibshirani, R.: Sparse principal component analysis. J. Comput. Graph. Stat. 15(2), 265–286 (2006)