Linear dimensionality reduction: Survey, insights, and generalizations

Journal of Machine Learning Research

Volumn 16, Issue , 2015, Pages 2859-2900

  Cunningham, John P ^a   Ghahramani, Zoubin ^b  

^a Columbia University ^*  (United States)

^b UNIVERSITY OF CAMBRIDGE   (United Kingdom)

Author keywords

Dimensionality reduction; Eigenvector problems; Matrix manifolds

Indexed keywords

CLUSTERING ALGORITHMS; DATA REDUCTION; DISCRIMINANT ANALYSIS; EIGENVALUES AND EIGENFUNCTIONS; FACE RECOGNITION; INDEPENDENT COMPONENT ANALYSIS; PRINCIPAL COMPONENT ANALYSIS; SURVEYS;

AUTO-CORRELATION FACTORS; CANONICAL CORRELATIONS ANALYSIS; DIMENSIONALITY REDUCTION; FISHER'S LINEAR DISCRIMINANT ANALYSIS; GEOMETRIC INTERPRETATION; LINEAR DIMENSIONALITY REDUCTION; MATRIX MANIFOLDS; MULTI-DIMENSIONAL SCALING;

MATRIX ALGEBRA;

EID: 84961575191     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (133)

1. T. E. Abrudan, J. Eriksson, and V. Koivunen. Steepest descent algorithms for optimization under unitary matrix constraint. IEEE Transactions on Signal Processing, 56:1134-1147, 2008.
2. P. Absil and J. Malick. Projection-like retractions on matrix manifolds. SIAM Journal on Optimization, 22(1):135-158, 2012.
3. P. A. Absil, R. Mahony, and R. Sepulchre. Optimization Algorithms on Matrix Manifolds. Princeton University Press, Princeton, 2008.
4. K. P. Adragni and D. Cook. Sufficient dimension reduction and prediction in regression. Philosophical Transactions of the Royal Society A, 367:4385-4405, 2009.
5. S. Amari. Natural gradient learning for over and under complete bases in ICA. Neural Computation, 11:1875-1883, 1999.
6. A. Baccini, P. Besse, and A. de Faguerolles. A L1-norm PCA and heuristic approach. In Proceedings of the International Conference on Ordinal and Symbolic Data Analysis, pages 359-368, 1996.
7. F. Bach, R. Jenatton, J. Mairal, and G. Obozinski. Convex optimization with sparsityinducing norms. Optimization for Machine Learning, pages 19-53, 2011.
8. G. H. Bakir, A. Gretton, M. Franz, and B. Schölkopf. Multivariate regression via Stiefel manifold constraints. In Pattern Recognition, pages 262-269. Springer, 2004.
9. A. Bar-Hillel, T. Hertz, N. Shental, and D. Weinshall. Learning distance functions using equivalence relations. In Proceedings of the International Conference on Machine Learning, volume 3, pages 11-18, 2003.
10. M. Belkin and P. Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation, 15(6):1373-1396, 2003.
11. D. P. Bertsekas. On the goldstein-levitin-polyak gradient projection method. IEEE Transactions on Automatic Control, 21(2):174-184, 1976.
12. C. M. Bishop. Pattern Recognition and Machine Learning. Springer, New York, 2006.
13. I. Borg and P. J. Groenen. Modern Multidimensional Scaling: Theory and Applications. Springer Verlag, 2005.
14. M. Borga, T. Landelius, and H. Knutsson. A unified approach to PCA, PLS, MLR, and CCA. Technical Report, 1997.
15. N. Boumal, B. Mishra, P. Absil, and R. Sepulchre. Manopt, a matlab toolbox for optimization on manifolds. Journal of Machine Learning Research, 15:1455-1459, 2014.
16. S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations & Trends in Machine Learning, 3(1):1-122, 2011.
17. A. Bray and D. Martinez. Kernel-based extraction of slow features: Complex cells learn disparity and translation invariance from natural images. In Advances in Neural Information Processing Systems, pages 253-260, 2002.
18. W. Brendel, R. Romo, and C. K. Machens. Demixed principal component analysis. In Advances in Neural Information Processing Systems, pages 2654-2662, 2011.
19. W. Buntine. Variational extensions to EM and multinomial PCA. In Proceedings of the European Conference on Machine Learning, 2002.
20. C. J. C. Burges. Dimension reduction: a guided tour. Foundations & Trends in Machine Learning, 2(4):275-365, 2010.
21. E. J. Candes, X. Li, Y. Ma, and J. Wright. Robust principal component analysis? Journal of the ACM, 58(3):11:1-11:37, 2011.
22. G. Chechik, U. Shalit, V. Sharma, and S. Bengio. An online algorithm for large scale image similarity learning. In Advances in Neural Information Processing Systems, pages 306-314, 2009.
23. V. Choulakian. L1-norm projection pursuit principal component analysis. Computational Statistics and Data Analysis, 50(6):1441-1451, 2006.
24. M. M. Churchland, J. P. Cunningham, M. T. Kaufman, J. D. Foster, P. Nuyujukian, S. I. Ryu, and K. V. Shenoy. Neural population dynamics during reaching. Nature, 487:51-56, 2012.
25. R. R. Coifman and S. Lafon. Diffusion maps. Applied and Computational Harmonic Analysis, 21(1):5-30, 2006.
26. M. Collins, S. Dasgupta, and R. E. Schapire. A generalization of principal component analysis to the exponential family. In Advances in Neural Information Processing Systems, 2002.
27. T. F. Cox and M. A. Cox. Multidimensional scaling, volume 88. CRC Press, 2001.
28. J. P. Cunningham and B. M. Yu. Dimensionality reduction for large-scale neural recordings. Nature Neuroscience, 17:1500-1509, 2014.
29. A d'Aspremont, L. El Ghaoui, M. I. Jordan, and G. R. Lanckriet. A direct formulation for sparse PCA using semidefinite programming. SIAM Review, 49:434-448, 2007.
30. A d'Aspremont, F. R. Bach, and L. El Ghaoui. Optimal solutions for sparse principal component analysis. Journal of Machine Learning Research, 9:1269-1294, 2008.
31. F. De la Torre. A least-squares framework for component analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(6):1041-1055, 2012.
32. D. De Ridder, R. P. W. Duin, and J. Kittler. Texture description by independent components. In Structural, Syntactic, and Statistical Pattern Recognition, pages 587-596. Springer, 2002.
33. S. C. Deerwester, S. T. Dumais, T. K. Landauer, G. W. Furnas, and R. A. Harshman. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6):391-407, 1990.
34. A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm (with discussion). Journal of the Royal Statistical Society, Series B, 39:1-38, 1977.
35. M. der and L. K. Saul. Latent coincidence analysis: a hidden variable model for distance metric learning. In Advances in Neural Information Processing Systems, pages 3230-3238, 2012.
36. P. Diaconis and D. Freedman. Asymptotics of graphical projection pursuit. The Annals of Statistics, pages 793-815, 1984.
37. C. Eckart and G. Young. The approximation of one matrix by another of lower rank. Psychometrika, 1:211-218, 1936.
38. A. Edelman, T. A. Arias, and S. T. Smith. The geometry of algorithms with orthogonality constraints. SIAM Journal of Matrix Analysis and Applications, 1998.
39. K. Fan and A. J. Hoffman. Some metric inequalities in the space of matrices. Proceedings of the American Mathematical Society, 6:111-116, 1955.
40. S. Fiori. Quasi-geodesic neural learning algorithms over the orthogonal group: a tutorial. Journal of Machine Learning Research, 6:743-781, 2005.
41. R. A. Fisher. The use of multiple measurements in taxonomic problems. Annals of Eugenics, 7(2):179-188, 1936.
42. J. H. Friedman. Exploratory projection pursuit. Journal of the American Statistical Association, 82(397):249-266, 1987.
43. K. Fukumizu, F. R. Bach, and M. I. Jordan. Dimensionality reduction for supervised learning with reproducing kernel hilbert spaces. Journal of Machine Learning Research, 5:73-99, 2004.
44. K. Fukumizu, F. R. Bach, and M. I. Jordan. Kernel dimension reduction in regression. The Annals of Statistics, pages 1871-1905, 2009.
45. K. Fukunaga. Introduction to Statistical Pattern Recognition. Academic press, 1990.
46. D. Gabay. Minimizing a differentiable function over a differentiable manifold. The Journal of Optimization Theory and Applications, 37(2):177-219, 1982.
47. J. S. Galpin and D. M. Hawkins. Methods of L1 estimation of a covariance matrix. Computational Statistics and Data Analysis, 5:305-319, 1987.
48. A. Globerson and S. T. Roweis. Metric learning by collapsing classes. In Advances in Neural Information Processing Systems, pages 451-458, 2005.
49. J. Goldberger, S. T. Roweis, G. Hinton, and R. Salakhutdinov. Neighbourhood components analysis. In Advances in Neural Information Processing Systems, 2004.
50. G. H. Golub and C. F. Van Loan. Matrix Computations, 3rd edition. Hopkins University Press, Baltimore, 1996.
51. A. Gretton, O. Bousquet, A. Smola, and B. Schölkopf. Measuring statistical dependence with Hilbert-Schmidt norms. In Algorithmic Learning Theory, pages 63-77. Springer, 2005.
52. A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A kernel twosample test. Journal of Machine Learning Research, 13(1):723-773, 2012.
53. D. R. Hardoon and J. Shawe-Taylor. Convergence analysis of kernel canonical correlation analysis: theory and practice. Machine Learning, 74(1):23-38, 2009.
54. D. R. Hardoon, S. Szedmak, and J Shawe-Taylor. Canonical correlation analysis: an overview with application to learning methods. Neural Computation, 16(12):2639-2664, 2004.
55. T. Hastie, R. Tibshirani, and J. Friedman. Elements of Statistical Learning, 2nd Edition. Cambridge University Press, Cambridge, UK, 2008.
56. X. He and P. Niyogi. Locality preserving projections. In Advances in Neural Information Processing Systems, volume 16, page 153, 2004.
57. X. He, D. Cai, S. Yan, and H. Zhang. Neighborhood preserving embedding. In IEEE International Conference on Computer Vision, volume 2, pages 1208-1213, 2005.
58. N. J. Higham. Matrix nearness problems and applications. In Applications of Matrix Theory, pages 1-27. Oxford University Press, 1989.
59. H. Hotelling. Relations between two sets of variates. Biometrika, 28:321-377, 1936.
60. P. J. Huber. Projection pursuit. The Annals of Statistics, pages 435-475, 1985.
61. A. Hyvarinen, J. Karhunen, and E. Oja. Independent Component Analysis. John Wiley and Sons, 2001.
62. M. Joho, H. Mathis, and R. H. Lambert. Overdetermined blind source separation: Using more sensors than source signals in a noisy mixture. In Independent Component Analysis and Blind Signal Separation, pages 81-86, 2000.
63. M. Journee, Y. Nesterov, P. Richtarik, and R. Sepulchre. Generalized power method for sparse principal component analysis. Journal of Machine Learning Research, 11:517-553, 2010.
64. R. E. Kalman. A new approach to linear filtering and prediction problems. The Journal of Basic Engineering, 82:35-45, 1960.
65. T. Kaneko, S. Fiori, and T. Tanaka. Empirical arithmetic averaging over the compact stiefel manifold. IEEE Transactions on Signal Processing, 61(4):883-894, 2013.
66. Y. H. Kao and B. Van Roy. Learning a factor model via regularized PCA. Machine Learning, 91(279-303), 2013.
67. B. Kulis. Metric learning: A survey. Foundations & Trends in Machine Learning, 5(4):287-364, 2012.
68. R. Larsen. Decomposition using maximum autocorrelation factors. Journal of Chemometrics, 16:427-435, 2002.
69. N. D. Lawrence. A unifying probabilistic perspective for spectral dimensionality reduction: insights and new models. Journal of Machine Learning Research, 13(1):1609-1638, 2012.
70. D. D. Lee and H. S. Seung. Learning the parts of objects by non-negative matrix factorization. Nature, 401:788-791, 1999.
71. J. F. Li and X. Y. Hu. Procrustes problems and associated approximation problems for matrices with k-involutory symmetries. Linear Algebra and its Applications, 434:820-829, 2011.
72. K. C. Li. Sliced inverse regression for dimension reduction. Journal of the American Statistical Association, 86(414):316-327, 1991.
73. D. Luenberger. The gradient projection method along geodesics. Management Science, 18(11), 1972.
74. J. H. Manton. Optimization algorithms exploiting unitary constraints. IEEE Transactions on Signal Processing, 50:635-650, 2002.
75. J. H. Manton. On the various generalizations of optimization algorithms to manifolds. Proceedings of Mathematical Theory of Network and Systems, 2004.
76. K. V. Mardia, J. T. Kent, and J. M. Bibby. Multivariate Analysis. Probability and mathematical statistics. Academic Press, 1979.
77. S. Mika, G. Rätsch, J. Weston, B. Schölkopf, and K. R. Müller. Fisher discriminant analysis with kernels. In Proceedings of the IEEE Signal Processing Society, pages 41-48, 1999.
78. L. Mirsky. Symmetric gauge functions and unitarily invariant norms. Quarterly Journal of Mathematics, 11:80-89, 1960.
79. A. Mnih and R. Salakhutdinov. Probabilistic matrix factorization. In Advances in Neural Information Processing Systems, pages 1257-1264, 2007.
80. S. Mohamed, K. Heller, and Z. Ghahramani. Bayesian exponential family PCA. In Advances in Neural Information Processing Systems, 2008.
81. R. J. Muirhead. Aspects of Multivariate Statistical Theory, 2nd Edition. Wiley, 2005.
82. J. Nilsson, F. Sha, and M. I. Jordan. Regression on manifolds using kernel dimension reduction. In Proceedings of the International Conference on Machine Learning, pages 697-704, 2007.
83. Y. Nishimori and S. Akaho. Learning algorithms utilizing quasi-geodesic flows on the Stiefel manifold. Neurocomputing, 67:106-135, 2005.
84. K. Pearson. On lines and planes of closest fit to systems of points in space. Philosophical Magazine, 2:559-572, 1901.
85. J. Peltonen, J. Goldberger, and S. Kaski. Fast semi-supervised discriminative component analysis. In Proceeding of the IEEE Workshop on Machine Learning for Signal Processing, pages 312-317. IEEE, 2007.
86. C. R. Rao. The utilization of multiple measurements in problems of biological classification. Journal of the Royal Statistical Society, Series B, 10(2):159-203, 1948.
87. C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning. MIT Press, Cambridge, 2006.
88. J. D. Rennie and N. Srebro. Fast maximum margin matrix factorization for collaborative prediction. In Proceedings of the International Conference on Machine Learning, pages 713-719, 2005.
89. S. T. Roweis. EM algorithms for PCA and sensible PCA. In Advances in Neural Information Processing Systems, 1997.
90. S. T. Roweis and L. K. Saul. Nonlinear dimensionality reduction by locally linear embedding. Science, 290(5500):2323-2326, December 2000.
91. E. Rubinshtein and A. Srivastava. Optimal linear projections for enhancing desired data statistics. Statistical Computing, 20:267-282, 2010.
92. A. Ruhe. Closest normal matrix finally found! Technical Report, University of Goteberg, 1987.
93. B. Schölkopf, A. Smola, and R. K. Muller. Kernel principal component analysis. Advances in Kernel Methods: Support Vector Learning, pages 327-352, 1999.
94. P. H. Schonemann. A generalized solution of the orthogonal Procrustes problem. Psychometrika, 31:1-10, 1966.
95. C. Shen, H. Li, and M. J. Brooks. A convex programming approach to the trace quotient problem. In Proceedings of the Asian Conference on Computer Vision, pages 227-235. Springer, 2007.
96. C. Spearman. General intelligence, objectively determined and measured. American Journal of Psychology, 15:201-293, 1904.
97. N. Srebro and T. S. Jaakkola. Weighted low-rank approximations. Proceedings of the International Conference on Machine Learning, pages 720-727, 2003.
98. N. Srebro, J. Rennie, and T. S. Jaakkola. Maximum-margin matrix factorization. In Advances in Neural Information Processing Systems, pages 1329-1336, 2004.
99. A. Srivastava and X. Liu. Tools for application-driven linear dimension reduction. Neurocomputing, 67:136-160, 2005.
100. J. V. Stone and J. Porrill. Undercomplete independent component analysis for signal separation and dimension reduction. Technical Report, 1998.
101. M. Stone and R. J. Brooks. Continuum regression: cross-validated sequentially constructed prediction embracing ordinary least squares, partial least squares and principal components regression. Journal of the Royal Statistical Society, Series B, pages 237-269, 1990.
102. L. Sun, S. Ji, and J. Ye. A least squares formulation for a class of generalized eigenvalue problems in machine learning. In Proceedings of the International Conference on Machine Learning, pages 977-984. ACM, 2009.
103. P. Switzer and A. A. Green. Min/max autocorrelation factors for multivariate spatial imagery. Technical Report, Stanford University, 1984.
104. J. B. Tenenbaum, V. deSilva, and J. C. Langford. A global geometric framework for nonlinear dimensionality reduction. Science, 290(5500):2319-2323, December 2000.
105. C. M. Theobald. An inequality with application to multivariate analysis. Biometrika, 62(2):461-466, 1975.
106. R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B, 58:267-288, 1996.
107. N. H. Timm. Applied Multivariate Analysis. Springer, 2002.
108. M. E. Tipping and C. M. Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Society, Series B, 61(3):611-622, 1999.
109. W. S. Torgerson. Multidimensional scaling: I. theory and method. Psychometrika, 17(4):401-419, 1952.
110. L. Torresani and K. Lee. Large margin component analysis. In Advances in Neural Information Processing Systems, pages 1385-1392, 2006.
111. R. Turner and M. Sahani. A maximum-likelihood interpretation for slow feature analysis. Neural Computation, 19(4):1022-1038, 2007.
112. M. O. Ulfarsson and V. Solo. Sparse variable PCA using geodesic steepest descent. IEEE Transactions on Signal Processing, 56:5823-5832, 2008.
113. L. J. P. Van der Maaten, E. O. Postma, and H. J. Van den Herik. Dimensionality reduction: A comparative review. Tilburg University Technical Report, TiCC-TR 2009-005, 2009.
114. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9(2579-2605):85, 2008.
115. K. R. Varshney and A. S. Willsky. Linear dimensionality reduction for margin-based classification: high-dimensional data and sensor networks. IEEE Transactions on Signal Processing, 59:2496-2512, 2011.
116. M. Wang, F. Sha, and M. I. Jordan. Unsupervised kernel dimension reduction. In Advances in Neural Information Processing Systems, pages 2379-2387, 2010.
117. K. Q. Weinberger and L. K. Saul. Unsupervised learning of image manifolds by semidefinite programming. International Journal of Computer Vision, 70(1):77-90, 2006.
118. K. Q. Weinberger and L. K. Saul. Distance metric learning for large margin nearest neighbor classification. Journal of Machine Learning Research, 10:207-244, 2009.
119. K. Q. Weinberger, J. Blitzer, and L. K. Saul. Distance metric learning for large margin nearest neighbor classification. In Advances in Neural Information Processing Systems, pages 1473-1480, 2005.
120. M. Welling, F. Agakov, and C. K. I. Williams. Extreme component analysis. In Advances in Neural Information Processing Systems, 2003.
121. M. Welling, R. S. Zemel, and G. E. Hinton. Probabilistic sequential independent components analysis. In IEEE Transactions on Neural Networks, 2004.
122. C. K. I. Williams. On a connection between kernel PCA and metric multidimensional scaling. Machine Learning, 46(1-3):11-19, 2002.
123. C. K. I. Williams and F. Agakov. Products of Gaussians and probabilistic minor component analysis. Neural Computation, 14(5):1169-1182, 2002.
124. L. Wiskott. Slow feature analysis: A theoretical analysis of optimal free responses. Neural Computation, 15(9):2147-2177, 2003.
125. L. Wiskott and T. Sejnowski. Slow feature analysis: unsupervised learning of invariances. Neural Computation, 14(4):715-770, 2002.
126. E. P. Xing, M. I. Jordan, S. Russell, and A. Y. Ng. Distance metric learning with application to clustering with side-information. In Advances in Neural Information Processing Systems, pages 505-512, 2002.
127. S. Yan and X. Tang. Trace quotient problems revisited. In Proceedings of the European Conference on Computer Vision, pages 232-244. Springer, 2006.
128. L. Yang. An overview of distance metric learning. Proceedings of Computer Vision and Pattern Recognition, 7, 2007.
129. L. Yang and R. Jin. Distance metric learning: A comprehensive survey. Michigan State Universiy Technical Report, 2006.
130. S. Yu, K. Yu, V. Tresp, H. P. Kriegel, and M. Wu. Supervised probabilistic principal component analysis. In Proceedings of the International Conference on Knowledge Discovery and Data Mining, pages 464-473, 2006.
131. L. Zhang, A. Cichocki, and S. Amari. Natural gradient algorithm for blind separation of overdetermined mixture with additive noise. IEEE Signal Processing Letters, 6(11):293-295, 1999.
132. D. Zhao, Z. Lin, and X. Tang. Laplacian PCA and its applications. In Proceedings of the International Conference on Computer Vision, pages 1-8, 2007.
133. H. Zou, T. Hastie, and R. Tibshirani. Sparse principal component analysis. Journal of Computational and Graphical Statistics, 15(2):265-286, 2006.