Trace optimization and eigenproblems in dimension reduction methods

Numerical Linear Algebra with Applications

Volumn 18, Issue 3, 2011, Pages 565-602

  Kokiopoulou, E ^a   Chen, J ^b   Saad, Y ^b  

^a ETH ZURICH   (Switzerland)

^b University of Minnesota   (United States)

Author keywords

Kernel methods; Laplacean eigenmaps; Linear dimension reduction; Locality preserving projections (LPP); Locally linear embedding (LLE); Nonlinear dimension reduction; Principal component analysis; Projection methods

Indexed keywords

EID: 79954475526     PISSN: 10705325     EISSN: 10991506     Source Type: Journal    
DOI: 10.1002/nla.743     Document Type: Article

References (69)

1. Webb A. Statistical Pattern Recognition (2nd edn). Wiley: Hoboken, NJ, 2002.
2. Koren Y. On spectral graph drawing. COCOON 03, LNCS, vol. 2697. Springer: Berlin, 2003; 496-508.
3. Noack A. An energy model for visual graph clustering. Proceedings of the 11th International Symposium on Graph Drawing (GD 2003), LNCS 2912. Springer: Berlin, 2004; 425-436.
4. Curtarolo S, Morgan D, Persson K, Rodgers J, Ceder G. Predicting crystal structures with data mining of quantum calculations. Physical Review Letters 2003; 91(13):135503.
5. Ceder G, Morgan D, Fischer C, Tibbetts K, Curtarolo S. Data-mining-driven quantum mechanics for the prediction of structure. MRS Bulletin 2006; 31:981-985.
6. Sebastian SE, Harrison N, Batista CD, Balicas L, Jaime M, Sharma PA, Kawashima N, Fisher IR. Dimensional reduction at a quantum critical point. Nature 2006; 441:617.
7. http://www.cs.toronto.edu/~roweis/data.html.
8. Bishop CM. Pattern Recognition and Machine Learning. Information Science and Statistics. Springer: Berlin, 2006.
9. Roweis S, Saul L. Nonlinear dimensionality reduction by locally linear embedding. Science 2000; 290:2323-2326.
10. John Lee A, Verleysen M. Nonlinear Dimensionality Reduction. Information Science and Statistics. Springer: Berlin, 2007.
11. Belkin M, Niyogi P. Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation 2003; 15(6):1373-1396.
12. Zhang Z, Zha H. Principal manifolds and nonlinear dimensionality reduction via tangent space alignment. SIAM Journal on Scientific Computing 2005; 26(1):313-338.
13. Tenenbaum JB, de Silva V, Langford JC. A global geometric framework for nonlinear dimensionality reduction. Science 2000; 290:2319-2323.
14. Weinberger KQ, Saul LK. Unsupervised learning of image manifolds by semidefinite programming. Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), Washington, DC, U.S.A., 2004.
15. Sha F, Saul LK. Analysis and extension of spectral methods for nonlinear dimensionality reduction. Proceedings of the 22nd International Conference on Machine Learning (ICML), Bonn, Germany, 2005.
16. Kokiopoulou E, Saad Y. Orthogonal neighborhood preserving projections. In IEEE 5th International Conference on Data Mining (ICDM05), Houston, TX, 27-30 November Han J et al. (eds). IEEE: New York, 2005; 234-241.
17. Ham J, Lee DD, Mika S, Schölkopf B. A kernel view of the dimensionality reduction of manifolds. ICML '04: Proceedings of the 21st International Conference on Machine Learning. ACM: New York, NY, U.S.A., 2004; 47.
18. Williams CKI. On a connection between kernel PCA and metric multidimensional scaling. Machine Learning 2002; 46(1-3):11-19.
19. Parlett BN. The Symmetric Eigenvalue Problem. Number 20 in Classics in Applied Mathematics. SIAM: Philadelphia, 1998.
20. Saad Y. Numerical Methods for Large Eigenvalue Problems. Halstead Press: New York, 1992.
21. Guo Y-F, Li S-J, Yang J-Y, Shu T-T, Wu L-D. A generalized Foley-Sammon transform based on generalized Fisher discriminant criterion and its application to face recognition. Pattern Recognition Letters 2003; 24(1-3):147-158.
22. Wang H, Yan SC, Xu D, Tang XO, Huang T. Trace ratio vs. ratio trace for dimensionality reduction. IEEE Conference on Computer Vision and Pattern Recognition, 2007; 17-22.
23. Xiang S, Nie F, Zhang C. Learning a mahalanobis distance metric for data clustering and classification. Pattern Recognition 2008; 41(12):3600-3612.
24. Yan S, Tang X. Trace quotient problems revisited. In Proceedings of the European Conference on Computer Vision (Lecture Notes in Computer Science, vol. 2), Leonardis A, Bischof H, Pinz A (eds). Springer: Berlin-Heidelberg, 2006; 232-244, Number 3952.
25. Nie F, Xiang S, Jia Y, Zhang C. Semi-supervised orthogonal discriminant analysis via label propagation. Pattern Recognition 2009; 42:2615-2627.
26. Saul L, Roweis S. Think globally, fit locally: unsupervised learning of nonlinear manifolds. Journal of Machine Learning Research 2003; 4:119-155.
27. Belkin M, Niyogi P. Laplacian eigenmaps and spectral techniques for embedding and clustering. Advances in Neural Information Processing Systems 14. MIT Press: Cambridge, MA, 2001; 585-591.
28. He X, Niyogi P. Locality preserving projections. Proceedings of Conference on Advances in Neural Information Processing Systems, Vancouver, BC, Canada, 2003.
29. Kokiopoulou E, Saad Y. Orthogonal neighborhood preserving projections: a projection-based dimensionality reduction technique. IEEE TPAMI 2007; 29:2143-2156.
30. Jolliffe IT. Principal Component Analysis. Springer: New York, 1986.
31. Torgerson WS. Multidimensional scaling: I. Theory and method. Psychometrika 1952; 17(4):401-419.
32. Cai D, He X, Han J, Zhang H-J. Orthogonal Laplacian faces for face recognition. IEEE Transactions on Image Processing 2006; 15(11):3608-3614.
33. Howland P, Park H. Generalizing discriminant analysis using the generalized singular value decomposition. IEEE Transactions on Pattern Analysis and Machine Intelligence 2004; 26(8):995-1006.
34. Zhang D, Zhou Z-H, Chen S. Semi-supervised dimensionality reduction. SIAM Data Mining, 2007; 629-634.
35. Cai D, He X, Han J. Semi-supervised discriminant analysis. IEEE International Conference on Computer Vision (ICCV), Rio de Janeiro, Brazil, 2007.
36. Song Y, Nie F, Zhang C, Xiang S. A unified framework for semi-supervised dimensionality reduction. Pattern Recognition 2008; 41:2789-2799.
37. Zhang Y, Yeung D-Y. Semi-supervised discriminant analysis using robust path-based similarity. IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), Anchorage, Alaska, 2008.
38. Zhang Y, Yeung D-Y. Semi-supervised discriminant analysis via CCCP. ECML/PKDD, 2008; 644-659.
39. Yang X, Fu H, Zha H, Barlow J. Semi-supervised nonlinear dimensionality reduction. The 23rd International Conference on Machine Learning (ICML), Pittsburg, PA, U.S.A., 2006.
40. Zhang Z, Zha H, Zhang M. Spectral methods for semi-supervised manifold learning. IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), Anchorage, Alaska, 2008.
41. Shi J, Malik J. Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis Machine Intelligence 2000; 22(8):888-905.
42. Ng AY, Jordan M, Weiss Y. On spectral clustering: analysis and an algorithm. Advances in Neural Information Processing Systems 14, 2002.
43. Ding C. Spectral clustering. International Conference on Machine Learning 2004 Tutorial, 2004.
44. Luxburg U. A tutorial on spectral clustering. Statistics and Computing 2007; 17(4):395-416.
45. Hagen L, Kahng AB. New spectral methods for ratio cut partitioning and clustering. IEEE Transactions on Computer-Aided Design of Integrated Circuits Systems 1992; 11(9):1074-1085.
46. Fiedler M. Algebraic connectivity of graphs. Czechoslovak Mathematical Journal 1973; 23:298-305.
47. Fiedler M. A property of eigenvectors of nonnegative symmetric matrices and its applications to graph theory. Czechoslovak Mathematical Journal 1975; 25:619-633.
48. Müller KR, Mika S, Ratsch G, Tsuda K, Scholkopf B. An introduction to kernel-based learning algorithms. IEEE Transactions on Neural Networks 2001; 12:181-201.
49. Vapnik V. Statistical Learning Theory. Wiley: New York, 1998.
50. Schlköpf B, Smola AJ. Learning with Kernels: Support Vector Machines, Regularization, Optimization and Beyond. MIT Press: Cambridge, MA, 2001.
51. Shawe-Taylor J, Cristianini N. Kernel Methods for Pattern Analysis. Cambridge University Press: Cambridge, 2004.
52. Aronszajn N. Theory of reproducing kernels. Transactions of the American Mathematical Society 1950; 68:337-404.
53. Schölkopf B, Smola A, Muller K. Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation 1998; 10:1299-1319.
54. Fouss F, Pirotte A, Renders J-M, Saerens M. Random-walk computation of similarities between nodes of a graph with application to collaborative recommendation. IEEE Transactions on Knowledge and Data Engineering 2007; 19(3):355-369.
55. Bengio Y, Paiement J-F, Vincent P, Delalleau O, Le Roux N, Ouimet M. Out-of-sample extensions for LLE, Isomap, MDS, Eigenmaps, and spectral clustering. In Advances in Neural Information Processing Systems 16, Vancouver, Canada, Thrun S, Saul L, Schölkopf B (eds). MIT Press: Cambridge, MA, 2004.
56. Boley D. Principal direction divisive partitioning. Data Mining and Knowledge Discovery 1998; 2(4):325-344.
57. Schölkopf B, Smola A. Learning with Kernels. MIT press: Cambridge, MA, 2002.
58. Asuncion A, Newman DJ. UCI machine learning repository (multiple features data set). Available from.
59. Graham DB, Allinson NM. Characterizing virtual eigensignatures for general purpose face recognition. Face Recognition: From Theory to Applications 1998; 163:446-456.
60. Samaria F, Harter A. Parameterisation of a stochastic model for human face identification. The 2nd IEEE Workshop on Applications of Computer Vision, Sarasota FL, December 1994.
61. Martinez AM, Benavente R. The AR face database. Technical Report 24, CVC, 1998.
62. Bennett KP, Parrado-Hernandez E. The interplay of optimization and machine learning research. Journal of Machine Learning Research 2006; 7:1265-1281.
63. Cortes C, Vapnik V. Support-vector networks. Machine Learning 1995; 20(3):273-297.
64. Weinberger KQ, Saul LK. Unsupervised learning of image manifolds by semidefinite programming. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR-04), 2004; 2:988-995.
65. Weinberger KQ, Saul LK. An introduction to nonlinear dimensionality reduction by maximum variance unfolding. AAAI'06: Proceedings of the 21st National Conference on Artificial Intelligence. AAAI Press, 2006; 1683-1686.
66. Xu L, Neufeld J, Larson B, Schuurmans D. Maximum margin clustering. Advances in Neural Information Processing Systems 17, 2005.
67. Xu L, Schuurmans D. Unsupervised and semi-supervised multi-class support vector machines. Proceedings of the 20th National Conference on Artificial Intelligence, Pittsburg, PA, U.S.A., 2005.
68. Bach F, Harchaoui Z. Diffrac: a discriminative and flexible framework for clustering. Advances in Neural Information Processing Systems 20, 2008.
69. Weinberger K, Packer B, Saul L. Nonlinear dimensionality reduction by semidefinite programming and kernel matrix factorization. Proceedings of the 10th International Workshop on Artificial Intelligence and Statistics, Barbados, 2005.