Dimension reduction: A guided tour

Foundations and Trends in Machine Learning

Volumn 2, Issue 4, 2009, Pages 275-365

  Burges, Christopher J C ^a  

^a MICROSOFT RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CANONICAL CORRELATION ANALYSIS; CORRELATION DIMENSIONS; DATA SETS; DIMENSION REDUCTION; FISHER DISCRIMINANT ANALYSIS; INTRINSIC DIMENSIONS; KERNEL PCA; LAPLACIAN EIGENMAPS; LOCALLY LINEAR EMBEDDING; M METHOD; MULTI-DIMENSIONAL SCALING; PROBABILISTIC PCA; PROJECTION PURSUITS; PROJECTIVE METHODS; SPECTRAL CLUSTERING; SUFFICIENT DIMENSION REDUCTION;

DISCRIMINANT ANALYSIS; FISHER INFORMATION MATRIX;

PRINCIPAL COMPONENT ANALYSIS;

EID: 77958497920     PISSN: 19358237     EISSN: 19358245     Source Type: Journal    
DOI: 10.1561/2200000002     Document Type: Article

References (100)

1. D. H. Ackley, G. E. Hinton, and T. J. Sejnowski, "A learning algorithm for Boltzmann machines," Cognitive Science, vol. 9, 1985.
2. M. A. Aizerman, E. M. Braverman, and L. I. Rozoner, "Theoretical foundations of the potential function method in pattern recognition learning," Automation and Remote Control, vol. 25, pp. 821-837, 1964.
3. S. Akaho, "A kernel method for canonical correlation analysis," in Proceedings of the International Meeting of the Psychometric Society (IMPS2001), 2001.
4. T. W. Anderson, An Introduction to Multivariate Statistical Analysis. Wiley Series in Probability and Statistics, 2003.
5. F. R. Bach and M. I. Jordan, "Kernel independent component analysis," Journal of Machine Learning Research, vol. 3, pp. 1-48, 2002.
6. P. F. Baldi and K. Hornik, "Learning in linear neural networks: A survey," IEEE Transactions on Neural Networks, vol. 6, pp. 837-858, July 1995.
7. A. Basilevsky, Statistical Factor Analysis and Related Methods. Wiley, New York, 1994.
8. M. Belkin and P. Niyogi, "Laplacian Eigenmaps for dimensionality reduction and data representation," Neural Computation, vol. 15, pp. 1373-1396, 2003.
9. Y. Bengio, J. Paiement, and P. Vincent, "Out-of-sample extensions for LLE, Isomap, MDS, Eigenmaps and spectral clustering," in Advances in Neural Information Processing Systems 16, MIT Press, 2004.
10. C. Berg, J. P. R. Christensen, and P. Ressel, Harmonic Analysis on Semigroups. Springer-Verlag, 1984.
11. C. M. Bishop, "Bayesian PCA," in Advances in Neural Information Processing Systems 11, MIT Press, 1999.
12. I. Borg and P. Groenen, "Modern Multidimensional Scaling: Theory and Applications," Springer, 1997.
13. B. E. Boser, I. M. Guyon, and V. Vapnik, "A training algorithm for optimal margin classifiers," in Fifth Annual Workshop on Computational Learning Theory, pp. 144-152, ACM, Pittsburgh, 1992.
14. S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge University Press, 2004.
15. C. J. C. Burges, "A tutorial on support vector machines for pattern recognition," Data Mining and Knowledge Discovery, vol. 2, pp. 121-167, 1998.
16. C. J. C. Burges, "Some notes on applied mathematics for machine learning," in Advanced Lectures on Machine Learning, (O. Bousquet, U. von Luxburg, and G. Rätsch, eds.), pp. 21-40, Springer Lecture Notes in Artificial Intelligence, 2004.
17. C. J. C. Burges, "Geometric methods for feature selection and dimensional reduction," in Data Mining and Knowledge Discovery Handbook: A Complete Guide for Practitioners and Researchers, (L. Rokach and O. Maimon, eds.), Kluwer Academic, 2005.
18. C. J. C. Burges, "Simplified support vector decision rules," in Proceedings of the Thirteenth International Conference on Machine Learning, pp. 71-77, 1996.
19. C. J. C. Burges, J. C. Platt, and S. Jana, "Extracting noise-robust features from audio," in Proceedings of the IEEE Conference on Acoustics, Speech and Signal Processing, pp. 1021-1024, IEEE Signal Processing Society, 2002.
20. C. J. C. Burges, J. C. Platt, and S. Jana, "Distortion discriminant analysis for audio fingerprinting," IEEE Transactions on Speech and Audio Processing, vol. 11, pp. 165-174, 2003.
21. F. R. K. Chung, Spectral Graph Theory. American Mathematical Society, 1997.
22. R. D. Cook, Regression Graphics. Wiley, 1998.
23. R. D. Cook, "Model based sufficient dimension reduction for regression," Isaac Newton Institute Lectures on Contemporary Frontiers in High-Dimensional Statistical Data Analysis, http://www.newton.ac.uk/webseminars/ pg+ws/ 2008/sch/schw01/0108/cook/, 2008.
24. R. D. Cook and L. Forzani, "Likelihood-based sufficient dimension reduction," Journal of the American Statistical Association, vol. 104, pp. 197-208, 2009.
25. R. D. Cook and H. Lee, "Dimension reduction in binary response regression," Journal of the American Statistical Association, vol. 94, pp. 1187-1200, 1999.
26. R. D. Cook and S. Weisberg, "Sliced inverse regression for dimension reduction: Comment," Journal of the American Statistical Association, vol. 86, pp. 328-332, 1991.
27. T. F. Cox and M. A. A. Cox, Multidimensional Scaling. Chapman and Hall, 2001.
28. R. B. Darlington, "Factor analysis," Technical report, Cornell University, http://comp9.psych.cornell.edu/Darlington/factor.htm, 1997.
29. V. De Silva and J. B. Tenenbaum, "Global versus local methods in nonlinear dimensionality reduction," Advances in Neural Information Processing Systems, vol. 15, pp. 705-712, MIT Press, 2002.
30. A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum likelihood from incomplete data via the EM algorithm," Journal of the Royal Statistical Society B, vol. 39, pp. 1-22, 1977.
31. P. Diaconis and D. Freedman, "Asymptotics of graphical projection pursuit," Annals of Statistics, vol. 12, pp. 793-815, 1984.
32. K. I. Diamantaras and S. Y. Kung, Principal Component Neural Networks. Wiley, 1996.
33. R. O. Duda and P. E. Hart, Pattern Classification and Scene Analysis. Wiley, 1973.
34. C. Fowlkes, S. Belongie, F. Chung, and J. Malik, "Spectral grouping using the Nyström method," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 26, 2004.
35. J. H. Friedman and W. Stuetzle, "Projection pursuit regression," Journal of the American Statistical Association, vol. 76, pp. 817-823, 1981.
36. J. H. Friedman, W. Stuetzle, and A. Schroeder, "Projection Pursuit density estimation," Journal of the American Statistical Association, vol. 79, pp. 599- 608, 1984.
37. J. H. Friedman and J. W. Tukey, "A projection pursuit algorithm for exploratory data analysis," IEEE Transactions on Computers, vol. 23, pp. 881-890, 1974.
38. K. Fukumizu, F. R. Bach, and M. I. Jordan, "Kernel dimension reduction in regression," Annals of Statistics, vol. 37, pp. 1871-1905, 2009.
39. A. Globerson and N. Tishby, "Sufficient dimensionality reduction," Journal of Machine Learning Research, vol. 3, 2003.
40. J. Goldberger, S. T. Roweis, G. E. Hinton, and R. Salakhutdinov, "Neighbourhood components analysis," Advances in Neural Information Processing Systems, vol. 17, 2005.
41. G. H. Golub and C. F. Van Loan, Matrix Computations. Johns Hopkins, 3rd ed., 1996.
42. M. Gondran and M. Minoux, Graphs and Algorithms. Wiley, 1984.
43. P. Grassberger and I. Procaccia, "Measuring the strangeness of strange attractors," Physica, vol. 9D, pp. 189-208, 1983.
44. A. Gretton, O. Bousquet, A. Smola, and B. Schölkopf, "Measuring statistical dependence with Hilbert-Schmidt norms," in Algorithmic Learning Theory, Springer Lecture Notes in Computer Science, vol. 3734, pp. 63-77, 2005.
45. G. Grimmet and D. Stirzaker, Probability and Random Processes. Oxford University Press, 3rd ed., 2001.
46. J. Ham, D. D. Lee, S. Mika, and B. Schölkopf, "A kernel view of dimensionality reduction of manifolds," in Proceedings of the International Conference on Machine Learning, 2004.
47. D. R. Hardoon, S. Szedmak, and J. Shawe-Taylor, "Canonical correlation analysis: An overview with application to learning methods," Neural Computation, vol. 12, pp. 2639-2664, 2004.
48. T. J. Hastie and W. Stuetzle, "Principal curves," Journal of the American Statistical Association, vol. 84, pp. 502-516, 1989.
49. N. J. Higham, "Computing the nearest symmetric positive semidefinite matrix," Linear Algebra and its Applications, vol. 103, pp. 103-118, 1988.
50. G. E. Hinton, "Training products of experts by minimizing contrastive divergence," Neural Computation, vol. 14, pp. 1771-1800, 2002.
51. G. E. Hinton, S. Osindero, and Y. Teh, "A fast learning algorithm for deep belief nets," Neural Computation, vol. 7, pp. 1527-1554, 2006.
52. G. E. Hinton and S. E. Roweis, "Stochastic neighbor embedding," Advances in Neural Information Processing Systems, vol. 14, pp. 833-840, 2002.
53. G. E. Hinton and R. Salakhutdinov, "Reducing the dimensionality of data with neural networks," Science, vol. 313, pp. 504-507, 2007.
54. R. A. Horn and C. R. Johnson, Matrix Analysis. Cambridge University Press, 1985.
55. H. Hotelling, "Relations between two sets of variates," Biometrika, vol. 28, pp. 321-377, 1936.
56. T. Hsing and H. Ren, "An RKHS formulation of the inverse regression dimension-reduction problem," Annals of Statistics, vol. 37, pp. 726-755, 2009.
57. P. J. Huber, "Projection pursuit," Annals of Statistics, vol. 13, pp. 435-475, 1985.
58. A. Hyvärinen, J. Karhunen, and E. Oja, Independent Component Analysis. Wiley, 2001.
59. T. L. Kelley, Crossroads in the Mind of Man: A study of Differentiable Mental Abilities. Stanford University Press, 1928.
60. G. S. Kimeldorf and G. Wahba, "Some results on Tchebycheffian spline functions," Journal of Mathematical Analysis and Applications, vol. 33, pp. 82-95, 1971.
61. Knowledge Discovery and Data Mining Cup, http://www.kddcup2008.com/ index.html, 2008.
62. J. Lee and M. Verleysen, Nonlinear Dimensionality Reduction. Springer, 2007.
63. B. Li, H. Zha, and F. Chiaromonte, "Contour regression: A general approach to dimension reduction," The Annals of Statistics, vol. 33, pp. 1580-1616, 2005.
64. C.-K. Li, "Sliced Inverse Regression for dimension reduction," Journal of the American Statistical Association, vol. 86, pp. 316-327, 1991.
65. C.-K. Li, "On Principal Hessian Directions for data visualization and dimension reduction: Another application of Stein's lemma," Journal of the American Statistical Association, vol. 87, pp. 1025-1039, 1992.
66. M. Meila and J. Shi, "Learning segmentation by random walks," Advances in Neural Information Processing Systems, vol. 12, pp. 873-879, 2000.
67. S. Mika, B. Schölkopf, A. J. Smola, K.-R. Müller, M. Scholz, and G. Rätsch, "Kernel PCA and de-noising in feature spaces," Advances in Neural Information Processing Systems, vol. 11, MIT Press, 1999.
68. A. Y. Ng, M. I. Jordan, and Y. Weiss, "On spectral clustering: Analysis and an algorithm," Advances in Neural Information Processing Systems, vol. 14, MIT Press, 2002.
69. J. Nilsson, F. Sha, and M. I. Jordan, "Regression on manifolds using kernel dimension reduction," in Proceedings of the 24th International Conference on Machine Learning, 2007.
70. J. C. Platt, "Fastmap, MetricMap, and landmark MDS are all Nyström algorithms," in Proceedings of the 10th International Conference on Artificial Intelligence and Statistics, 2005.
71. W. H. Press, B. P. Flannery, S. A. Teukolsky, and W. T. Vettering, Numerical Recipes in C: The Art of Scientific Computing. Cambridge University Press, 2nd ed., 1992.
72. A. Rahimi and B. Recht, "Clustering with normalized cuts is clustering with a hyperplane," Workshop on Statistical Learning in Computer Vision, Prague, 2004.
73. S. M. Ross, Introduction to Probability Models. Academic Press, 10th ed., 2010.
74. S. M. Ross and E. A. Peköz, A Second Course in Probability. www.Probability Bookstore.com, Boston, MA, 2007.
75. S. T. Roweis and L. K. Saul, "Nonlinear dimensionality reduction by locally linear embedding," Science, vol. 290, pp. 2323-2326, 2000.
76. L. K. Saul and S. T. Roweis, "Think globally, fit locally: Unsupervised learning of low dimensional manifolds," Journal of Machine Learning Research, vol. 4, pp. 119-155.
77. L. K. Saul, K. Q. Weinberger, J. H. Ham, F. Sha, and D. D. Lee, "Spectral methods for dimensionality reduction," in Semisupervised Learning, (O. Chapelle, B. Schölkopf, and A. Zien, eds.), pp. 293-308, MIT Press, 2006.
78. I. J. Schoenberg, "Remarks to Maurice Frechet's article Sur la définition axiomatique d'une classe d'espace distancíes vectoriellement applicable sur l'espace de hilbert," Annals of Mathematics, vol. 36, pp. 724-732, 1935.
79. B. Schölkopf, "The kernel trick for distances," Advances in Neural Information Processing Systems, vol. 13, pp. 301-307, MIT Press, 2001.
80. B. Schölkopf and A. Smola, Learning with Kernels. MIT Press, 2002.
81. B. Schölkopf, A. Smola, and K.-R. Muller, "Nonlinear component analysis as a kernel eigenvalue problem," Neural Computation, vol. 10, pp. 1299-1319, 1998.
82. J. Shi and J. Malik, "Normalized cuts and image segmentation," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, pp. 888-905, 2000.
83. C. E. Spearman, "'General intelligence' objectively determined and measured," American Journal of Psychology, vol. 5, pp. 201-293, 1904.
84. I. Steinwart, "On the influence of the kernel on the consistency of support vector machines," Journal of Machine Learning Research, vol. 37, pp. 726- 755, 2001.
85. C. J. Stone, "Optimal global rates of convergence for nonparametric regression," Annals of Statistics, vol. 10, pp. 1040-1053, 1982.
86. J. Sun, S. Boyd, L. Xiao, and P. Diaconis, "The fastest mixing Markov process on a graph and a connection to a maximum variance unfolding problem," Society for Industrial and Applied Mathematics (SIAM) Review, vol. 48, pp. 681-699, 2006.
87. J. B. Tenenbaum, "Mapping a manifold of perceptual observations," Advances in Neural Information Processing Systems, vol. 10, MIT Press, 1998.
88. M. E. Tipping and C. M. Bishop, "Mixtures of probabilistic principal component analyzers," Neural Computation, vol. 11, pp. 443-482, 1999.
89. M. E. Tipping and C. M. Bishop, "Probabilistic principal component analysis," Journal of the Royal Statistical Society, vol. 61, p. 611, 1999.
90. N. Tishby, F. C. Pereira, and W. Bialek, "The information bottleneck method," in Proceedings of the 37th Annual Allerton Conference on Communication, Control and Computing, pp. 368-377, 1999.
91. L. van der Maaten, "t-Distributed Stochastic Neighbor Embedding," http://homepage.tudelft.nl/19j49/t-SNE.html.
92. L. van der Maaten, "Learning a parametric embedding by preserving local structure," in Proceedings of the 12th International Conference on Artificial Intelligence and Statistics (AISTATS), 2009.
93. L. van der Maaten and G. Hinton, "Visualizing data using t-SNE," Journal of Machine Learning Research, vol. 9, pp. 2579-2605, 2008.
94. L. van der Maaten, E. Postma, and J. van den Herik, "Dimensionality reduction: a comparative review," Tilburg University Technical Report TiCC-TR 2009-005, 2009.
95. U. von Luxburg, "A tutorial on spectral clustering," Statistics and Computing, vol. 17, no. 4, pp. 395-416, 2007.
96. K. Q. Weinberger and L. K. Saul, "An introduction to nonlinear dimensionality reduction by maximum variance unfolding," in Proceedings of the Twenty First National Conference on Artificial Intelligence (AAAI-06), pp. 1683- 1686, 2006.
97. S. Wilks, Mathematical Statistics. Wiley, 1962.
98. C. K. I. Williams, "On a connection between kernel PCA and metric multidimensional scaling," Advances in Neural Information Processing Systems, vol. 13, pp. 675-681, MIT Press, 2001.
99. C. K. I. Williams and M. Seeger, "Using the Nyström method to speed up kernel machines," Advances in Neural Information Processing Systems, vol. 13, pp. 682-688, MIT Press, 2001.
100. Z. Zhang and M. I. Jordan, "Multiway spectral clustering: A margin-based perspective," Statistical Science, vol. 23, pp. 383-403, 2008.