Unsupervised dimensionality reduction: Overview and recent advances

Proceedings of the International Joint Conference on Neural Networks

Volumn , Issue , 2010, Pages

  Lee, John A ^a,c,d   Verleysen, Michel ^a,b,c  

^a UNIVERSITÉ CATHOLIQUE DE LOUVAIN   (Belgium)

^b UCL   (France)

^c UNIVERSITÉ PARIS 1 PANTHÉON SORBONNE   (France)

^d Belgian National Fund of Scientific Research (FRS FNRS)   (Belgium)

Author keywords

[No Author keywords available]

Indexed keywords

CLUSTERING ALGORITHMS; PATTERN RECOGNITION;

DE-NOISING; DIMENSIONALITY REDUCTION; HIGH DIMENSIONAL DATA; LOW-DIMENSIONAL SPACES; PROPERTY; QUALITY ASSESSMENT; TIME LINE;

DATA VISUALIZATION;

EID: 79959447008     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/IJCNN.2010.5596721     Document Type: Conference Paper

References (79)

1. M. Belkin and P. Niyogi, "Laplacian eigenmaps and spectral techniques for embedding and clustering," in Advances in Neural Information Processing Systems (NIPS 2001), T. Dietterich, S. Becker, and Z. Ghahramani, Eds. MIT Press, 2002, vol. 14.
2. -, "Laplacian eigenmaps for dimensionality reduction and data representation," Neural Computation, vol. 15, no. 6, pp. 1373-1396, Jun. 2003.
3. Y. Bengio, "Learning deep architectures for AI," Foundations and Trends in Machine Learning archive, vol. 2, no. 1, pp. 1-127, 2009.
4. Y. Bengio, J.-F. Paiement, P. Vincent, O. Delalleau, N. Le Roux, and M. Ouimet, "Out-of-sample extensions for LLE, isomap, MDS, eigenmaps, and spectral clustering," in Advances in Neural Information Processing Systems (NIPS 2003), S. Thrun, L. Saul, and B. Schölkopf, Eds. Cambridge, MA: MIT Press, 2004, vol. 16.
5. Y. Bengio, P. Vincent, J.-F. Paiement, O. Delalleau, M. Ouimet, and N. Le Roux, "Spectral clustering and kernel PCA are learning eigenfunctions," Département d'Informatique et Recherche Opérationnelle, Université de Montréal, Montréal, Tech. Rep. 1239, Jul. 2003.
6. M. Bernstein, V. de Silva, J. Langford, and J. Tenenbaum, "Graph approximations to geodesics on embedded manifolds," Stanford University, Palo Alto, CA, Tech. Rep., Dec. 2000.
7. C. Bishop, Neural Networks for Pattern Recognition. Oxford University Press, Oxford, 1995.
8. C. Bishop, M. Svensén, and C. Williams, "GTM: A principled alternative to the self-organizing map," in Advances in Neural Information Processing Systems (NIPS 1996), M. Mozer, M. Jordan, and T. Petsche, Eds. Cambridge, MA: MIT Press, 1997, vol. 9, pp. 354-360.
9. C. Bishop, M. Svensén, and K. Williams, "GTM: A principled alternative to the self-organizing map," Neural Computation, vol. 10, no. 1, pp. 215-234, 1998.
10. M. Brand, "Charting a manifold," in Advances in Neural Information Processing Systems (NIPS 2002), S. Becker, S. Thrun, and K. Obermayer, Eds. Cambridge, MA: MIT Press, 2003, vol. 15.
11. M. Brand and K. Huang, "A unifying theorem for spectral embedding and clustering," in Proceedings of International Workshop on Artificial Intelligence and Statistics (AISTATS'03), C. Bishop and B. Frey, Eds., Jan. 2003.
12. M. Carreira-Perpiñán, "A review of dimension reduction techniques," University of Sheffield, Sheffield, Tech. Rep., Jan. 1997.
13. D. de Ridder and R. Duin, "Sammon's mapping using neural networks: A comparison," Pattern Recognition Letters, vol. 18, no. 11-13, pp. 1307-1316, 1997.
14. V. de Silva and J. Tenenbaum, "Global versus local methods in nonlinear dimensionality reduction," in Advances in Neural Information Processing Systems 15, S. Becker, S. Thrun, and K. Obermayer, Eds. Cambridge, MA: MIT Press, 2003, pp. 705-712.
15. P. Demartines and J. Hérault, "Vector quantization and projection neural network," ser. Lecture Notes in Computer Science, A. Prieto, J. Mira, and J. Cabestany, Eds. New York: Springer-Verlag, 1993, vol. 686, pp. 328-333.
16. -, "Curvilinear component analysis: A self-organizing neural network for nonlinear mapping of data sets," IEEE Transactions on Neural Networks, vol. 8, no. 1, pp. 148-154, Jan. 1997.
17. D. DeMers and G. Cottrell, "Nonlinear dimensionality reduction," in Advances in Neural Information Processing Systems (NIPS 1992), D. Hanson, J. Cowan, and L. Giles, Eds. San Mateo, CA: Morgan Kaufmann, 1993, vol. 5, pp. 580-587.
18. G. Di Battista, P. Eades, R. Tamassia, and I. Tollis, Graph drawing: Algorithms for the visualization of graphs. Prentice-Hall, 1999.
19. D. Donoho and C. Grimes, "Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data," in Proceedings of the National Academy of Arts and Sciences, vol. 100, 2003, pp. 5591-5596.
20. A. Gersho and R. Gray, Vector Quantization and Signal Processing. Boston: Kluwer Academic Publisher, 1992.
21. T. Hastie and W. Stuetzle, "Principal curves," Journal of the American Statistical Association, vol. 84, no. 406, pp. 502-516, 1989.
22. J. Hérault, C. Jaussions-Picaud, and A. Guérin- Dugué, "Curvilinear component analysis for high dimensional data representation: I. Theoretical aspects and practical use in the presence of noise," in Proceedings of IWANN'99, J. Mira and J. Sánchez, Eds. Alicante, Spain: Springer, Jun. 1999, vol. II, pp. 635-644.
23. G. Hinton and S. Roweis, "Stochastic neighbor embedding," in Advances in Neural Information Processing Systems (NIPS 2002), S. Becker, S. Thrun, and K. Obermayer, Eds. MIT Press, 2003, vol. 15, pp. 833-840.
24. G. Hinton and R. Salakhutdinov, "Reducing the dimensionality of data with neural networks," Science, vol. 313, no. 5786, pp. 504-507, Jul. 2006.
25. J. Hopfield, "Neural networks and physical systems with emergent collective computational abilities," in Proc. Natl. Acad. Sci. USA 79, 1982, pp. 2554-2558.
26. H. Hotelling, "Analysis of a complex of statistical variables into principal components," Journal of Educational Psychology, vol. 24, pp. 417-441, 1933.
27. I. Jolliffe, Principal Component Analysis. New York, NY: Springer-Verlag, 1986.
28. K. Karhunen, "Zur Spektraltheorie stochastischer Prozesse," Ann. Acad. Sci. Fennicae, vol. 34, 1946.
29. T. Kohonen, "Self-organization of topologically correct feature maps," Biological Cybernetics, vol. 43, pp. 59-69, 1982.
30. -, Self-Organizing Maps, 2nd ed. Heidelberg: Springer, 1995.
31. M. Kramer, "Nonlinear principal component analysis using autoassociative neural networks," AIChE Journal, vol. 37, no. 2, pp. 233-243, 1991.
32. J. Kruskal, "Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis," Psychometrika, vol. 29, pp. 1-28, 1964.
33. J. Lee, C. Archambeau, and M. Verleysen, "Locally linear embedding versus Isotop," in Proceedings of ESANN 2003, 11th European Symposium on Artificial Neural Networks, M. Verleysen, Ed. Bruges, Belgium: d-side, Apr. 2003, pp. 527-534.
34. J. Lee, A. Lendasse, N. Donckers, and M. Verleysen, "A robust nonlinear projection method," in Proceedings of ESANN 2000, 8th European Symposium on Artificial Neural Networks, M. Verleysen, Ed. Bruges, Belgium: D-Facto public., Apr. 2000, pp. 13-20.
35. J. Lee and M. Verleysen, "Curvilinear distance analysis versus isomap," Neurocomputing, vol. 57, pp. 49-76, Mar. 2004.
36. -, "Nonlinear dimensionality reduction of data manifolds with essential loops," Neurocomputing, vol. 67, pp. 29-53, 2005.
37. -, Nonlinear dimensionality reduction. Springer, 2007.
38. -, "Quality assessment of nonlinear dimensionality reduction based on k-ary neighborhoods," in JMLR Workshop and Conference Proceedings (New challenges for feature selection in data mining and knowledge discovery), Y. Saeys, H. Liu, I. Inza, L. Wehenkel, and Y. Van de Peer, Eds., Sep. 2008, vol. 4, pp. 21-35.
39. -, "Quality assessment of dimensionality reduction: Rank-based criteria," Neurocomputing, vol. 72, no. 7-9, pp. 1431-1443, 2009.
40. -, "Scale-independent quality criteria for dimensionality reduction," Pattern Recognition Letters, 2010, in press.
41. M. Loève, "Fonctions aléatoire du second ordre," in Processus stochastiques et mouvement Brownien, P. Lévy, Ed. Paris: Gauthier-Villars, 1948, p. 299.
42. J. Mao and A. Jain, "Artificial neural networks for feature extraction and multivariate data projection," IEEE Transactions on Neural Networks, vol. 6, no. 2, pp. 296-317, Mar. 1995.
43. B. Nadler, S. Lafon, R. Coifman, and I. Kevrekidis, "Diffusion maps, spectral clustering and eigenfunction of Fokker-Planck operators," in Advances in Neural Information Processing Systems (NIPS 2005), Y. Weiss, B. Schölkopf, and J. Platt, Eds. Cambridge, MA: MIT Press, 2006, vol. 18.
44. E. Oja, "Data compression, feature extraction, and autoassociation in feedforward neural networks," in Artificial Neural Networks, T. Kohonen, K. Mäkisara, O. Simula, and J. Kangas, Eds. North-Holland: Elsevier Science Publishers, B.V., 1991, vol. 1, pp. 737-745.
45. V. Onclinx, J. A. Lee, V. Wertz, and M. Verleysen, "Dimensionality reduction by rank preservation," in Proceedings of IJCNN 2010, Barcelona, Spain, 2010, in press.
46. K. Pearson, "On lines and planes of closest fit to systems of points in space," Philosophical Magazine, vol. 2, pp. 559-572, 1901.
47. H. Ritter, T. Martinetz, and K. Schulten, Neural Computation and Self-Organizing Maps. Reading, MA: Addison-Wesley, 1992.
48. S. Roweis and L. Saul, "Nonlinear dimensionality reduction by locally linear embedding," Science, vol. 290, no. 5500, pp. 2323-2326, 2000.
49. S. Roweis, L. Saul, and G. Hinton, "Global coordination of local linear models," in Advances in Neural Information Processing Systems (NIPS 2001), T. Dietterich, S. Becker, and Z. Ghahramani, Eds. Cambridge, MA: MIT Press, 2002, vol. 14.
50. D. Rumelhart, G. Hinton, and R. Williams, "Learning representations by back-propagating errors," Nature, vol. 323, pp. 533-536, 1986.
51. M. Saerens, F. Fouss, L. Yen, and P. Dupont, "The principal components analysis of a graph, and its relationships to spectral clustering," in Proceddings of the 15th European Conference on Machine Learning (ECML 2004), 2004, pp. 371-383.
52. J. Sammon, "A nonlinear mapping algorithm for data structure analysis," IEEE Transactions on Computers, vol. CC-18, no. 5, pp. 401-409, 1969.
53. L. Saul and S. Roweis, "Think globally, fit locally: Unsupervised learning of nonlinear manifolds," Journal of Machine Learning Research, vol. 4, pp. 119-155, Jun. 2003.
54. L. Saul, K. Weinberger, J. Ham, F. Sha, and D. Lee, "Spectral methods for dimensionality reduction," in Semisupervised Learning, O. Chapelle, B. Schoelkopf, and A. Zien, Eds. MIT Press, 2006.
55. B. Schölkopf, A. Smola, and K.-R. Müller, "Nonlinear component analysis as a kernel eigenvalue problem," Neural Computation, vol. 10, pp. 1299-1319, 1998.
56. -, "Nonlinear component analysis as a kernel eigenvalue problem," Neural Computation, vol. 10, pp. 1299-1319, 1998, also available as technical report 44 at the Max Planck Institute for Biological Cybernetics, Tübingen, Germany, December 1996.
57. R. Shepard, "The analysis of proximities: Multidimensional scaling with an unknown distance function (parts 1 and 2)," Psychometrika, vol. 27, pp. 125-140, 219-249, 1962.
58. J. Svensén, "GTM: The generative topographic mapping," Ph.D. dissertation, Aston University, Aston, UK, Apr. 1998.
59. Y. Takane, F. Young, and J. de Leeuw, "Nonmetric individual differences multidimensional scaling: an alternating least squares method with optimal scaling features," Psychometrika, vol. 42, pp. 7-67, 1977.
60. Y. Teh and S. Roweis, "Automatic alignment of hidden representations," in Advances in Neural Information Processing Systems (NIPS 2002), S. Becker, S. Thrun, and K. Obermayer, Eds. Cambridge, MA: MIT Press, 2003, vol. 15.
61. J. Tenenbaum, V. de Silva, and J. Langford, "A global geometric framework for nonlinear dimensionality reduction," Science, vol. 290, no. 5500, pp. 2319-2323, Dec. 2000.
62. W. Torgerson, "Multidimensional scaling, I: Theory and method," Psychometrika, vol. 17, pp. 401-419, 1952.
63. S. Usui, S. Nakauchi, and M. Nakano, "Internal colour representation acquired by a five-layer neural network," in Artificial Neural Networks, T. Kohonen, K. Makisara, O. Simula, and J. Kangas, Eds. North-Holland: Elsevier Science Publishers, B.V., 1991.
64. L. van der Maaten and G. Hinton, "Visualizing data using t-SNE," Journal of Machine Learning Research, vol. 9, pp. 2579-2605, 2008.
65. J. Venna and S. Kaski, "Neighborhood preservation in nonlinear projection methods: An experimental study," in Proceedings of ICANN 2001, G. Dorffner, H. Bischof, and K. Hornik, Eds. Berlin: Springer, 2001, pp. 485-491.
66. -, "Local multidimensional scaling," Neural Networks, vol. 19, pp. 889-899, 2006.
67. -, "Visualizing gene interaction graphs with local multidimensional scaling," in Proceedings of ESANN 2006, 14th European Symposium on Artificial Neural Networks, M. Verleysen, Ed. Bruges, Belgium: d-side, Apr. 2006, pp. 557-562.
68. J. Venna, J. Peltonen, K. Nybo, H. Aidos, and S. Kaski, "Information retrieval perspective to nonlinear dimensionality reduction for data visualization," Journal of Machine Learning Research, vol. 11, pp. 451-490, 2010.
69. J. Verbeek, N. Vlassis, and B. Kröse, "Coordinating mixtures of probabilistic principal component analyzers," Computer Science Institute, University of Amsterdam, Amsterdam, Tech. Rep. IAS-UVA-02-01, Feb. 2002.
70. -, "Self-organizing mixture models," Neurocomputing, vol. 63, pp. 99-123, 2005.
71. C. von der Malsburg, "Self-organization of orientation sensitive cells in the striate cortex," Kybernetik, vol. 14, pp. 85-100, 1973.
72. K. Weinberger and L. Saul, "Unsupervised learning of image manifolds by semidefinite programming," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR-04), vol. 2, Washington, DC, 2004, pp. 988-995.
73. -, "Unsupervised learning of image manifolds by semidefinite programming," International Journal of Computer Vision, vol. 70, no. 1, pp. 77-90, 2006.
74. K. Weinberger, F. Sha, and L. Saul, "Learning a kernel matrix for nonlinear dimensionality reduction," in Proceedings of the Twenty-First International Conference on Machine Learning (ICML-04), Banff, Canada, 2004, pp. 839-846.
75. A. Wismüller, "The exploration machine - a novel method for data visualization," in Lecture Notes in Computer Science. Advances in Self-Organizing Maps, 2009, pp. 344-352.
76. L. Xiao, J. Sun, and S. Boyd, "A duality view of spectral methods for dimensionality reduction," in Proceedings of the 23rd International Conference on Machine Learning, Pittsburg, PA, 2006, pp. 1041-1048.
77. L. Yang, "Sammon's nonlinear mapping using geodesic distances," in Proc. 17th International Conference on Pattern Recognition (ICPR'04), 2004, vol. 2.
78. L. Yen, D. Vanvyve, F. Wouters, F. Fouss, M. Verleysen, and M. Saerens, "Clustering using a random-walk based distance measure," in Proceedings of ESANN 2005, 13th European Symposium on Artificial Neural Networks, M. Verleysen, Ed. Bruges, Belgium: dside, Apr. 2005, pp. 317-324.
79. G. Young and A. Householder, "Discussion of a set of points in terms of their mutual distances," Psychometrika, vol. 3, pp. 19-22, 1938.