Type 1 and 2 mixtures of Kullback-Leibler divergences as cost functions in dimensionality reduction based on similarity preservation

Neurocomputing

Volumn 112, Issue , 2013, Pages 92-108

  Lee, John A ^a   Renard, Emilie ^a   Bernard, Guillaume ^a   Dupont, Pierre ^a   Verleysen, Michel ^a  

^a UNIVERSITÉ CATHOLIQUE DE LOUVAIN   (Belgium)

Author keywords

Dimensionality reduction; Divergence; Manifold learning; Stochastic neighbor embedding

Indexed keywords

DIMENSIONALITY REDUCTION; DIVERGENCE; HIGH-DIMENSIONAL DATA SPACE; JENSEN-SHANNON DIVERGENCE; KULLBACK LEIBLER DIVERGENCE; MANIFOLD LEARNING; STATE-OF-THE-ART METHODS; STOCHASTIC NEIGHBOR EMBEDDING;

COST FUNCTIONS; EMBEDDED SYSTEMS;

MIXTURES;

ACCURACY; ARTICLE; COST FUNCTION; DATA ANALYSIS; DIMENSIONALITY REDUCTION; INTERMETHOD COMPARISON; JENSEN SHANNON DIVERGENCE; KULLBACK LEIBLER DIVERGENCE; MATHEMATICAL ANALYSIS; MATHEMATICAL AND STATISTICAL PROCEDURES; MATHEMATICAL MODEL; MATHEMATICAL PARAMETERS; MATHEMATICAL VARIABLE; NONLINEAR SYSTEM; PRIORITY JOURNAL; PROBABILITY; PROCEDURES; PROCESS DEVELOPMENT; PROCESS OPTIMIZATION; STOCHASTIC NEIGHBOR EMBEDDING;

EID: 84877607226     PISSN: 09252312     EISSN: 18728286     Source Type: Journal    
DOI: 10.1016/j.neucom.2012.12.036     Document Type: Article

References (47)

1. Pearson K. On lines and planes of closest fit to systems of points in space. Philos. Mag. 1901, 2:559-572.
2. Hotelling H. Analysis of a complex of statistical variables into principal components. J. Educ. Psychol. 1933, 24:417-441.
3. Jolliffe I. Principal Component Analysis 1986, Springer-Verlag, New York, NY.
4. Young G., Householder A. Discussion of a set of points in terms of their mutual distances. Psychometrika 1938, 3:19-22.
5. Torgerson W. Multidimensional scaling, I. theory and method. Psychometrika 1952, 17:401-419.
6. Borg I., Groenen P. Modern Multidimensional Scaling. Theory and Applications 1997, Springer-Verlag, New York.
7. Sammon J. A nonlinear mapping algorithm for data structure analysis. IEEE Trans. Comput. 1969, CC-18(5):401-409.
8. Tenenbaum J. Mapping a manifold of perceptual observations. Advances in Neural Information Processing Systems (NIPS) 1997 1998, vol. 10:682-688. MIT Press, Cambridge, MA.
9. Tenenbaum J., de Silva V., Langford J. A global geometric framework for nonlinear dimensionality reduction. Science 2000, 290(5500):2319-2323.
10. Lee J., Lendasse A., Donckers N., Verleysen M. A robust nonlinear projection method. Proceedings of ESANN 2000, Eighth European Symposium on Artificial Neural Networks 2000, 13-20. D-Facto Publications, Bruges, Belgium. M. Verleysen (Ed.).
11. Lee J., Lendasse A., Verleysen M. Curvilinear distances analysis versus isomap. Proceedings of ESANN 2002, 10th European Symposium on Artificial Neural Networks 2002, 185-192. d-side, Bruges, Belgium. M. Verleysen (Ed.).
12. Lee J., Verleysen M. Curvilinear distance analysis versus isomap. Neurocomputing 2004, 57:49-76.
13. Belkin M., Niyogi P. Laplacian eigenmaps and spectral techniques for embedding and clustering. Advances in Neural Information Processing Systems (NIPS 2001) 2002, vol. 14. MIT Press.
14. Roweis S., Saul L. Nonlinear dimensionality reduction by locally linear embedding. Science 2000, 290(5500):2323-2326.
15. Nadler B., Lafon S., Coifman R., Kevrekidis I. Diffusion maps spectral clustering and eigenfunction of Fokker-Planck operators. Advances in Neural Information Processing Systems (NIPS 2005) 2006, vol. 18. MIT Press, Cambridge, MA.
16. Lee J., Verleysen M. Nonlinear Dimensionality Reduction 2007, Springer.
17. Lee J., Verleysen M. Quality assessment of dimensionality reduction. rank-based criteria. Neurocomputing 2009, 72(7-9):1431-1443.
18. van der Maaten L., Hinton G. Visualizing data using t-SNE. J. Mach. Learn. Res. 2008, 9:2579-2605.
19. Schölkopf B., Smola A., Müller K.-R. Nonlinear component analysis as a kernel eigenvalue problem. Neural Comput. 1998, 10:1299-1319. also available as technical report 44 at the Max Planck Institute for Biological Cybernetics, Tübingen, Germany, December 1996.
20. J. Ham, D. Lee, S. Mika, B. Schölkopf, A kernel view of the dimensionality reduction of manifolds, in: 21th International Conference on Machine Learning (ICML-04), 2004, pp. 369-376, also available as technical report TR-102 at Max Planck Institute for Biological Cybernetics, Tübingen, Germany, 2003.
21. M. Saerens, F. Fouss, L. Yen, P. Dupont, The principal components analysis of a graph, and its relationships to spectral clustering, in: Proceedings of the 15th European Conference on Machine Learning (ECML 2004), 2004, pp. 371-383.
22. Y. Bengio, P. Vincent, J.-F. Paiement, O. Delalleau, M. Ouimet, N. Le Roux, Spectral clustering and kernel PCA are learning eigenfunctions, Technical Report. 1239, Département d'Informatique et Recherche Opérationnelle, Université de Montréal, Montréal (Jul. 2003).
23. Hinton G., Roweis S. Stochastic neighbor embedding. Advances in Neural Information Processing Systems (NIPS 2002) 2003, vol. 15:833-840. MIT Press.
24. Venna J., Peltonen J., Nybo K., Aidos H., Kaski S. Information retrieval perspective to nonlinear dimensionality reduction for data visualization. J. Mach. Learn. Res. 2010, 11:451-490.
25. J. Lee, M. Verleysen, Shift-invariant similarities circumvent distance concentration in stochastic neighbor embedding and variants, in: Proceedings of the International Conference on Computational Science (ICCS 2011), Singapore, 2011, pp. 538-547.
26. François D., Wertz V., Verleysen M. The concentration of fractional distances. IEEE Trans. Knowl. Data Eng. 2007, 19(7):873-886.
27. Bellman R. Adaptive Control Processes. A Guided Tour 1961, Princeton University Press, Princeton, NJ.
28. D. Donoho, High-Dimensional Data Analysis: The Curse and Blessings of Dimensionality, aide-mémoire for a lecture for the American Mathematical Society "Mathematical Challenges of the 21st Century", 2000.
29. M. Strickert, S. Teichmann, N. Sreenivasulu, U. Seiffert, High-throughput multi-dimensional scaling (HiT-MDS) for cDNA-array expression data, in: W. Duch (Ed.), Proceedings of the ICANN 2005, Artificial Neural Networks: Biological Inspirations (Part I), Lecture Notes in Computer Science, vol. 3696, Springer, 2005, pp. 625-634.
30. Strickert M., Sreenivasulu N., Usadel B., Seiffert U. Correlation-maximizing surrogate gene space for visual mining of gene expression patterns in developing barley endosperm tissue. BMC Bioinformatics 2007, 8:165.
31. Bunte K., Haase S., Biehl M., Villmann T. Stochastic neighbor embedding (SNE) for dimension reduction and visualization using arbitrary divergences. Neurocomputing 2012, 90:23-45.
32. Cichocki A., Amari S.-i. Families of alpha- beta- and gamma-divergences. flexible and robust measures of similarities. Entropy 2010, 12:1532-1568.
33. Burbea J., Rao C. On the convexity of some divergence measures based on entropy functions. IEEE Trans. Inf. Theory 1982, 28(3):489-495.
34. Lin J. Divergence measures based on the Shannon entropy. IEEE Trans. Inf. Theory 1991, 37(1):145-151.
35. J. Lee, M. Verleysen, On the role and impact of the metaparameters in t-distributed stochastic neighbor embedding, in: Y. Lechevallier, G. Saporta (Eds.), Proceedings of the 19th COMPSTAT, Paris (France), 2010, pp. 337-348.
36. G. Di Battista, P. Eades, R. Tamassia, I. Tollis, Algorithms for drawing graphs: an annotated bibliography, Technical Report. Brown University (Jun. 1994).
37. Di Battista G., Eades P., Tamassia R., Tollis I. Graph Drawing: Algorithms for the Visualization of Graphs 1999, Prentice-Hall.
38. Venna J., Kaski S. Local multidimensional scaling. Neural Networks 2006, 19:889-899.
39. Demartines P., Hérault J. Curvilinear component analysis. a self-organizing neural network for nonlinear mapping of data sets. IEEE Trans. Neural Networks 1997, 8(1):148-154.
40. S.A. Nene, S.K. Nayar, H. Murase, Columbia object image library (COIL-20), Technical Report. CUCS-005-96, Columbia University, 1996.
41. LeCun Y., Bottou L., Bengio Y., Haffner P. Gradient-based learning applied to document recognition. Proc. IEEE 1998, 86(11):2278-2324.
42. Lee J., Verleysen M. Scale-independent quality criteria for dimensionality reduction. Pattern Recognition Lett. 2010, 31(14):2248-2257.
43. Venna J., Kaski S. Neighborhood preservation in nonlinear projection methods. an experimental study. Proceedings of ICANN 2001 2001, 485-491. Springer, Berlin. G. Dorffner, H. Bischof, K. Hornik (Eds.).
44. Gower J. Some distance properties of latent root and vector methods used in multivariate analysis. Biometrika 1966, 53:325-338.
45. Shepard R. The analysis of proximities. multidimensional scaling with an unknown distance function (parts 1 and 2). Psychometrika 1962, 27:125-140. 219-249.
46. Kruskal J. Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis. Psychometrika 1964, 29:1-28.
47. Cichocki A., Cruces S., Amari S.-i. Generalized alpha-beta divergences and their application to robust nonnegative matrix factorization. Entropy 2011, 13:134-170.