Divergence-based vector quantization

Neural Computation

Volumn 23, Issue 5, 2011, Pages 1343-1392

  Villmann, Thomas ^a   Haase, Sven ^a  

^a UNIVERSITY OF APPLIED SCIENCES MITTWEIDA   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK; AUTOMATED PATTERN RECOGNITION; COGNITION; COMPUTER SIMULATION; HUMAN; MATHEMATICAL PHENOMENA; METHODOLOGY; PHYSIOLOGY; STANDARD; THEORETICAL MODEL;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COGNITION; COMPUTER SIMULATION; HUMANS; MATHEMATICAL CONCEPTS; MODELS, THEORETICAL; NEURAL NETWORKS (COMPUTER); PATTERN RECOGNITION, AUTOMATED;

EID: 79958244935     PISSN: 08997667     EISSN: 1530888X     Source Type: Journal    
DOI: 10.1162/NECO_a_00110     Document Type: Article

References (71)

1. Alex, N., Hasenfuss, A., & Hammer, B. (2009). Patch clustering for massive data sets. Neurocomputing, 72 (7-9), 1455-1469.
2. Amari, S.-I. (1985). Differential-geometrical methods in statistics. Berlin: Springer.
3. Amari, S.-I.,&Nagaoka,H. (2000). Methods of information geometry.New York: Oxford University Press.
4. Banerjee, A., Merugu, S., Dhillon, I., & Ghosh, J. (2005). Clustering with Bregman divergences. Journal of Machine Learning Research, 6, 1705-1749.
5. Basseville, M. (1988). Distance measures for signal processing and pattern recognition. (Tech. Rep. 899). Paris: Institut National de Recherche en Informatique et en Automatique.
6. Basu, A., Harris, I., Hjort, N., & Jones, M. (1998). Robust and efficient estimation by minimising a density power divergence. Biometrika, 85(3), 549-559.
7. Bertin, N., Fevotte, C., & Badeau, R. (2009). A tempering approach for Itakurasaito non-negative matrix factorization. with application to music transcription. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1545-1548). Piscataway, NJ: IEEE Press.
8. Bishop, C. M., Svensén, M., Williams, & C.K.I. (1998). GTM: The generative topographic mapping. Neural Computation, 10, 215-234.
9. Bregman, L. (1967). The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR Computational Mathematics and Mathematical Physics, 7 (3), 200-217.
10. Bunte, K., Hammer, B., Villmann, T., Biehl, M., & Wismüller, A. (2010). Exploratory observation machine (XOM) with Kullback-Leibler divergence for dimensionality reduction and visualziation. InM. Verleysen (Ed.), Proc. of European Symposium on Artificial Neural Networks (pp. 87-92). Evere, Belgium: d-side publications.
11. Cichocki, A., & Amari, S.-I. (2010). Families of alpha-beta-and gamma-divergences: Flexible and robust measures of similarities. Entropy, 12, 1532-1568.
12. Cichocki, A., Lee, H., Kim, Y.-D.,&Choi, S. (2008). Non-negativematrix factorization with α-divergence. Pattern Recognition Letters, 29, 1433-1440.
13. Cichocki, A., Zdunek, R., Phan, A., & Amari, S.-I. (2009). Nonnegative matrix and tensor factorizations. Hoboken, NJ: Wiley.
14. Crammer, K., Gilad-Bachrach, R., Navot, A., & Tishby, A. (2002). Margin analysis of the LVQ algorithm. In S. Becker, S. Thrün, & K. Obermayer (Eds.), Advances in neural information processing systems, 15 (pp. 462-468): Cambridge, MA: MIT Press.
15. Csiszár, I. (1967). Information-type measures of differences of probability distributions and indirect observations. Studia Sci. Math. Hungaria, 2, 299-318.
16. Eguchi, S., & Kano, Y. (2001). Robustifying maximum likelihood estimation (Tech. Rep. 802). Tokyo: Tokyo Institute of Statistical Mathematics.
17. Erdogmus, D. (2002). Information theoretic learning: Renyi's entropy and its application to adaptive systems training.Unpublished doctoral dissertation, University of Florida.
18. Fichtenholz, G. (1964). Differential-und Integralrechnung (9th ed.). Berlin: Deutscher Verlag der Wissenschaften.
19. Frigyik, B., Srivastava, S., & Gupta, M. (2008a). Functional Bregman divergence and Bayesian estimation of distributions. IEEE Transactions on Information Theory, 54(11), 5130-5139.
20. Frigyik, B. A., Srivastava, S., & Gupta, M. (2008b). An introduction to functional derivatives (Tech. Rep. UWEETR-2008-0001). Seattle: Department of Electrical Engineering, University ofWashington.
21. Fujisawa, H., & Eguchi, S. (2008). Robust parameter estimation with a small bias against heavy contamination. Journal of Multivariate Analysis, 99, 2053-2081.
22. Graepel, T., Burger, M., & Obermayer, K. (1998). Self-organizing maps: Generalizations and new optimization techniques. Neurocomputing, 21(1-3), 173-190.
23. Hammer, B., & Villmann, T. (2002). Generalized relevance learning vector quantization. Neural Networks, 15(8-9), 1059-1068.
24. Hegde, A., Erdogmus, D., Lehn-Schiøler, T., Rao, Y., & Principe, J. (2004). Vector quantization by density matching in the minimum Kullback-Leibler-divergence sense. In Proc. of the International Joint Conference on Artificial Neural Networks (pp. 105-109). Piscataway, NJ: IEEE Press.
25. Heskes, T. (1999). Energy functions for self-organizing maps. In E. Oja & S. Kaski (Eds.), Kohonen maps (pp. 303-316). Amsterdam: Elsevier.
26. Hulle, M.M.V. (2000). Faithful representations and topographic maps. Hoboken, NJ:Wiley.
27. Hulle, M.M.V. (2002a). Joint entropy maximization in kernel-based topographic maps. Neural Computation, 14(8), 1887-1906.
28. Hulle, M.M.V. (2002b). Kernel-based topographic map formation achieved with an information theoretic approach. Neural Networks, 15, 1029-1039.
29. Itakura, F., & Saito, S. (1973). Analysis synthesis telephony based on the maximum likelihood method. In J. Flanagan & R. Rabiner (Eds.), Speech synthesis (pp. 289-292). Stroudsburg, PA: Dowden, Hutchinson, & Ross.
30. Jang, E., Fyfe, C., & Ko,H. (2008). Bregman divergences and the self organising map. In C. Fyfe, D. Kim, S.-Y., Lee, & H. Yin (Eds.), Intelligent data engineering and automated learning (pp. 452-458). New York: Springer.
31. Jenssen, R. (2005). An information theoretic approach to machine learning. Unpublished doctoral dissertation, University of Tromsø.
32. Jenssen, R., Principe, J., Erdogmus, D., & Eltoft, T. (2006). The Cauchy-Schwarz divergence and Parzen windowing: Connections to graph theory and Mercer kernels. Journal of the Franklin Institute, 343(6), 614-629.
33. Kantorowitsch, I., & Akilow, G. (1978). Funktionalanalysis in normierten Räumen (2nd ed.). Berlin: Akademie-Verlag.
34. Kapur, J. (1994). Measures of information and their application. Hoboken, NJ:Wiley.
35. Kohonen, T. (1997). Self-organizing maps (2nd ext. ed.). New York: Springer.
36. Kullback, S., & Leibler, R. (1951). On information and sufficiency. Annals of Mathematical Statistics, 22, 79-86.
37. Lai, P., & Fyfe, C. (2009). Bregman divergences and multi-dimensional scaling. In M. K̈oppen, N., Kasabov, & N. G. Coghill (Eds.), Proceedings of the International Conference on Information Processing 2008 (pp. 935-942). New York: Springer.
38. Lee, J., & Verleysen, M. (2005). Generalization of the l p norm for time series and its application to self-organizing maps. In M. Cottrell (Ed.), Proc. of Workshop on Self-Organizing Maps (pp. 733-740). Paris: Sorbonne.
39. Lee, J.,&Verleysen, M. (2007). Nonlinear dimensionality reduction.NewYork: Springer.
40. Lehn-Schiøler, T., Hegde, A., Erdogmus, D.,&Principe, J. (2005). Vector quantization using information theoretic concepts. Natural Computing, 4(1), 39-51.
41. Liese, F., & Vajda, I. (1987). Convex statistical distances. Leipzig: Teubner-Verlag.
42. Liese, F., & Vajda, I. (2006). On divergences and informations in statistics and information theory. IEEE Transactions on Information Theory, 52(10), 4394-4412.
43. Linde, Y., Buzo, A.,&Gray, R. (1980). An algorithm for vector quantizer design. IEEE Transactions on Communications, 28, 84-95.
44. Maaten, L., & Hinton, G. (2008). Visualizing data using t-SNE. Journal of Machine Learning Research, 9, 2579-2605.
45. Martinetz, T.M., Berkovich, S. G., & Schulten, K. J. (1993). "Neural-gas" network for vector quantization and its application to time-series prediction. IEEE Trans. on Neural Networks, 4(4), 558-569.
46. Minami, M., & Eguchi, S. (2002). Robust blind source separation by beta divergence. Neural Computation, 14, 1859-1886.
47. Minka, T. (2005). Divergence measures and message passing (Tech. Rep. 173). Cambridge, UK: Microsoft Research.
48. Mwebaze, E., Schneider, P., Schleif, F.-M., Haase, S., Villmann, T., & Biehl, M. (2010). Divergence based learning vector quantization. In M. Verleysen (Ed.), Proc. of European Symposium on Artificial Neural Networks (pp. 247-252). Evere, Belgium: D-side.
49. Nielsen, F., & Nock, R. (2009). Sided and symmetrized Bregman centroids. IEEE Transactions on Information Theory, 55(6), 2882-2903.
50. Principe, J. C. III, & Xu, D. (2000). Information theoretic learning. In S. Haykin & J. Fisher (Eds.), Unsupervised adaptive filtering. Hoboken, NJ: Wiley.
51. Qiao, Y., & Minematsu, N. (2008). f -divergence is a generalized invariant measure between disributions. In INTERSPEECH-Proc. of the Annual Conference of the International Speech Communication Association (pp. 1349-1352).N.p.: International Speech Communication Association.
52. Ramsay, J., & Silverman, B. (2006). Functional data analysis (2nd ed.) New York: Springer.
53. Renyi, A. (1961).Onmeasures of entropy and information. In Proceedings of the Fourth Berkeley Symposium onMathematical Statistics and Probability. Berkeley: University of California Press.
54. Renyi, A. (1970). Probability theory. Amsterdam: North-Holland.
55. Rossi, F., Delannay, N., Conan-Gueza, B., & Verleysen, M. (2005). Representation of functional data in neural networks. Neurocomputing, 64, 183-210.
56. Santos-Rodríguez, R., Guerrero-Curieses, A., Alaiz-Rodríguez, R., & Cid-Sueiro, J. (2009). Cost-sensitive learning based on Bregman divergences. Machine Learning, 76(2-3), 271-285.
57. Sato, A., & Yamada, K. (1996). Generalized learning vector quantization. In D. S. Touretzky, M. C. Mozer, & M. E. Hasselmo (Eds.), Advances in neural information processing systems, 8 (pp. 423-429). Cambridge, MA: MIT Press.
58. Schneider, P., Biehl, M., & Hammer, B. (2009). Hyperparameter learning in robust soft LVQ. In M. Verleysen (Ed.), Proceedings of the European Symposium on Artificial Neural Networks (pp. 517-522). Evere, Belgium: D-side.
59. Shannon, C. (1948). A mathematical theory of communication. Bell System Technical Journal, 27, 379-432.
60. Taneja, I., & Kumar, P. (2004). Relative information of type s, Csiszr's f -divergence, and information inequalities. Information Sciences, 166, 105-125.
61. Torkkola, K. (2003). Feature extraction by non-parametric mutual information maximization. Journal of Machine Learning Research, 3, 1415-1438.
62. Torkkola,K.,&Campbell,W. (2000).Mutual information in learning feature transformations. In Proc. of the International Conference onMachine Learning. San Francisco: Morgan Kaufmann.
63. Villmann, T. (2007). Sobolev metrics for learning of functional data-mathematical and theoretical aspects (Machine Learning Reports 1 1-15). Available online at http://www.uni-leipzig.de/compint/mlr/mlr012007.pdf.
64. Villmann, T., Haase, S., Schleif, F.-M., & Hammer, B. (2010). Divergence based online learning in vector quantization. In L. Rutkowski,W. Duch, J. Kaprzyk, J. Korbicz, & R. Tadeusiewicz (Eds.), Proc. of the International Conference on Artifial Intelligence and Soft Computing. New York: Springer.
65. Villmann, T., Haase, S., Simmuteit, S., Haase, M., & Schleif, F.-M. (2010). Functional vector quantization based on divergence learning. Ulmer Informatik-Berichte, 2010-05, 8-11.
66. Villmann, T., Hammer, B., Schleif, F.-M., Geweniger, T., & Herrmann, W. (2006). Fuzzy classification by fuzzy labeled neural gas. Neural Networks, 19, 772-779.
67. Villmann, T., Hammer, B., Schleif, F.-M., Hermann, W., & Cottrell, M. (2008). Fuzzy classification using information theoretic learning vector quantization. Neurocomputing, 71, 3070-3076.
68. Villmann, T., & Schleif, F.-M. (2009). Functional vector quantization by neural maps. In J. Chanussot (Ed.), Proceedings of FirstWorkshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (pp. 1-4). Piscataway, NJ: IEEE Press.
69. Villmann, T., Schleif, F.-M., Kostrzewa, M., Walch, A., & Hammer, B. (2008). Classification of mass-spectrometric data in clinical proteomics using learning vector quantization methods. Briefings in Bioinformatics, 9(2), 129-143.
70. Wismüller, A. (2009). The exploration machine: A novel method for data visualization. In J. Principe & R. Miikkulainen (Eds.), Advances in self-organizing maps-Proceedings of the 7th International Workshop (pp. 344-352). New York: Springer.
71. Zador, P. L. (1982). Asymptotic quantization error of continuous signals and the quantization dimension. IEEE Transactions on Information Theory, 28, 149-159.