Deterministic annealing for clustering, compression, classification, regression, and related optimization problems

Proceedings of the IEEE

Volumn 86, Issue 11, 1998, Pages 2210-2239

  Rose, Kenneth ^a,b  

^a University of California at Santa Barbara   (United States)

^b IEEE   (United States)

Author keywords

Classification; Clustering; Compression; Deterministic annealing; Maximum entropy; Optimization methods; Regression; Vector quantization

Indexed keywords

PATTERN RECOGNITION; RANDOM PROCESSES; REGRESSION ANALYSIS; SIGNAL DISTORTION; SIMULATED ANNEALING; VECTOR QUANTIZATION;

DETERMINISTIC ANNEALING METHOD; SHANNON RATE-DISTORTION FUNCTIONS;

DIGITAL SIGNAL PROCESSING;

EID: 0032202775     PISSN: 00189219     EISSN: None     Source Type: Journal    
DOI: 10.1109/5.726788     Document Type: Article

References (110)

1. H. Akaike, "A new look at statistical model identification," IEEE Trans. Automat. Contr., vol. 19, pp. 713-723, Dec. 1974.
2. S. Arimoto, "An algorithm for calculating the capacity of an arbitrary discrete memorylcss channel," IEEE Trans. Inform. Theory, vol. IT-18, pp. 14-20, Jan. 1972.
3. E. Ayanoglu and R. M. Gray, "The design of joint source and channel trellis waveform coders," IEEE Trans. Inform. Theory, vol. IT-33, pp. 855-865, Nov. 1987.
4. G. Ball and D. Hall, "A clustering technique for summarizing multivariatc data," Behav. Sci., vol. 12, pp. 153-155, Mar. 1967.
5. L. E. Baum, T. Petrie, G. Soules, and N. Weiss, "A maximization technique occurring in the statistical analysis of probabilistic functions of Markov chains," Ann. Math. Statist., vol. 41, no. 1, pp. 164-171, 1970.
6. G. Beni and X. Liu, "A least biased fuzzy clustering method," IEEE Trans. Pattern Anal. Machine Intell., vol. 16, pp. 954-960, Sept. 1994.
7. T. Berger, Rate Distortion Theory. Englewood Cliffs, NJ: Prentice-Hall, 1971.
8. _, "Minimum entropy quantizers and permutation codes," IEEE Trans. Inform. Theory, vol. IT-28, pp. 149-157, Mar. 1982.
9. J. C. Bezdek, "A convergence theorem for the fuzzy ISODATA clustering algorithms," IEEE Trans. Pattern Anal. Machine Intell., vol. PAMI-2, pp. 1-8, Jan. 1980.
10. G. L. Bilbro, W. E. Snyder, S. J. Gamier, and J. W. Gault, "Mean field annealing: A formalism for constructing GNC-like algorithms," IEEE Trans. Neural Networks, vol. 3, pp. 131-138, Jan. 1992.
11. R. E. Blahut, "Computation of channel capacity and rate-distortion functions," IEEE Trans. Inform. Theory, vol. IT-18, pp. 460-473, July 1972.
12. _, Principles and Practice of Information Theory. Reading, MA: Addison-Wesley, 1987.
13. L. Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone, Classification and Regression Trees, (Wadsworth Statistics/Probability Series). Belmont, CA: Wadsworth, 1980.
14. J. Buhmann and H. Kuhnel, "Vector quantization with complexity costs," IEEE Trans. Inform. Theory, vol. 39, pp. 1133-1145, July 1993.
15. A. Buzo, A. H. Gray, Jr., R. M. Gray, and J. D. Market, "Speech coding based on vector quantization," IEEE Trans. Acoustics, Speech, Signal Processing, vol. 28, pp. 562-574, Oct. 1980.
16. [ 16] P.-C. Chang and R. M. Gray, "Gradient algorithms for designing predictive vector quantizers," IEEE Trans. Acoustics, Speech, Signal Processing, vol. ASSP-34, pp. 679-690, Aug. 1986.
17. V. Cherkassky, Y. Lee, and H. Lari-Najafi, "Self-organizing network for regression: Efficient implementation and comparative evaluation," in Proc. Int. Joint Conf. Neural Networks, 1991, vol. 1, pp. 79-84.
18. P. A. Chou, "Optimal partitioning for classification and regression trees," IEEE Trans. Pattern Anal. Machine Intell., vol. 13, pp. 340-354, Apr. 1991.
19. P. A. Chou, T. Lookabaugh, and R. M. Gray, "Entropy-constrained vector quantization," IEEE Trans. Acoustics, Speech, Signal Processing, vol. ASSP-37, pp. 31-42, Jan. 1989.
20. _, "Optimal pruning with applications to tree-structured source coding and modeling," IEEE Trans. Inform. Theory, vol. 35, pp. 299-315, Mar. 1989.
21. A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum-likelihood from incomplete data via the EM algorithm," J. Roy. Stat. Soc., vol. 39, no. 1, pp. 1-38, 1977.
22. R. D. Dony and S. Haykin, "Neural network approaches to image compression," Proc. IEEE, vol. 83, pp. 288-303, Feb. 1995.
23. R. O. Duda and P. E. Hart, Pattern Classification and Scene Analysis. New York: Wiley, 1974.
24. J. G. Dunham and R. M. Gray, "Joint source and channel trellis encoding," IEEE Trans. Inform. Theory, vol. IT-27, pp. 516-519, July 1981.
25. J. C. Dunn, "A fuzzy relative of the ISODATA process and its use in detecting compact well-separated clusters," J. Cybern., vol. 3, no. 3, pp. 32-57, 1974.
26. R. Durbin, R. Szeliski, and A. Yuille, "An analysis of the elastic net approach to the travelling salesman problem," Neural Computation, vol. 1, no. 3, pp. 348-358, 1989.
27. R. Durbin and D. Willshaw, "An analogue approach to the travelling salesman problem using an elastic net method," Nature, vol. 326, pp. 689-691, 1987.
28. N. Farvardin, "A study of vector quantization for noisy channels," IEEE Trans. Inform. Theory, vol. 36, pp. 799-809, July 1990.
29. N. Farvardin and J. W. Modestino, "Optimum quantizer performance for a class of non-Gaussian memoryless sources," IEEE Trans. Inform. Theory, vol. IT-30, pp. 485-497, May 1984.
30. N. Farvardin and V. Vaishampayan, "On the performance and complexity of channel-optimized vector quantizers," IEEE Trans. Inform. Theory, vol. 37, pp. 155-160, Jan. 1991.
31. S. L. Fix, "Rate distortion functions for squared error distortion measures," in Proc. 16th Anna. Allerton Conf. Communications Control and Computers, Oct. 1978, pp. 704-711.
32. R. G. Gallager, Information Theory and Reliable Communication. New York: Wiley, 1968.
33. D. Geiger and F. Girosi, "Parallel and deterministic algorithms from MRFs: Surface reconstruction," IEEE Trans. Pattern Anal. Machine Intell., vol. 13, pp. 401-412, May 1991.
34. S. Geman and D. Geman, "Stochastic relaxation, Gibbs distribution, and the Bayesian restoration of images," IEEE Trans. Pattern Anal. Machine Intell., vol. 6, pp. 721-741, Nov. 1984.
35. A. Gersho, "Optimal nonlinear interpolative vector quantizers," IEEE Trans. Commun., vol. COM-38, pp. 1285-1287, Sept. 1990.
36. A. Gersho and R. M. Gray, Vector Quantization and Signal Compression. Boston, MA: Kluwer, 1991.
37. S. Gold and A. Rangarajan, "A graduated assignment algorithm for graph matching," IEEE Trans. Pattern Anal. Machine Intell., vol. 18, pp. 377-388, Apr. 1996.
38. T. Graepel, M. Burger, and K. Obermayer, "Phase transitions in stochastic self-organizing maps," Phys. Rev. E, vol. 56, no. 4, pp. 3876-3890, 1997.
39. R. M. Gray, Probability, Random Processes, and Ergodic Properties. New York: Springer-Verlag, 1988.
40. _, Source Coding Theory. Boston, MA: Kluwer, 1990.
41. R. M. Gray and E. D. Karnin, "Multiple local minima in vector quantizers," IEEE Trans. Inform. Theory, vol. IT-28, pp. 256-261, Mar. 1982.
42. B. Hajek, "A tutorial survey of theory and applications of simulated annealing," in Proc. 24th IEEE Conf. Decision and Control, 1985, pp. 755-760.
43. D. Harrison and D. L. Rubinfeld, "Hedonic prices and the demand for clean air," J. Environ. Economics and Management, vol. 5, pp. 81-102, 1978.
44. J. A. Hartigan, Clustering Algorithms. New York: Wiley, 1975.
45. S. Haykin, Neural Nehvorks: A Comprehensive Foundation, 2nd ed. Englewoods Cliffs, NJ: Prentice-Hall, 1998.
46. G. E. Hinton and M. Revow, "Using pairs of data points to define splits for decision trees," Neural Inform. Processing Syst., vol. 8, pp. 507-513, 1995.
47. T. Hofmann and J. M. Buhmann, "Pairwise data clustering by deterministic annealing," IEEE Trans. Pattern Anal. Machine Intell., .vol. 19, pp. 1-14, Jan. 1997.
48. Y. H. Hu, S. Palreddy, and W. J. Tompkins, "Customized ECG beat classifier using mixture of experts," in Proc. IEEE Workshop Neural Networks for Signal Processing, 1995, pp. 459-464.
49. J.-N. Hwang, S.-R. Lay, M. Maechler, R. D. Martin, and J. Schimert, "Regression modeling in back-propagation and projection pursuit learning," IEEE Trans. Neural Networks, vol. 5, pp. 342-353, May 1994.
50. R. A. Jacobs and M. I. Jordan, "Learning piecewise control strategies in a modular neural network architecture," IEEE Trans. Syst., Man, Cybern., vol. 23, pp. 337-345, Mar./Apr. 1993.
51. R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton, "Adaptive mixtures of local experts," Neural Computation, vol. 3, no. 1, pp. 79-87, 1991.
52. E. T. Jaynes, "Information theory and statistical mechanics," in Papers on Probability, Statistics and Statistical Physics, R. D. Rosenkrantz, Ed. Dordrecht, The Netherlands: Kluwer, 1989.
53. M. I. Jordan and R. A. Jacobs, "Hierarchical mixtures of experts and the EM algorithm," Neural Computation, vol. 6, no. 2, pp. 181-214, 1994.
54. S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi, "Optimization by simulated annealing," Science, vol. 220, pp. 671-680, 1983.
55. M. Kloppenburg and P. Tavan, "Deterministic annealing for density estimation by multivariate normal mixtures," Phys. Rev. E, vol. 55, no. 3, pp. 2089-2092, 1997.
56. T. Kohonen, G. Barna, and R. Chrisley, "Statistical pattern recognition with neural networks: Benchmarking studies," in Proc. IEEE Int. Conf. Neural Networks, 1988, vol. I, pp. 61-68.
57. H. Kumazawa, M. Kasahara, and T. Namekawa, "A construction of vector quantizers for noisy channels," Electron. Commun. Japan, vol. 67, no. 4, pp. 39-47, 1984.
58. A. Kurtenbach and P. Wintz, "Quantizating for noisy channels," IEEE Trans. Commun., vol. COM-17, pp. 291-302, Apr. 1969.
59. Y. Linde, A. Buzo, and R. M. Gray, "An algorithm for vector quantizer design," IEEE Trans. Commun., vol. COM-28, pp. 84-95, Jan. 1980.
60. S. P. Lloyd, "Least squares quantization in PCM," IEEE Trans. Inform. Theory, vol. IT-28, pp. 129-137, Mar. 1982.
61. S. P. Luttrell, "Hierarchical vector quantization (image compression)," Proc. Inst. Elect. Eng. I (Commun. Speech and Vision), vol. 136, no. 6, 1989, pp. 405-413.
62. _, "Derivation of a class of training algorithms," IEEE Trans. Neural Networks, vol. 1, pp. 229-232, June 1990.
63. J. MacQueen, "Some methods for classification and analysis of multivariate observations," in Proc. 5th Berkeley Symp. Math. Statistics and Probability, 1967, 281-297.
64. T. M. Martinetz, S. G. Berkovich, and K. J. Schulten, "'Neuralgas' network for vector quantization and its application to time-series prediction," IEEE Trans. Neural Networks, vol. 4, pp. 558-569, July 1993.
65. J. Max, "Quantizing for minimum distortion," IRE Trans. Inform. Theory, vol. IT-6, pp. 7-12, Mar. 1960.
66. G. C. McDonald and R. C. Schwing, "Instabilities of regression estimates relating air pollution to mortality," Technometrics, vol. 15, pp. 463-482, 1973.
67. G. J. McLachlan and K. E. Basford, Mixture Models: Inference and Application to Clustering. New York: Marcel Dekker, 1988.
68. N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, and E. Teller, "Equations of state calculations by fast computing machines," J. Chem. Phys., vol. 21, no. 6, pp. 1087-1091, 1953.
69. D. Miller, "An information-theoretic framework for optimization with applications in source coding and pattern recognition," Ph.D. dissertation, Univ. California, Santa Barbara, 1995.
70. D. Miller, A. V. Rao, K. Rose, and A. Gersho, "A global optimization technique for statistical classifier design," IEEE Trans. Signal Processing, vol. 44, pp. 3108-3122, Dec. 1996.
71. D. Miller and K. Rose, "An improved sequential search multistage vector quantizer," in Proc. IEEE Data Computers Conf. 1993, pp. 12-21.
72. _, "Combined source-channel vector quantization using deterministic annealing," IEEE Trans. Commun., vol. 42, pp 347-356, Feb. 1994.
73. _, "Hierarchical, unsupervised learning with growing via phase transitions," Neural Computation, vol. 8, no. 2, pp. 425-450, 1996.
74. D. Miller, K. Rose, and P. A. Chou, "Deterministic annealing for trellis quantizer and HMM design using Baum-Welch re-estimation," in Proc. IEEE Int. Conf. Acoustics, Speech, Signal Processing, 1994, vol. V, pp. 261-264.
75. J. Moody and C. J. Darken, "Fast learning in locally-tuned processing units," Neural Computation, vol. 1, no. 2, pp. 281-294, 1989.
76. M. T. Musavi, W. Ahmed, K. H. Chan, K. B. Fans, and D. M. Hummels, "On the training of radial basis function classifiers," Neural Networks, vol. 5, no. 4, pp. 595-604, 1992.
77. A. V. Rao, "Design of pattern recognition systems using deterministic annealing: Applications in speech recognition, regression, and data compression," Ph.D. dissertation, Univ. California, Santa Barbara, 1998.
78. A. Rao, D. Miller, K. Rose, and A. Gersho, "A deterministic annealing approach for parsimonious design of piecewise regression models," submitted for publication.
79. _, "A generalized VQ method for combined compression , and estimation," in Proc. IEEE Int. Conf. Acoustics, Speech, Signal Processing, 1996, vol. IV, pp. 2032-2035.
80. A. Rao, K. Rose, and A. Gersho, "Design of robust HMM speech recognizer using deterministic annealing," in Proc. IEEE Workshop Automatic Speech Recognition and Understanding, Santa Barbara, CA, Dec. 1997, pp. 466-473.
81. _, "A deterministic annealing approach to discriminative hidden Markov model design," in Proc. IEEE Workshop Neural Networks in Signal Processing, Amelia Island, FL, Sept. 1997, pp. 266-275.
82. A. Rao, D. Miller, K. Rose, and A. Gersho, "Mixture of experts regression modeling by deterministic annealing," IEEE Trans. Signal Processing, vol. 45, pp. 2811-2820, Nov. 1997.
83. B. D. Ripley, "Neural networks and related methods for classification," J. Roy. Stat. Soc., Ser. B, vol. 56, no. 3, pp. 409-456, Nov. 1994.
84. E. A. Riskin and R. M. Gray, "A greedy tree growing algorithm for the design of variable rate vector quantizers," IEEE Trans. Signal Processing, vol. 39, pp. 2500-2507, Nov. 1991.
85. J. Rissanen, "Stochastic complexity and modeling," Ann. Statist., vol. 14, pp. 1080-1100, 1986.
86. K. Rose, "Deterministic annealing, clustering, and optimization," Ph.D. dissertation, California Inst. Technol., Pasadena, 1991.
87. _, "A mapping approach to rate-distortion computation and analysis," IEEE Trans. Inform. Theory, vol. 40, pp. 1939-1952, Nov. 1994.
88. K. Rose, E. Gurewitz, and G. C. Fox, "A deterministic annealing approach to clustering," Pattern Recognition Lett., vol. 11, no. 9, pp. 589-594, 1990.
89. _, "Statistical mechanics and phase transitions in clustering," Phys. Rev. Lett., vol. 65, no. 8, pp. 945-948, 1990.
90. _, "Vector quantization by deterministic annealing," IEEE Trans. Inform. Theory, vol. 38, pp. 1249-1257, July 1992.
91. _, "Constrained clustering as an optimization method," IEEE Trans. Pattern Anal. Machine Intell., vol. 15, pp. 785-794, Aug. 1993.
92. K. Rose and D. Miller, "Constrained clustering for data assignment problems with examples of module placement," in Proc. IEEE Int. Symp. Circuits and Systems, San Diego, CA, May 1992, pp. 1937-1940.
93. H. L. Royden, Real Analysis, 3rd ed. New York: Macmillan, 1988.
94. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, Parallel Distributed Processing. Cambridge, MA: MIT, 1986.
95. C. E. Shannon, "A mathematical theory of communication," Bell Syst. Tech. J., vol. 27, pp. 379-423, 623-656, 1948.
96. _, "Coding theorems for a discrete source with a fidelity criterion," IRE Nat. Conv: Rec., pt. 4, pp. 142-163, 1959.
97. J. E. Shore and R. W. Johnson, "Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy," IEEE Trans. Inform. Theory, vol. IT-26, pp. 26-37, Jan. 1980.
98. P. D. Simic, "Statistical mechanics as the underlying theory of elastic and neural optimization," Network, vol. I, no. 1, pp. 89-103, 1990.
99. _, "Constrained nets for graph matching and other quadratic assianment problems," Neural Computation, vol. 3, no. 2, pp. 268-281, 1991.
100. H. Szu and R. Hartley, "Nonconvex optimization by fast simulated annealing," Proc. IEEE, vol. 75, pp. 1538-1540, Nov. 1987.
101. L. Tarassenko and S. Roberts, "Supervised and unsupervised learning in radial basis function classifiers," Proc. Inst. Elect. Eng.-Visual Image Signal Processing, vol. 141, no. 4, pp. 210-216, 1994.
102. D. M. Titterington, A. F. M. Smith, and U. E. Makov, Analysis of Finite Mixture Distributions. New York: Wiley, 1985.
103. N. Ueda and R. Nakano, "Deterministic annealing variant of the EM algorithm," in Proc. Neural Information Processing Systems. Nov. 1994, pp. 545-552.
104. S. R. Waterhouse and A. J. Robinson, "Non-linear prediction of acoustic vectors using hierarchical mixtures of experts," in Proc. Neural Inform. Processing Syst., vol. 7, pp. 835-842, 1994.
105. A. S. Weigend, M. Mangeas, and A. N. Srivastava, "Nonlinear gated experts for time series: Discovering regimes and avoiding overfilling," Int. J. Neural Syst., vol. 6, no. 4, pp. 373-399, 1995.
106. Y.-F. Wong, "Clustering data by melting," Neural Computation, vol. 5, no. 1, pp. 89-104, 1993.
107. E. Yair, K. Zeger, and A. Gersho, "Competitive learning and soft competition for vector quantizer design," IEEE Trans. Signal Processing, vol. 40, pp. 294-309, Feb. 1992.
108. A. L. Yuille, "Generalized deformable models, statistical physics, and matching problems," Neural Computation, vol. 2, no. 1, pp. 1-24, 1990.
109. K. Zeger and A. Gersho, "Vector quantizer design for memoryless noisy channels," in Proc. IEEE Int. Conf. Communications, Philadelphia, 1988, pp. 1693-1597.
110. J. Zhao and J. Shawe-Taylor, "Neural network optimization for good generalization performance," in Proc. Int. Conf. Artificial Neural Networks, 1994, pp. 561-564.