Mixture of experts regression modeling by deterministic annealing

IEEE Transactions on Signal Processing

Volumn 45, Issue 11, 1997, Pages 2811-2820

  Rao, Ajit V ^a   Miller, David ^a   Rose, Kenneth ^b   Gersho, Alien ^b  

^a University of California   (United States)

^b PENNSYLVANIA STATE UNIVERSITY   (United States)

Author keywords

Deterministic annealing; Mixture of experts; Neural networks; Regression

Indexed keywords

EXPERT SYSTEMS; LAGRANGE MULTIPLIERS; LEARNING ALGORITHMS; MATHEMATICAL MODELS; OPTIMIZATION; REGRESSION ANALYSIS;

DETERMINISTIC ANNEALING (DA); HIERARCHICAL MIXTURE OF EXPERTS (HME); MAXIMUM LIKELIHOOD (ML) METHOD; NORMALIZED RADIAL BASIS FUNCTIONS (NRBF);

NEURAL NETWORKS;

EID: 0031272103     PISSN: 1053587X     EISSN: None     Source Type: Journal    
DOI: 10.1109/78.650107     Document Type: Article

References (41)

1. L. Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone, Classification and Regression Trees. Belmont, ÇA: Wadsworth, 1984.
2. D. S. Broomhead and D. Löwe, "Multivariable functional interpolation and adaptive networks," Complex Syst., vol. 2, no. 3, pp. 321-355, June 1988.
3. S. Chen, S. A. Billings, C. F. N. Cowan, and P. M. Grant, "Nonlinear systems identification using radial basis functions," Int. J. Syst. Sei., vol. 21, no. 12, pp. 2513-2539, Dec. 1990.
4. V. Cherkassky, Y. Lee, and H. Lari-Najafi, "Self-organizing network for regression: Efficient implementation and comparative evaluation," in Int. Joint Conf. Neural Networks, 1991, vol. 1, pp. 79-84.
5. A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum-likelihood from incomplete data via the EM algorithm," J. R. Stat. Soc., B, vol. 39, pp. 1-38, 1977.
6. F. Diaz-de-Maria and A. R. Figueiras-Vidal, "Nonlinear prediction for speech coding using radial basis functions," in Proc. ICASSP, vol. 1, pp. 788-791, 1995.
7. H. Do-Tu and M. Installe, "Learning algorithms for nonparametric solution to the minimum probability of error classification problem," IEEE Trans. Comput., vol. C-27, pp. 648-659, 1978.
8. R. Durbin and D. Willshaw, "An analogue approach to the traveling salesman problem using an elastic net method," Nature, vol. 326, pp. 689-691, 1987.
9. F. Girosi and T. Poggio, "Networks and the best approximation property," Biolog. Cybern., vol. 63, no. 3, pp. 169-176, 1990.
10. D. Harrison and D. L. Rubinfeld, "Hedonic prices and the demand for clean air," J. Environ. Econom. Manag, vol. 5, pp. 81-102, 1978.
11. J. B. Hampshire and A. H. Waibel, "A novel objective function for improved phoneme recognition using time-delay neural networks," IEEE Trans. Neural Networks, vol. 1, pp. 216-228, 1990.
12. E. J. Hartman, J. D. Keeler, and J. M. Kowalski, "Layered neural networks with Gaussian hidden units as universal approximations," Neural Comput., vol. 2, no. 2, pp. 210-215, 1990.
13. Y. H. Hu, S. Palreddy, and W. J. Tompkins, "Customized ECG beat classifier using mixture of experts," Neural Networks Signal Process., pp. 45964, 1995.
14. J.-H. Hwang, S.-R. Lay, M. Maechler, R. D. Martin, and J. Schimert, "Regression modeling in back-propagation and projection pursuit learning," IEEE Trans. Neural Networks, vol. 5, pp. 342-353, May 1994.
15. R. A. Jacobs and M. I. Jordan, "Learning piecewise control strategies in a modular neural network architecture," IEEE Trans. Syst., Man Cybern., vol. 23, pp. 337-345, 1993.
16. R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton, "Adaptive mixtures of local experts," Neural Comput., vol. 3, no. 1, pp. 79-87, 1991.
17. M. I. Jordan and R. A. Jacobs, "Hierarchical mixtures of experts and the EM algorithm," Neural Comput., vol. 6, no. 2, pp. 181-214, 1994.
18. B. H. Juang and S. Katagiri, "Discriminative learning for minimum error classification," IEEE Trans. Signal Processing, vol. 40, pp. 3043-3054, 1992.
19. B. H. Juang, R. J. Perdue, Jr., and D. L. Thomson, "Déployable automatic speech recognition systems: Advances and challenges," AT&T Tech. J., vol. 74, pp. 45-56, 1995.
20. S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi, "Optimization by simulated annealing," Sei., vol. 220, pp. 671-680, 1983.
21. T. Kohonen, G. Barna, and R. Chrisley, "Statistical pattern recognition with neural networks: Benchmarking studies," in Proc. IEEE ICAW, 1988, vol. 1.
22. S. Lee and J. C.-J. Pan, "Unconstrained handwritten numeral recognition based on radial basis competitive and co-operative networks with spatiotemporal feature representation," IEEE Trans. Neural Networks, vol. 7, pp. 455-474, Mar. 1996.
23. J. A. Leonard and M. A. Kramer, "Radial basis function networks for classifying process faults," IEEE Contr. Syst. Mag., vol. 11, pp. 31-38, Apr. 1991.
24. Y. Linde, A. Buzo, and R. M. Gray, "An algorithm for vector quantizer design," IEEE Trans. Commun., vol. COMM-28, pp. 84-95, 1980.
25. G. C. McDonald and R. C. Schwing, "Instabilities of regression estimates relating air pollution to mortality," Technometr., vol. 15, pp. 463-482, 1973.
26. G. J. McLachlan and K. E. Basford, Mixture Models: Inference and Application to Clustering. New York: Marcel Dekker, 1988.
27. D. Miller, A. Rao, K. Rose, and A. Gersho, "A global optimization method for statistical classifier design," IEEE Trans. Signal Processing, vol. 44, pp. 3108-3122, Dec. 1996.
28. J. Moody and C. J. Darken, "Fast learning in networks of locally-tuned processing units," Neural Comput., vol. 1, no. 2, pp. 281-294, 1989.
29. N. Morgan and H. A. Boulard, "Neural networks for statistical recognition of continuous speech," Proc. IEEE, vol. 83, pp. 742-772, 1995.
30. B. Mulgrew, "Applying radial basis functions," IEEE Signal Processing Mag., vol. 13, pp. 50-65, Mar. 1996.
31. M. Niranjan and V. Kadirkamanathan, "A nonlinear model for time series prediction and signal interpolation," in Proc. ICASSP, 1991, vol. 3, pp. 1713-1716.
32. M. J. D. Powell, "Radial basis functions for multivariable interpolation: A review," in Algorithms for Approximation. Oxford, U.K.: Clarendon, 1987, pp. 143-167.
33. A. Rao, D. Miller, K. Rose, and A. Gersho, "An annealing approach to parsimonious modeling in statistical regression," submitted for publication.
34. K. Rose, E. Gurewitz, and G. C. Fox, "Vector quantization by deterministic annealing," IEEE Trans. Inform. Theory, vol. 38, pp. 1249-1258, 1992.
35. _, "Constrained clustering as an optimization method," IEEE Trans. Pattern Anal. Machine Intell., vol. 15, pp. 785-794, 1993.
36. _, "Statistical mechanics and phase transitions in clustering," Phys. Rev. Lett., vol. 65, pp. 945-948, 1990.
37. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, Parallel Distributed Processing. Cambridge, MA: MIT Press, 1986.
38. D. M. Titterington, A. F. M. Smith, and U. E. Makov, Analysis of Finite Mixture Distributions. New York: Wiley, 1985.
39. S. R. Waterhouse and A. J. Robinson, "Non-linear prediction of acoustic vectors using hierarchical mixtures of experts," Neural Inform. Process. Syst., vol. 7, pp. 835-842, 1995.
40. A. S. Weigend, M. Mangeas, and A. N. Srivastava, "Nonlinear gated experts for time series: Discovering regimes and avoiding overfilling," Int. J. Neural Syst., vol. 4, pp. 373-399, 1995.
41. L. Xu and M. I. Jordan, "On convergence properlies of the EM algorithm for Gaussian mixlures," Neural Comput., vol. 8, pp. 129-151, 1996.