A global optimization technique for statistical classifier design

IEEE Transactions on Signal Processing

Volumn 44, Issue 12, 1996, Pages 3108-3122

  Miller, David ^a   Rao, Ajit V ^a   Rose, Kenneth ^a   Gersho, Allen ^a  

^a The Pennsylvania State University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; DATA STRUCTURES; ERRORS; OPTIMIZATION; PROBABILITY; STATISTICAL METHODS;

SHANNON'S ENTROPY; STATISTICAL CLASSIFIERS;

DIGITAL SIGNAL PROCESSING;

EID: 0030412880     PISSN: 1053587X     EISSN: None     Source Type: Journal    
DOI: 10.1109/78.553484     Document Type: Article

References (56)

1. L. Atlas et al. A performance comparison of trained multi-layer perceptions and trained classification trees Proc. IEEE vol. 78 pp. 1614-1619 1990.
2. G. L. Bilbro W. E. Snyder and R. C. Mann Mean-field approximation minimizes relative entropy J. Opt. Soc. Amer. vol. 8 pp. 290-294 1991.
3. L. Breiman J. H. Friedman R. A. Olshen and C. J. Stone Classification and Regression Trees The Wadsworth Statistics/Probability Series. Belmont CA: Wadsworth 1980.
4. D. Chandler Introduction to Modern Statistical Mechanics. Oxford UK: Oxford University Press 1987.
5. P. A. Chou Optimal partitioning for classification and regression trees IEEE Trans. Patt. Anal. Machinr Intell. vol. 13 pp. 340-354 1991.
6. I. Csiszar Sanov property generalized I-projection and a conditional limit theorem Annals Prob. vol. 12 pp. 768-793 1984.
7. A.P. Dempster N. M. Laird and D. B. Rubin Maximum-likelihood from incomplete data via the EM algorithm J. Roy. Stat. Soc. Ser. B vol. 39 pp. 1-38 1977.
8. H. Do-Tu and M. Installe Learning algorithms for nonparametric solution to the minimum probability of error classification problem IEEE Trans. Comput. vol. C27 pp. 648-659 1978.
9. R. O. Duda and P. E. Hart Pattern Classification and Scene Analysis. New York: Wiley-Interscience 1974.
10. D. Geiger and F. Girosi Parallel and deterministic algorithms from MRFs: Surface reconstruction IEEE Trans. Patt. Anal. Machine Intell. vol. 13 pp. 401-412 1991.
11. A. Gersho Optimal vector quantized nonlinear estimation in IEEE Int. Symp. Inform. Theory San Antonio TX 1993.
12. H. Guo and S. B. Gelfand Classification trees with neural network feature extraction IEEE Trans. Neural Net. vol. 3 pp. 923-933 1992.
13. J. B. Hampshire and A. H. Waibel A novel objective function for improved phoneme recognition using time-delay neural networks IEEE Trans. Neural Net. vol. 1 pp. 216-228 1990.
14. W. H. Highleyman Linear decision functions with application to pattern recognition Proc. IRE vol. 50 pp. 1501-1514 1962.
15. D. Hush J. M. Salas and B. Home Error surfaces for multi-layer perceptrons IEEE Trans. Syst. Man Cybern. vol. 22 1992.
16. D. R. Hush and B. G. Home Progress in supervised neural networks IEEE Signal Processing Mag. pp. 8-39 Jan. 1993.
17. E. T. Jaynes Information theory and statistical mechanics in R. D. Rosenkrantz Ed. Papers on Probability Statistics and Statistical Physics. Dordrecht The Netherlands: Kluwer 1989. (Reprint of the original 1957 papers in Phys. Rev.).
18. M. I. Jordan and R. A. Jacobs Hierarchical mixtures of experts and the EM algorithm Neural Comp. vol. 6 pp. 181-214 1994.
19. B. H. Juang and S. Katagiri Discriminative learning for minimum error classification IEEE Trans. Signal Processing vol. 40 pp. 3043-3054 1992.
20. S. Kirkpatrick C. D. Gelatt and M. P. Vecchi Optimization by simulated annealing Sci. vol. 220 pp. 671-680 1983.
21. T. Kohonen G. Barna and R. Chrisley Statistical pattern recognition with neural networks: Benchmarking studies in Proc. IEEE ICNN vol. 1 1988.
22. B. Kosko Neural networks and Fuzzy Systems. Englewood Cliffs NJ: Prentice-Hall 1992.
23. R. Lippmann Review of neural networks for speech recognition Neural Comput. vol. 1 pp. 1-39 1989.
24. _ Pattern classification using neural networks IEEE Commun. Mag. pp. 47-64 1989.
25. J. Makhoul A. El-Jaroudi and R. Schwartz Formation of disconnected decision regions with a single hidden layer in Proc. Int. Joint Conf. Neural Networks vol. 1 Washington DC 1989 pp. 455-460.
26. D. Miller An information-theoretic framework for optimization with applications in source coding and pattern recognition Ph.D. thesis Univ. of California Santa Barbara 1995.
27. D. Miller A. Rao K. Rose and A. Gersho An information-theoretic framework for optimization with application to supervised learning in Abs. Int. Symp. Inform. Theory 1995.
28. D. Miller K. Rose and P. A. Chou Deterministic annealing for trellis quantizer and HMM design using Baum-Welch re-estimation in IEEE Int. Conf. Acoust. Speech Signal Processing Adelaide Australia 1994.
29. J. Moody and C. J. Darken Fast learning in locally-tuned processing units Neural Comput. vol. 1 pp. 281-294 1989.
30. M. T. Musavi W. Ahmed K. H. Chan K. B. Faris and D. M. Hummels On the training of radial basis function classifiers Neural Net. vol. 5 pp. 595-604 1992.
31. V. Nedeljkovic A novel multilayer neural networks training algorithm that minimizes the probability of classification error IEEE Trans. Neural Net. vol. 4 pp. 650-659 1993.
32. J. R. Quinlan Simplifying decision trees Int. J. Man-Machine Studies vol. 27 pp. 221-234 1987.
33. A. Rao D. Miller K. Rose and A. Gersho Generalized vector quantization To be submitted for publication 1995.
34. M. D. Richard and R. P. Lippmann Neural network classifiers estimate Bayesian a posteriori probabilities Neural Comput. vol. 3 pp. 461-483 1991.
35. B. D. Ripley Neural networks and related methods for classification J. Royal Stat. Soc. Ser. B vol. 56 pp. 409-456 1994.
36. K. Rose Deterministic annealing clustering and optimization Ph.D. thesis Calif. Inst. of Technol. 1991.
37. K. Rose E. Gurewitz and G. C. Fox Statistical mechanics and phase transitions in clustering Phys. Rev. Lett. vol. 65 pp. 945-948 1990.
38. _ Vector quantization by deterministic annealing IEEE Trans. Inform. Theory vol. 38 pp. 1249-1258 1992.
39. _ Constrained clustering as an optimization method IEEE Trans. Patt. Anal. Machinr Intell. vol. 15 pp. 785-794 1993.
40. F. Rosenblatt The perceptron - A perceiving and recognizing automaton Tech. Rep. Cornell Univ. Ithaca NY 1957.
41. D. W. Ruck S. K. Rogers M. Kabrisky M. E. Oxley and B. W. Suter The multilayer perceptron as an approximation to a Bayes optimal discriminant function IEEE Trans. Neural Net. vol. 1 pp. 296-298 1990.
42. D. E. Rumelhart G. E. Hinton and R. J. Williams Parallel Distributed Processing. Cambridge MA: MIT Press 1986.
43. J. E. Shore and R. W. Johnson Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy IEEE Trans. Inform. Theory vol. IT26 pp. 26-37 1980.
44. P. D. Simic Statistical mechanics as the underlying theory of elastic and neural optimization Network vol. 1 pp. 89-103 1990.
45. _ Constrained nets for graph matching and other quadratic assignment problems Neural Comput. vol. 3 pp. 268-281 1991.
46. F. W. Smith Pattern classifier design by linear programming IEEE Trans. Comput. vol. C17 pp. 367-372 1968.
47. E. D. Sontag and H. J. Sussman Back propagation separates where perceptrons do Neural Net. vol. 4 pp. 243-249 1991.
48. L. Tarassenko and S. Roberts Supervised and unsupervised learning in radial basis function classifiers Proc. Inst. Elect. Eng. - Vis. Image Signal Processing vol. 141 pp. 210-216 1994.
49. B. A. Telfer and H. H. Szu Energy functions for minimizing misclassification error with minimum complexity networks Neural Net. vol. 7 pp. 809-817 1994.
50. J. M. Van Campenhout and T. M. Cover Maximum entropy and conditional probability IEEE Trans. Inform. Theory vol IT27 pp. 483-489 1981.
51. D. Van den Bout and T. K. Miller Graph partitioning using annealed neural networks IEEE Trans. Neural Net. pp. 192-203 1990.
52. E. A. Wan Neural network classification: a Bayesian interpretation IEEE Trans. Neural Net. vol. 4 pp. 303-305 1990.
53. L. F. A. Wessels and E. Barnard Avoiding false local minima by proper initialization of connections IEEE Trans. Neural Net. vol. 3 pp. 899-905 1992.
54. A. L. Yuille Generalized deformable models statistical physics and matching problems Neural Comp. vol. 2 pp. 1-24 1990.
55. A. L. Yuille P. Stolorz and J. Utans Statistical physics mixtures of distributions and the EM algorithm Neural Comput. vol. 6 pp. 334-340 1994.
56. J. Zhang The mean field theory in EM procedures for Markov random fields IEEE Trans. Signal Processing vol. 40 pp. 2570-2583 1992.