General statistical inference for discrete and mixed spaces by an approximate application of the maximum entropy principle

IEEE Transactions on Neural Networks

Volumn 11, Issue 3, 2000, Pages 558-573

  Yan, Lian ^b   Miller, David J ^a,c  

^a IEEE   (United States)

^b Athene Software Inc   (United States)

^c The Pennsylvania State University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CONTINUOUS AND DISCRETE FEATURE SPACES; MAXIMUM ENTROPY; MULTIPLE INFERENCE TASK; PROBABILISTIC EXPERT SYSTEM;

COMPUTATIONAL COMPLEXITY; COMPUTER SIMULATION; CONSTRAINT THEORY; INFERENCE ENGINES; LAGRANGE MULTIPLIERS; LEARNING ALGORITHMS; LEARNING SYSTEMS; OPTIMIZATION; PROBABILITY DISTRIBUTIONS; RANDOM PROCESSES; STATISTICAL METHODS; VECTOR QUANTIZATION;

MULTILAYER NEURAL NETWORKS;

EID: 0034187697     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/72.846727     Document Type: Article

References (41)

1. M. Ben-Bassat, "Multimembership and multiperspective classification: Introduction, applications, and a Bayesian model," IEEE Trans. Syst., Man, Cybern., vol. SMC-10, pp. 331-336, 1980.
2. A. L. Berger, S. A. Della Pietra, and V. J. Della Pietra, "A maximum entropy approach to natural language processing," Comput. Linguist., vol. 22, pp. 39-68, 1996.
3. G. Bilbro and D. Van den Bout, "Maximum entropy and learning theory," Neural Comp., vol. 4, pp. 839-853, 1992.
4. C. Blake, E. Keogh, and C. J. Merz, "UCI repository of machine learning databases, 1998," Univ. of Cal., Irvine, Dept. of Information and Computer Sciences, http://www.ics.uci.edu/∼mlearn/MLRepository.html.
5. L. Breiman, "Bagging predictors," Mach. Learn., vol. 24, pp. 123-140, 1996.
6. J. Buhmann and H. Kuhnel, "Vector quantization with complexity costs," IEEE Trans. Inform. Theory, vol. 39, pp. 1133-1145, 1994.
7. W. Buntine, "A guide to the literature on learning probabilistic networks from data," IEEE Trans. Knowl. Data Eng., vol. 8, pp. 195-210, 1996.
8. J. P. Burg, "Maximum entropy spectral analysis," Ph.D. dissertation, Dept. of Geophysics, Stanford Univ., Stanford, CA, 1975.
9. R. Caruana, "Learning many related tasks at the same time with back propagation," in NIPS 7, 1995, pp. 657-664.
10. P. Cheeseman, "A method of computing generalized Bayesian probability values for expert systems," in Proc. 8th Int. Joint Conf. A1, vol. 1, 1983, pp. 198-202.
11. C. Chow and C. Liu, "Approximating discrete probability distributions with dependence trees," IEEE Trans. Inform. Theory, vol. IT-14, pp. 462-467, 1968.
12. G. F. Cooper and E. Herskovits, "A Bayesian method for the induction of probabilistic networks from data," Mach. Learning, vol. 9, pp. 309-347, 1992.
13. T. M. Cover and J. A. Thomas, Elements of Information Theory. New York: Wiley, 1991.
14. T. G. Dietterich, "Machine-learning research: Four current directions," AI Mag., pp. 97-136, Winter 1997.
15. S. Dumais, J. Platt, D. Heckerman, and M. Sahami, "Inductive learning algorithms and representations for text categorization," in Proc. Conf. Info. and Knowl. Manag., 1998.
16. N. Friedman and M. Goldszmidt, "Building classifiers using Bayesian networks," in Proc. National Conf. A1, vol. 2, 1996, pp. 1277-1284.
17. R. Fung and B. D. Favero, "Applying Bayesian networks to information retrieval," Commun. ACM, vol. 38, no. 3, pp. 42-48, 1995.
18. W. B. Gevarter, "Automatic probabilistic knowledge acquisition from data,", NASA Tech. Mem. 88224, 1986.
19. S. Haykin, Neural Networks: A Comprehensive Foundation. Englewood Cliffs, NJ: Prentice-Hall, 1999.
20. D. Heckerman, A. Geiger, and D. M. Chickering, "Learning Bayesian networks: The combination of knowledge and statistical data," Mach. Learning, vol. 20, pp. 197-243, 1995.
21. D. E. Heckerman, J. S. Breese, and K. Rommelse, "Decision-theoretic troubleshooting," Commun. ACM, vol. 38, pp. 49-57, 1995.
22. D. E. Heckerman, E. J. Horvitz, and B. N. Nathwani, "Toward normative expert systems - Part 1: The Pathfinder project," Methods Info. Med., vol. 31, pp. 90-105, 1992.
23. E. Herskovits and G. Cooper, "Kutato: An entropy-driven system for construction of probabilistic expert systems from databases," in Proc. Uncertainty in AI 6, 1991, pp. 117-125.
24. C. S. Hood and C. Ji, "Proactive network fault detection," in Proc, IEEE INFOCOM, 1997, pp. 1147-1155.
25. E. T. Jaynes, Papers on Probability, Statistics and Statistical Physics. Dordrecht, The Netherlands: Reidel, 1982.
26. H. H. Ku and S. Kullback, "Approximating discrete probability distributions," IEEE Trans. Inform. Theory, vol. IT-15, no. 4, pp. 444-447, 1969.
27. M. Meila and M. I. Jordan, "Estimating dependency structure as a hidden variable," in Proc. NIPS 10, 1998, pp. 584-590.
28. D. J. Miller, A. V. Rao, K. Rose, and A. Gersho, "A global optimization technique for statistical classifier design," IEEE Trans. Signal Processing, vol. 44, no. 12, pp. 3108-3122, 1996.
29. D. J. Miller and L. Yan, "Approximate maximum entropy joint feature inference consistent with arbitrary lower order probability constraints: Application to statistical classification," Neural Computat., 1999, to be published.
30. D. J. Miller and L. Yan, "Approximate maximum entropy joint feature inference for discrete space classification," in Proc. Int. Joint Conf. Neural Networks, 1999, pp. 1419-1424.
31. J. Miller and L. Yan, "An approximate maximum entropy method for feature inference: Relation to other maximum entropy methods and to naive Bayes," in Princeton Conf. Information Sciences and Systems, 2000, pp. 1-6.
32. J. Pearl, Probabilistic Reasoning in Intelligent Systems: Networks for Plausible Inference. San Mateo, CA: Morgan Kauffman, 1988.
33. J. A. Reggia, Y. Peng, and S. Tuhrim, "Connectionist approach to diagnostic problem solving using causal networks," Inform. Sci., vol. 70, pp. 21-40, 1993.
34. B. D. Ripley, Pattern Recognition and Neural Networks. Cambridge, U.K.: Cambridge Univ. Press, 1996.
35. K. Rose, E. Gurewitz, and G. C. Fox, "Vector quantization by deterministic annealing," IEEE Trans. Inform. Theory, vol. 38, pp. 1249-1257, 1992.
36. J. E. Shore and R. W. Johnson, "Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy," IEEE Trans. Inform. Theory, vol. IT-26, pp. 26-37, 1980.
37. J. Sill and Y. S. Abu-Mostafa, "Monotonicity hints," in Proc. NIPS 8, 1996, pp. 634-640.
38. J. Skilling, Ed., Maximum Entropy and Bayesian Methods. Amsterdam, The Netherlands: Kluwer, 1988.
39. P. Smyth and R. M. Goodman, "An information theoretic approach to rule induction from databases," IEEE Trans. Knowl. Data Eng., vol. 4, pp. 301-316, 1992.
40. D. R. Wilson and T. R. Martinez, "Improved heterogeneous distance functions," J. AI Res., vol. 6, pp. 1-34, 1997.
41. S. C. Zhu, Y. N. Wu, and D. Mumford, "Minimax entropy principle and its application to texture modeling," Neural Comput., vol. 9, no. 8, pp. 1627-1660, 1997.