Axiomatic Derivation of the Principle of Maximum Entropy and the Principle of Minimum Cross-Entropy

IEEE Transactions on Information Theory

Volumn 26, Issue 1, 1980, Pages 26-37

  Shore, John E ^a   Johnson, Rodney W ^a  

^a NAVAL RESEARCH LABORATORY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

INFORMATION THEORY;

EID: 0018877134     PISSN: 00189448     EISSN: 15579654     Source Type: Journal    
DOI: 10.1109/TIT.1980.1056144     Document Type: Article

References (67)

1. E. T. Jaynes “Information theory and statistical mechanics I,” Phys. Rev., vol. 106, pp. 620–630, 1957.
2. E. T. Jaynes, “Information theory and statistical mechanics II,” Phys. Rev., vol. 108, pp. 171–190, 1957.
3. E. T. Jaynes, “Information theory and statistical mechanics,” in Statistical Physics, vol. 3, Brandeis Lectures, K. W. Ford, Ed. New York: Benjamin, 1963, pp. 182–218.
4. E. T. Jaynes, “Foundations of probability theory and statistical mechanics,” in Delaware Seminar in the Foundations of Science, Vol. I, M. Bunge, Ed. New York: Springer—Verlag, 1967, pp. 77–101.
5. O. C. de Beauregard and M. Tribus “Information theory and thermodynamics,” Helvetica Physica Acta, vol. 47, pp. 238–247, 1974.
6. M. Tribus, Thermostatics and Thermodynamics. Princeton, NJ: Van Nostrand, 1961.
7. A. Katz, Principles of Statistical Mechanics—The Information Theory Approach. New York: Freeman, 1967.
8. A. Hobson, Concepts in Statistical Mechanics. New York: Gordon and Breach, 1971.
9. I. J. Good “Maximum entropy for hypothesis formulation, especially for multidimensional contingency tables,” Annals Math. Stat., vol. 34, pp. 911–934, 1963.
10. J. P. Noonan, N. S. Tzannes, and T. Costello “On the inverse problem of entropy maximizations,” IEEE Trans. Inform. Theory, vol. IT-22, pp. 120–123, Jan. 1976.
11. M. Tribus, Rational Descriptions, Decisions, and Designs. New York: Pergamon, 1969.
12. M. Tribus, “The use of the maximum entropy estimate in the estimation of reliability,” in Recent Developments in Information and Decision Processes, R. E. Machol and D. Gray, Eds. New York: Macmillan, 1962, pp. 102–139.
13. V. E. Benes, Mathematical Theory of Connecting Networks and Telephone Traffic. New York: Academic, 1965.
14. A. E. Ferdinand “A statistical mechanics approach to systems analysis,” IBM J. Res. Develop., vol. 14, pp. 539–547, 1970.
15. J. E. Shore, “Derivation of equilibrium and time-dependent solutions to M/M/∞//N and M/M/∞ queueing systems using entropy maximization,” in 1978 Nat. Computer Conf. AFIPS Conf. Proc., pp. 483–487, 1978.
16. M. Chan “System simulation and maximum entropy,” Operations Research, vol. 19, pp. 1751–1753, 1971.
17. E. T. Jaynes, “New engineering applications of the information theory,” in Proceedings of the First Symposium on Engineering Applications of Random Function Theory and Probability, J. L. Bogdanoff, Ed. New York: Wiley, 1963, pp. 163–203.
18. M. Tribus and G. Fitts “The widget problem revisited,” IEEE Trans. Syst. Sci. Cybern., vol. SSC-4, pp. 241–248, 1968.
19. B. Ramakrishna Rau, “The exact analysis of models of program reference strings,” Stanford Elec. Lab., Stanford Univ., Stanford, CA, Tech. Rep. 124 (SU-SEL-77-003), Dec. 1976.
20. A. E. Ferdinand “A theory of general complexity,” Int. J. General Syst., vol. 1, pp. 19–33, 1974.
21. M. Takatsuji “An information theoretical approach to a system of interacting elements,” Biol. Cybern., vol. 17, pp. 207–210, 1975.
22. J. M. Cozzolino and M. J. Zahner “The maximum-entropy distribution of the future market price of a stock,” Operations Research, vol. 21, pp. 1200–1211, 1973.
23. E. T. Jaynes, “Probability theory in science and engineering,” Field Res. Lab., Socony Mobil Oil Co., Inc., Colloquium Lectures in Pure and Applied Science No. 4, 1958.
24. E. T. Jaynes, Probability Theory in Science and Engineering, unpublished lecture notes (available from Physics Dept., Washington Univ., St. Louis, MO), 1972.
25. E. T. Jaynes, “Prior probabilities,” IEEE Trans. Syst. Sci. Cybern., vol. SSC-4, pp. 227–241, 1968.
26. J. Burg, “Maximum entropy spectral analysis,” Ph.D. dissertation, Stanford Univ., Stanford, CA, Univ. Microfilms No. 75-25,499, 1975.
27. J. G. Abies “Maximum entropy spectral analysis,” Astron. Astrophys. Suppl., vol. 15, pp. 383–393, 1974.
28. T. J. Ulrych and T. N. Bishop “Maximum entropy spectral analysis and autoregressive decomposition,” Rev. Geophysics and Space Physics, vol. 43, no. 1, pp. 183–200, 1975.
29. S. J. Wenecke and L. R. D'Addario “Maximum entropy image reconstruction,” IEEE Trans. Comput., vol. C-26, pp. 351–364, 1977.
30. I. J. Good, Probability and the Weighing of Evidence. London: Griffin, 1950.
31. S. Kullback, Information Theory and Statistics. New York: Wiley, 1959.
32. A. Wehrl “Entropy,” Rev. Mod. Phys., vol. 50, no. 2, pp. 221–260, 1978.
33. A. Hobson and B. Cheng “A comparison of the Shannon and Kullback information measures,” J. Slat. Phys., vol. 7, no. 4, pp. 301–310, 1973.
34. R. W. Johnson “Axiomatic characterization of the directed divergences and their linear combinations,” IEEE Trans. Inform Theory, vol. IT-25, no. 6, pp. 709–716, Nov. 1979.
35. R. S. Ingarden and A. Kossakowski “The Poisson probability distribution and information therodynamics,” Bull. Acad. Polon. Sci. Ser. Sci. Math. Astronom Phys., vol. 9, no. 1, pp. 83–85, 1971.
36. D. V. Gokhale and S. Kullback, The Information in Contingency Tables. New York: Marcel Dekker, 1978.
37. G. S. Arnold and J. L. Kinsey “Information theory for marginal distributions: Application to energy disposal in an exothermic reaction,” J. Chem. Phys., vol. 67, no. 8, pp. 3530–3532, 1977.
38. R. L. Kashyap “Prior probability and uncertainty,” IEEE Trans. Inform. Theory, vol. IT-17, pp. 641–650, Nov. 1971.
39. R. W. Johnson “Comments on ‘Prior probability and uncertainty,’” IEEE Trans. Inform. Theory, vol. IT-25, pp. 129–132, 1979.
40. P. M. Lewis, II “Approximating probability distributions to reduce storage requirements,” Inform Contr., vol. 2, pp. 214–225, 1959.
41. J. E. Shore, “Minimum cross-entropy spectral analysis,” Naval Res. Lab., Washington, DC 20375, NRL Memo. Rep. 3921, Jan. 1979.
42. J. E. Shore and R. W. Johnson, “Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy,” Naval Res. Lab., Washington DC 20375, NRL Memo. Rep. 3898, Dec. 1978.
43. R. W. Johnson, “Determining probability distributions by maximum entropy and minimum cross-entropy,” in APL 1979 Conf. Proc., pp. 24–29, 1979.
44. C. E. Shannon, “A mathematical theory of communication,” Bell Syst. Tech. J., vol. 27, pp. 379–423, 623–656, 1948.
45. J. S. Rowlinson, “Probability, information, and entropy,” Nature, vol. 225, pp. 1196–1198, Mar. 28, 1970.
46. J. M. Jauch and J. G. Baron, “Entropy, information, and Szilard's paradox,” Helvetica Physica Acta, vol. 45, p. 220, 1972.
47. K. Friedman and A. Shimony “Jaynes' maximum entropy prescription and probability theory,” J. Stat. Phys., vol. 3, no. 4, pp. 381–384, 1971.
48. M. Tribus and H. Motroni, “Comments on the paper Jaynes' maximum entropy prescription and probability theory,” J. Stat. Phys., vol. 4, no. 2/3, pp. 227–228, 1972.
49. A. Hobson “The interpretation of inductive probabilities,” J. Stat. Phys., vol. 6, no. 2/3, pp. 189–193, 1972.
50. A. I. Khinchin, Mathematical Foundations of Information Theory. New York: Dover, 1957.
51. D. K. Faddeyev “On the concept of entropy of a finite probabilistic scheme,” Uspekhi Mat. Nauk, vol. 11, no. 1(67), pp. 227–231, 1956.
52. D. W. North, “The invariance approach to the probabilistic encoding of information,” Ph.D. dissertation, Stanford Univ., Stanford, CA, Univ. Microfilms No. 70-22,189, 1970.
53. I. M. Gel'fand and A. M. Yaglom, “Calculation of the amount of information about a random function contained in another such function,” Uspekhi Mat. Nauk, vol. 12, no. 1(73) (Trans: American Mathematical Society Translation, Series 2, vol. 12. Providence, RI: Amer. Math. Soc., 1959, pp. 199–246).
54. A. Hobson “A new theorem of information theory,” J. Stat. Phys., vol. 1, no. 3, pp. 383–391, 1969.
55. Pl. Kannappan “On Shannon's entropy, directed divergence, and inaccuracy,” Z. Wahrscheinlichkeitstheorie verw. Geb., vol. 22, pp. 95–100, 1972.
56. Pl. Kannappan, “On directed divergence and inaccuracy,” Z. Wahrscheinlichkeitstheorie verw. Geb., vol. 25, pp. 49–55, 1972.
57. J. Aczél and Z. Daróczy, On Measures of Information and Their Characterizations. New York: Academic, 1975.
58. J. Van Campenhout and T. M. Cover, “Maximum entropy and conditional probability,” Dept. Stat., Stanford Univ., Stanford, CA, Tech. Rep. No. 32, July 1978.
59. R. T. Cox “Probability, frequency, and reasonable expectation,” Am. J. Phys., vol. 14, pp. 1–13, 1946.
60. R. T. Cox, The Algebra of Probable Inference. Baltimore, MD: Hopkins, 1961.
61. L. Janossy “Remarks on the foundation of probability calculus,” Acta Phys. Acad. Hungar., vol. 4, pp. 333–349, 1955.
62. J. Aczél “A solution of some problems of K. Borsuk and L. Janossy,” Acta Phys. Acad. Hungar., vol. 4, pp. 351–362, 1955.
63. H. Jeffreys “An invariant form for the prior probability in estimation problems,” Proc. Roy. Soc., vol. A186, pp. 453–461, 1946.
64. J. Aczél, Lectures on Functional Equations and their Applications. New York: Academic, 1966.
65. C. T. Ng “Representation for measures of information with the branching property,” Inform. Contr., vol. 25, pp. 45–56, 1974.
66. I. Csiszár “I-divergence geometry of probability distributions and minimization problems,” Ann. Prob., vol. 3, pp. 146–158, 1975.
67. W. M. Elsasser “On quantum measurements and the role of the uncertainty relations in statistical mechanics,” Phys. Rev., vol. 52, pp. 987–999, 1937.