Information Measures in Perspective

International Statistical Review

Volumn 78, Issue 3, 2010, Pages 383-412

  Ebrahimi, Nader ^a   Soofi, Ehsan S ^b   Soyer, Refik ^c  

^a NORTHERN ILLINOIS UNIVERSITY   (United States)

^b UNIVERSITY OF WISCONSIN MILWAUKEE   (United States)

^c GEORGE WASHINGTON UNIVERSITY   (United States)

Author keywords

Bayesian information; Dynamic information; Entropy; Kullback Leibler information; Mutual information

Indexed keywords

EID: 78650261469     PISSN: 03067734     EISSN: 17515823     Source Type: Journal    
DOI: 10.1111/j.1751-5823.2010.00105.x     Document Type: Article

References (127)

1. Abbas, A.E. (2006). Maximum entropy utility. Oper. Res., 54, 277-290.
2. Abe, S. & Rajagopal, A.K. (2001). Information theoretic approach to statistical properties of multivariate Cauchy-Lorentz distributions. J. Phys. A: Math. General, 34, 8727-8731.
3. Abel, P.S. & Singpurwalla, N.D. (1994). To survive or to fail: That is the question. Amer. Statist., 48, 18-21.
4. Aitchison, J. (1975). Goodness of prediction fit. Biometrika, 62, 547-554.
5. Aitchison, J. (1990). On coherence in parametric density estimation. Biometrika, 77, 905-908.
6. Alwan, L.C., Ebrahimi, N. & Soofi, E.S. (1998). Information-theoretic framework for statistical process control. Eur. J. Oper. Res., 111, 526-541.
7. Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle. In 2nd International Symposium on Information Theory, Eds. B.N. Petrov & F. Csaki, pp. 267-281. Budapest Akademiai Kiado.
8. Amaral-Turkman, M.A. & Dunsmore, I. (1985). Measures of information in the predictive distribution. In Bayesian Statistics, 2, Eds. J.M. Bernardo, M.H. DeGroot, D.V. Lindley & A.F.M. Smith, pp. 603-612. Amsterdam Elsevier.
9. Arnold, B.C., Balakrishnan, N. & Nagaraja, H.N. (1992). A First Course in Order Statistics. New York Wiley.
10. Asadi, M., Ebrahimi, N., Hamedani, G.G. & Soofi, E.S. (2004). Maximum dynamic entropy models. J. Appl. Probab., 41, 379-390.
11. Asadi, M., Ebrahimi, N., Hamedani, G.G. & Soofi, E.S. (2005). Dynamic minimum discrimination information models. J. Appl. Probab., 42, 643-660.
12. Barlow, R.E. & Hsiung, J.H. (1983). Expected information from a life test experiment. Statistician, 48, 18-21.
13. Barron, A.R. (1986). Entropy and the Central Limit Theorem. Ann. Probab., 14, 936-942.
14. Bernardo, J.M. (1979a). Expected information as expected utility. Ann. Statist., 7, 686-690.
15. Bernardo, J.M. (1979b). Reference posterior distribution for Bayesian inference. J. R. Statist. Soc. Ser. B, 41, 605-647 (with discussion).
16. Bernardo, J.M. (2005). Reference analysis. In Handbook of Statistics, 25, Eds. D. K. Dey & C. R. Rao, pp. 17-90. Amsterdam Elsevier.
17. Bernardo, J.M. & Rueda, R. (2002). Bayesian hypothesis testing: A reference approach. Internat. Statist. Rev., 70, 351-372.
18. Bozdogan, H. (1994). Engineering & Scientific Applications of Informational Modeling, Vol. 3, Proceedings of First US/Japan Conference on The Frontiers of Statistical Modeling: An Informational Approach. Netherlands Kluwer.
19. Brockett, P.L. (1991). Information theoretic approach to actuarial science: A unification and extension of relevant theory and applications. Trans. Soci. Actuaries, 43, 73-135.
20. Brockett, P.L., Charnes, A., Cooper, W.W., Learner, D. & Phillips, F.Y. (1995). Information theory as a unifying statistical approach for use in marketing research. Eur. J. Oper. Res., 84, 310-329.
21. Burbea, J. & Rao, C.R. (1982). On the convexity of some divergence measures based on entropy functions. IEEE Trans. Info. Theory, 28, 489-495.
22. Burnham, K.P. & Anderson, D.R. (1998). Model Selection and Inference: A Practical Information-Theoretic Approach. New York Springer-Verlag.
23. Chaloner, K. & Verdinelli, I. (1995). Bayesian experimental design: A review. Statist. Sci., 10, 273-304.
24. Cover, T.M. & Thomas, J.A. (1991). Elements of Information Theory. New York Wiley.
25. Carlin, B.P. & Polson, N.G. (1991). An expected utility approach to influence diagnostics. J. Amer. Statist. Assoc., 87, 1013-1021.
26. Carota, C., Parmigiani, G. & Polson, N.G. (1996). Diagnostic measures for model criticism. J. Amer. Statist. Assoc., 91, 753-762.
27. Cox, D.R. (1961). Tests of separate families of hypotheses. In Proceedings of the Fourth Berkeley Symposium, Vol. 1, pp. 105-123. Berkeley UC Press.
28. Cressie, N. & Read, T.R.C. (1984). Multinomial goodness-of-fit tests. J. R. Statist. Soc. Ser. B, 46, 440-464.
29. Csiszar, I. (1975). I-divergence geometry of probability distributions. Ann. Probab., 3, 146-159.
30. Csiszar, I. (1991). Why least squares and maximum entropy? An axiomatic approach to inference in linear inverse problems. Ann. Statist., 19, 2032-2066.
31. Darbellay, G.A. & Vajda, I. (2000). Entropy expressions for multivariate continuous distributions. IEEE Trans. Info. Theory, 46, 709-712.
32. DeGroot, M.H. (1962). Uncertainty, information, and sequential experiments. Ann. Math. Statist., 33, 404-419.
33. Di Crescenzo, A. & Longobardi, M. (2002). Entropy-based measure of uncertainty in past lifetime distributions. J. Appl. Probab., 39, 434-440.
34. Di Crescenzo, A. & Longobardi, M. (2004). A measure of discrimination between past life-time distributions. Statist. Probab. Lett., 67, 173-182.
35. Ebrahimi, N. (1996). How to measure uncertainty in the residual lifetime distributions. Sankhya A, 58, 48-57.
36. Ebrahimi, N. & Kirmani, S.N.U.A. (1996). A characterization of the proportional hazards model through a measure of discrimination between two residual life distributions. Biometrika, 83, 233-235.
37. Ebrahimi, N. & Soofi, E.S. (2004). Information measures for reliability. In Mathematical Reliability: An Expository Perspective, Eds. R. Soyer, T.A. Mazzuchi & N.D. Singpurwalla, pp. 127-159. Netherlands Kluwer.
38. Ebrahimi, N., Kirmani, S.N.U.A. & Soofi, E.S. (2007). Dynamic multivariate information. J. Multivariate Anal., 98, 328-349.
39. Ebrahimi, N., Maasoumi, E. & Soofi, E.S. (1999). Ordering univariate distributions by entropy and variance. J. Econometrics, 90, 317-336.
40. Ebrahimi, N., Soofi, E.S. & Soyer, R. (2008). Multivariate maximum entropy identification, transformation, and dependence. J. Multivariate Anal., 99, 1217-1231.
41. Ebrahimi, N., Soofi, E.S. & Zahedi, H. (2004). Information properties of order statistics and spacings. IEEE Trans. Inform. Theory, 50, 177-183.
42. Esteban M.D. & Morales, D. (1995). A summary on entropy statistics. Kybernetica, 31, 337-346.
43. Ewens, W.J. & Grant, G.R. (2005). Statistical Methods in Bioinformatics: An Introduction, 2nd ed. New York Springer.
44. Fraser, D.A.S. (1965). On information in statistics. Ann. Math. Statist., 36, 890-896.
45. Fomby, T.B. & Hill, R.C. (1997). Advances in Econometrics: Applying Maximum Entropy to Econometric Problems. 12, New York JAI Press.
46. Gabriele, K.-I. (1999). A note on conditional logics and entropy. Int. J. Approx. Reason., 19, 231-246.
47. Geisser, S. (1993). Predictive Inference: An Introduction. New York Chapman-Hall.
48. Goel, P.K. & DeGroot, M.H. (1981). Information about hyperparameters in hierarchical models. J. Amer. Statist. Assoc., 76, 140-147.
49. Gokhale, D.V. & Kullback, S. (1978). The Information in Contingency Tables. New York Marcel Dekker.
50. Golan, A. (2006). Information and entropy econometrics-A review and synthesis. Found. Trends Econometrics, 2, 1-145.
51. Golan, A., Judge, G.G. & Miller, D. (1996). Maximum Entropy Econometrics: Robust Estimation with Limited Data. New York Wiley.
52. Good, I.J. (1971). Discussion of "measuring information and uncertainty". By R.J. Buehler. In Foundations of Statistical Inference, Eds. V.P. Godambe & D.A. Sprott, pp. 337-339. Toronto Holt, Rinehart & Winston.
53. Granger, C.W., Maasoumi, E. & Racine, J. (2004). A dependence metric for possibly nonlinear processes. J. Time Ser. Anal., 25, 649-669.
54. Haff, L.R. (1980). Empirical Bayes estimation of the multivariate normal covariance matrix. Ann. Statist., 8, 586-597.
55. Hastie, T. (1987). A closer look at the deviance. Amer. Statist., 41, 16-20.
56. Hirschberg, J., Maasoumi, E. & Slottje, D.J. (1991). Cluster analysis and the quality of life across countries. J. Econometrics, 50, 131-150.
57. Ibrahim, J.G., Chen, M.-H. & Sinha, D. (2003). On optimality of the power prior. J. Amer. Statist. Assoc., 98, 204-213.
58. Imbens, G., Johnson, P. & Spady, R. (1998). Information theoretic approaches to inference in moment condition models. Econometrica, 66, 333-357.
59. Inaba, T. & Shirahata, S. (1986). Measures of dependence in normal models and exponential models by information gain. Biometrika, 73, 345-352.
60. James, W. & Stein, C. (1961). Estimation with quadratic loss function. In Proceedings of the Fourth Berkeley Symposium, Vol. 1, pp. 361-375, Berkeley UC Press.
61. Jaynes, E.T. (1957). Information theory and statistical mechanics. Phys. Rev., 106, 620-630.
62. Jaynes, E.T. (1968). On the rationale of maximum-entropy methods. Proc. IEEE, 70, 939-952.
63. Jeffreys, H. (1946). An invariant form for the prior probability in estimation problems. Proc. R. Soc. A, 186, 453-461.
64. Johnson, W. & Geisser, S. (1983). A predictive view of the detection and characterization of influential observations in regression analysis. J. Amer. Statist. Assoc., 78, 137-144.
65. Joe, H. (1989). Relative entropy measures of multivariate dependence. J. Amer. Statist. Assoc., 84, 157-164.
66. Jizba, P. & Arimitsu, T. (2004). The world according to Rényi: Thermodynamics of multifractal systems. Ann. Phys., 312, 17-59.
67. Kapur, J.N. (1989). Maximum Entropy Models in Science and Engineering. New York Wiley.
68. Kapur, J.N. (1994). Measures of Information and Their Applications. New Dehli Wiley.
69. Karbelkar, S.N. (1986). On the axiomatic approach to the maximum entropy principle of inference. Pramana-J. Phys., 26, 301-310.
70. Karr, A.F., Kohnen, C.N., Oganian, A., Reiter, J.P. & Sanil, A.P. (2006). A framework for evaluating the utility of data altered to protect confidentiality. Amer. Statist., 60, 224-232.
71. Keller-McNulty, S., Nakhleh C.W. & Singpurwalla, N.D. (2005). A paradigm for masking (camouflaging) information. Internat. Statist. Rev., 73, 331-349.
72. Kent, J.T. (1982). Robust properties of likelihood ratio test. Biometrika, 69, 19-27.
73. Kent, J.T. (1983). Information gain and a general measure of correlation. Biometrika, 70, 163-173.
74. Keyes, T.K. & Levy, M.S. (1996). Goodness of prediction fit for multivariate linear models. J. Amer. Statist. Assoc., 91, 191-197.
75. Khinchin, A.I. (1957). Mathematical Foundations of Information Theory. New York Dover.
76. Kruskal, W. & Majors, R. (1989). Concepts of relative importance in scientific literature. Amer. Statist., 43, 2-6.
77. Kullback, S. (1954). Certain inequalities in information theory and the Cramer-Rao inequality. Ann. Math. Statist., 25, 745-751.
78. Kullback, S. (1959). Information Theory and Statistics. New York Wiley (reprinted in 1968 by Dover).
79. Kullback, S. (1987). The Kullback-Leibler distance. Amer. Statist., 41, 340.
80. Kullback, S. & Leibler, R. A. (1951). On information and sufficiency. Ann. Math. Statist., 22, 79-86.
81. Lindley, D.V. (1956). On a measure of information provided by an experiment. Ann. Math. Statist., 27, 986-1005.
82. Lindley, D.V. (1961). The use of prior probability distributions in statistical inference and decision. In Proc. Fourth Berkeley Symp., 1, pp. 436-468. Berkeley UC Press.
83. Maasoumi, E. (1993). Compendium to information theory in economics and econometrics. Econometric Rev., 12, 137-181.
84. McCulloch, R.E. (1989). Local model influence. J. Amer. Statist. Assoc., 84, 473-478.
85. Mazzuchi, T.A., Soofi, E.S. & Soyer, R. (2008). Bayes estimate and inference for entropy and information index of fit. Econometric Rev., 27, 428-456.
86. Nadarajah, S. & Zografos, K. (2003). Formulas for Rényi information and related measures for univariate distributions. Inform. Sci., 155, 119-138.
87. Nadarajah, S. & Zografos, K. (2005). Expressions for Rényi and Shannon entropies for bivariate distributions. Inform. Sci., 170, 173-189.
88. Park, S. (1995). The entropy of consecutive order statistics. IEEE Trans. Inform. Theory, 41, 2003-2007.
89. Pesaran, M.H. (1987). Global and Partial Non-Nested Hypotheses and Asymptotic Local Power. Econometric Theory, 3, 69-97.
90. Polson, N.G. (1992). On the expected amount of information from a nonlinear model. J. R. Statist. Soc. Ser. B, 54, 889-895.
91. Polson, N.G. (1993). A Bayesian perspective on the design of accelerated life tests. In Advances in Reliability, Ed. A. P. Basu, pp. 321-330. Amsterdam North Holland.
92. Polettini, S. (2003). Maximum entropy simulation for microdata protection. Statist. Comput., 13, 307-320.
93. Pourahmadi, M. & Soofi, E.S. (2000). Predictive variance and information worth of observations in time series. J. Time Ser. Anal., 21, 413-434.
94. Retzer, J.J., Soofi, E.S. & Soyer R. (2009). Information importance of predictors: Concepts, measures, Bayesian inference, and applications. Camput. Statist. Data Anal., 53, 2363-2377.
95. Rényi, A. (1961). On measures of entropy and information. In Proc. Fourth Berkeley Symp., Vol. 1, pp. 547-561. Berkeley UC Press.
96. Ryu, H.K. (1993). Maximum entropy estimation of density and regression functions. J. Econometrics, 56, 397-440.
97. San Martini, A. & Spezzaferri, F. (1984). A predictive model selection criteria. J. R. Statist. Soc. Ser. B, 46, 296-303.
98. Shannon, C.E. (1948). A mathematical theory of communication. Bell Sys. Tech. J., 27, 379-423.
99. Shore, J.E. & Johnson, R.W. (1980). Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy. IEEE Trans. Inform. Theory, 26, 26-37.
100. Singpurwalla, N.D. (1992). A Bayesian perspective on Taguchi's approach to quality engineering and tolerance design. Inst. Ind. Eng. Trans., 24, 18-31 (with discussion).
101. Singpurwalla, N.D. (2006). Reliability and Risk: A Bayesian Perspective. New York Wiley.
102. Soofi, E.S. (1990). Effects of collinearity on information about regression coefficients. J. Econometrics, 43, 255-274.
103. Soofi, E.S. (1992). A generalizable formulation of conditional logit with diagnostics. J. Amer. Statist. Assoc., 87, 812-816.
104. Soofi, E.S. (1994). Capturing the intangible concept of information. J. Amer. Statist. Assoc., 89, 1243-1254.
105. Soofi, E.S. (1997). Information theoretic regression methods. In Advances in Econometrics: Applying Maximum Entropy to Econometric Problems, Eds. T.B. Fomby & R.C. Hill, Vol. 12, pp. 25-83. New York JAI Press.
106. Soofi, E.S. & Retzer, J.J. (2002). Information indices: Unification and applications. J. Econometrics, 107, 17-40.
107. Soofi, E.S., Ebrahimi, N. & Habibullah, M. (1995). Information distinguishability with application to analysis of failure data. J. Amer. Statist. Assoc., 90, 57-668.
108. Soofi, E.S., Retzer, J.J. & Yasai-Ardekani, M. (2000). A framework for measuring the importance of variables with applications to management research and decision models. Decis. Sci., 31, 595-625.
109. Spigelhalter, D.J., Best, N.G., Carlin, B.P. & van der Linde, A. (2002). Bayesian measures of model complexity and fit. J. R. Statist. Soc. Ser. B, 64, 583-616 (with discussion).
110. Stone, M. (1959). Application of a measure of information to the design and comparison of regression experiments. Ann. Math. Statist., 29, 55-70.
111. Teitler, S., Rajagopal, A.K. & Ngai, K.L. (1986). Maximum Entropy and reliability distributions. IEEE Trans. Reliab. 35, 391-395.
112. Theil, H. (1967). Economics and Information Theory. Amsterdam North-Holland.
113. Theil, H. & Chung C. (1988). Information-theoretic measures of fit for univariate and multivariate linear regressions. Amer. Statist., 42, 249-252.
114. Theil, H. & Fiebig, D.Z. (1984). Exploiting Continuity: Maximum Entropy Estimation of Continuous Distributions. Cambridge, MA Ballinger.
115. Tsallis, C. (1988). Possible generalization of Bolzmann-Gibbs statistics. J. Statist. Phys., 52, 479-487.
116. Tsallis, C. (1998). Generalized entropy-based criterion for consistent testing. Phys. Rev. E, 58, 1442-1445.
117. Verdinelli, I., Polson, N.G. & Singpurwalla, N.D. (1993). Shannon information and Bayesian design for prediction in accelerated life-testing. In Reliability and Decision Making, Eds. R.E. Barlow, C.A. Clarotti & F. Spizzichino, pp. 247-256. London Chapman Hall.
118. Vasicek, O. (1976). A test for normality based on sample entropy. J. R. Statist. Soc. Ser. B, 38, 54-59.
119. Wong, K.M. & Chen, S. (1990). The entropy of ordered sequences and order statistics. IEEE Trans. Inform. Theory, 36, 276-284.
120. Yuan, A. & Clarke, B. (1999). An information criterion for likelihood selection. IEEE Trans. Inform. Theory, 45, 562-571.
121. Zellner, A. (1971). An Introduction to Bayesian Inference in Econometrics. New York Wiley (reprinted in 1996 by Wiley).
122. Zellner, A. (1977). Maximal data information prior distributions. In New Developments in the Applications of Bayesian Methods, Eds. A. Aykac & C. Brumat, pp. 211-232. Amsterdam North Holland.
123. Zellner, A. (1988). Optimal information processing and Bayes' Theorem. Amer. Statist., 42, 278-284 (with discussion).
124. Zellner, A. (1996). Bayesian method of moments/instrumental variable (BMOM/IV) analysis of mean and regression models. In Prediction and Modelling Honoring Seymour Geisser, Eds. J. C. Lee, A. Zellner & W. O. Johnson, pp. 61-74. Netherlands Springer-Verlag.
125. Zellner, A. (1997). Bayesian Analysis in Econometrics and Statistics: The Zellner View and Papers. Cheltenham, UK Edward Elgar.
126. Zellner, A. (2002). Information processing and Bayesian analysis. J. Econometrics, 107, 41-50.
127. Zografos, K. & Nadarajah, S. (2005). Expressions for Rényi and Shannon entropies for multivariate distributions. Statist. Probab. Lett., 71, 71-84.