Capturing the intangible concept of information

Journal of the American Statistical Association

Volumn 89, Issue 428, 1994, Pages 1243-1254

  Soofi, Ehsan S ^a  

^a UNIVERSITY OF WISCONSIN MILWAUKEE   (United States)

Author keywords

Chaos; Entropy; Kullback Leibler; Logit; Principal components

Indexed keywords

EID: 84950959794     PISSN: 01621459     EISSN: 1537274X     Source Type: Journal    
DOI: 10.1080/01621459.1994.10476865     Document Type: Article

References (71)

1. Akaike, H. (1973), “Information Theory and an Extension of the Maximum Likelihood Principle,” in 2nd International Symposium on Information Theory, eds. B. N. Petrov and F. Csaki, Budapest: Akademia Kiado, pp. 267–281.
2. Arizono, I., and Ohta, H. (1989), “A Test for Normality Based on Kullback–Leibler Information,” The American Statistician, 34, 20–23.
3. Ash, R. B. (1965), Information Theory, New York: Dover.
4. Berliner, L. M. (1992), “Statistics, Probability and Chaos” (with discussion), Statistical Science, 7, 69–90.
5. —— (1991), “Likelihood and Bayesian Prediction of Chaotic Systems,” Journal of the American Statistical Association, 86, 938–952.
6. Bernardo, J. M. (1979a), “Expected Information as Expected Utility,” The Annals of Statistics, 7, 686–690.
7. —— (1979b), “Reference Posterior Distributions for Bayesian Inference” (with discussion), Journal of the Royal Statistical Society, Ser. B, 41, 113–147.
8. Bozdogan, H. (1990), “On the Information-Based Measure of Covariance Complexity and Its Application to the Evaluation of Multivariate Linear Models,” Communications in Statistics, Part A—Theory and Methods, 19, 221–278.
9. Brockett, P. L. (1991), “Information-Theoretic Approach to Actuarial Science: A Unification and Extension of Relevant Theory and Applications,” (with discussion) Transactions of the Society of Actuaries, 43, 73–135.
10. Brooks, R. J. (1982), “On the Loss of Information Through Censoring,” Biometrika, 69, 137–144.
11. Chandra, M., De Wet, T., and Singpurwalla, N. D. (1982), “On the Sample Redundancy and a Test for Exponentiality,” Communications in Statistics, Part A—Theory and Methods, 11, 429–438.
12. Cover, T. M., and Thomas, J. A. (1991), Elements of Information Theory, New York: John Wiley.
13. Cramer, H. (1946), Mathematical Method of Statistics, Princeton, NJ: Princeton University Press.
14. Csiszar, I. (1991), “Why Least Squares and Maximum Entropy? An Axiomatic Approach to Inference in Linear Inverse Problems,” The Annals of Statistics, 19, 2032–66.
15. —— (1975) “I-Divergence Geometry of Probability Distributions and Minimization Problems,” Annals of Probability, 3, 146–158.
16. de Finetti, B. (1970), Theory of Probability, New York: John Wiley.
17. Donoho, D. L., Johnstone, I. M., Hoch, J. C, and Stern, A. S. (1992), “Maximum Entropy and Nearly Black Object” (with discussion), Journal of the Royal Statistical Society, Ser. B, 54, 41–81.
18. Dudewicz, E. J., and Van Der Meulen, E. C. (1981), “Entropy-Based Tests of Uniformity,” Journal of the American Statistical Association, 76, 967–974.
19. Ebrahimi, N., and Soofi, E. S. (1990), “Relative Information Loss Under Type II Censored Exponential Data,” Biometrika, 77, 429–35.
20. Ebrahimi, N., Habibullah, M., and Soofi, E. S. (1992), “Testing Exponentiality Based on Kullback–Leibler Information,” Journal of the Royal Statistical Society, Ser. B, 54, 739–748.
21. El-Sayyad, G. M. (1969), “Information and Sampling From Exponential Distribution,” Technometrics, 11, 4145.
22. Fisher, R. A. (1921), “On Mathematical Foundations of Theoretical Statistics,” Philosophical Transactions of the Royal Society of London, Ser. A, 222, 309–368.
23. Geweke, J. (1992), “Comment: Inference and Prediction in the Presence of Uncertainty and Determinism,” Statistical Science, 7, 94–101.
24. Goel, P. K. (1983), “Information Measures and Bayesian Hierarchical Models,” Journal of the American Statistical Association, 78, 408–410.
25. Goel, P. K., and DeGroot, M. H. (1979), “Comparison of Experiments and Information Measures,” The Annals of Statistics, 7, 1066–1077.
26. Gokhale, D. V. (1983), “On Entropy-Based Goodness-of-Fit Tests,” Computational Statistics and Data Analysis, 1, 157–165.
27. Gokhale, D. V., and Kullback, S. (1978), The Information in Contingency Tables, New York: Marcel Dekker.
28. Golan, A., Judge, G., and Jeffrey, P. (1993), “Recovering Information From Multinomial Response Data,” unpublished manuscript, University of California, Berkeley.
29. Good, I. J. (1950), Probability and the Weighing of Evidence, London: Griffin.
30. Haberman, S. J. (1984), “Adjustment by Minimum Discriminant Information,” The Annals of Statistics, 12, 971–988.
31. Hill, S. D., and Spall, J. C. (1987), “Noninformative Bayesian Priors for Large Samples Based on Shannon Information,” in Proceedings of the IEEE Conference on Decision and Control, New York: Institute of Electrical and Electronics Engineers, 1690–1693.
32. Jaynes, E. T. (1982), “On the Rationale of Maximum-Entropy Methods,” Proceedings of IEEE, 70, 939–952.
33. —— (1968), “Prior Probabilities,” IEEE Transactions on Systems Science and Cybernetics, SSC-4, 227–241.
34. —— (1957), “Information Theory and Statistical Mechanics,” Physical Review, 106, 620–630.
35. Jeffreys, H. (1946), “An Invariant Form for the Prior Probability in Estimation Problems,” Proceedings Royal Society London, Ser. A, 186, 453–461.
36. Johnson, W. (1987), “The Detection of Influential Observations for Allocation, Separation, and the Determination of Probabilities in a Bayesian Framework,” Journal of Business & Economic Statistics, 5, 369–381.
37. Johnson, W., and Geisser, S. (1982), “Assessing the Predictive Influence of Observations,” in Essays in Honor of C. R. Rao, eds. Kallianpur, Krishnaiah, and Ghosh, Amsterdam: North-Holland.
38. —— (1983), “A Predictive View of the Detection and Characterization of Influential Observations in Regression Analysis,” Journal of the American Statistical Association, 78, 137–144.
39. Kapur, J. N. (1989), Maximum Entropy Models in Science and Engineering, New York: John Wiley.
40. Kruskal, W. (1987), “Relative Importance by Averaging Over Orderings,” The American Statistician, 41, 6–10.
41. Kullback, S. (1959), Information Theory and Statistics, New York: John Wiley (reprinted in 1968 by Dover).
42. Kullback, S., and Leibler, R. A. (1951), “On Information and Sufficiency,” The Annals of Mathematical Statistics, 22, 79–86.
43. Lindley, D. V. (1956), “On a Measure of the Information Provided by an Experiment,” The Annals of Mathematical Statistics, 27, 986–1005.
44. —— (1961), “The Use of Prior Probability Distributions in Statistical Inference and Decision,” Proceedings of the Fourth Berkeley Symposium, 1, 436–468.
45. Loh, W. (1985), “A Note on the Geometry of Kullback-Leibler Information Numbers,” Communications in Statistics, Part A—Theory and Methods, 14, 895–904.
46. Mazzuchi, T. A., Soofi, E. S., and Soyer, R. (1993), “Information Diagnostics for Bayesian Reliability Analysis,” Proceedings of the Section on Bayesian Statistical Science of the American Statistical Association, pp. 173–178.
47. McCulloch, R. E. (1989), “Local Model Influence,” Journal of the American Statistical Association, 84, 473–478.
48. Rissanen, J. (1987), “Stochastic Complexity” (with discussion), Journal of the Royal Statistical Society, Ser. B, 49, 223–239.
49. —— (1986), “Stochastic Complexity and Modeling,” The Annals of Statistics, 14, 1080–1100.
50. Ruelle, D. (1989), Chaotic Evolution and Strange Attractors: The Statistical Analysis of Time Series for Deterministic Nonlinear Systems, Cambridge, U.K.: Cambridge University Press.
51. Ryu, H. K. (1993), “Maximum Entropy Estimation of Density and Regression Functions,” Journal of Econometrics, 56, 397–440.
52. Sawa, T. (1978), “Information Criteria for Discriminating Among Alternative Regression Models Econometrica, 46, 1273–1292.
53. Shannon, C. E. (1948), “A Mathematical Theory of Communication,” Bell System Technical Journal, 27, 379–423.
54. Shore, J. E., and Johnson, R. W. (1980), “Axiomatic Derivation of the Principle of Maximum Entropy and the Principle of Minimum Cross-Entropy,” IEEE Transactions on Information Theory, IT-26, 26–37.
55. Soofi, E. S. (1988), “Principal Component Regression Under Exchangeability,” Communications in Statistics, Part A—Theory and Methods, 17, 1717–1733.
56. —— (1990), “Effects of Collinearity on Information About Regression Coefficients,” Journal of Econometrics, 43, 255–274.
57. —— (1992), “A Generalizable Formulation of Conditional Logit with Diagnostics,” Journal of the American Statistical Association, 87, 412–816.
58. —— (1995), “The Information Theory and Bayesian Statistics,” in Bayesian Statistics and Applications: A Tribute to Arnold Zellner, eds. D. Berry, K. Chaloner, and J. Geweke, New York: John Wiley.
59. Soofi, E., Ebrahimi, N., and Habibullah, M. (1995), “Information Distinguishability with Application to Analysis of Failure Data,” submitted to Journal of the American Statistical Association.
60. Soofi, E. S., and Gokhale, D. V. (1991), “Minimum Discrimination Information Estimator of the Mean With Known Coefficient of Variation,” Computational Statistics and Data Analysis, 11, 165–177.
61. Spall, J. C., and Hill, S. D. (1990), “Least-Informative Bayesian Prior Distributions for Finite Samples Based on Information Theory,” IEEE Transactions on Automatic Control, 35, 580–583.
62. Stone, M. (1959), “Application of a Measure of Information to the Design and Comparison of Regression Experiments,” The Annals of Mathematical Statistics, 30, 55–70.
63. Turrero, A. (1989), “On the Relative Efficiency of Grouped and Censored Survival Data,” Biometrika, 76, 125–131.
64. Vasicek, O. (1976), “A Test for Normality Based on Sample Entropy,” Journal of the Royal Statistical Society, Ser. B, 38, 54–59.
65. Wegman, E. J. (1988), “On Randomness, Determinism and Computability” Journal of Statistical Planning and Inference, 20, 279–294.
66. —— (1990), “Hyperdimensional Data Analysis Using Parallel Coordinates,” Journal of the American Statistical Association, 85, 664–675.
67. Young, A. S. (1987), “On the Information Criterion for Selecting Regressors,” Metrika, 34, 185–194.
68. Zellner, A. (1971), An Introduction to Bayesian Inference in Econometrics, New York: John Wiley (reprinted in 1987 by Krieger, Malabar, FL).
69. —— (1982), “On Assessing Prior Distributions and Bayesian Regression Analysis With g-Prior Distributions,” in Bayesian Decision Techniques: Essays in Honor of Bruno de Finetti, eds. P. Goel and A. Zellner, Amsterdam: North-Holland, pp. 233–243.
70. —— (1984), Basic Issues in Econometrics, Chicago: University of Chicago Press.
71. Zellner, A., and Min, C. (1992), “Bayesian Analysis, Model Selection and Prediction,” invited paper presented at a symposium in honor of E. T. Jaynes, H. G. B. Alexander Research Foundation, Graduate School of Business, University of Chicago.