Functional properties of minimum mean-square error and mutual information

IEEE Transactions on Information Theory

Volumn 58, Issue 3, 2012, Pages 1289-1301

  Wu, Yihong ^a   Verdú, Sergio ^b  

^a UNIVERSITY OF PENNSYLVANIA   (United States)

^b Princeton University   (United States)

Author keywords

Bayesian statistics; central limit theorem; Gaussian noise; minimum mean square error (MMSE); mutual information; non Gaussian noise

Indexed keywords

BAYESIAN STATISTICS; CENTRAL LIMIT THEOREM; MINIMUM MEAN-SQUARE ERROR; MUTUAL INFORMATIONS; NON-GAUSSIAN NOISE;

GAUSSIAN NOISE (ELECTRONIC); INFORMATION THEORY;

MEAN SQUARE ERROR;

EID: 84863264784     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2011.2174959     Document Type: Article

References (46)

1. Y.Wu and S. Verdú, "Functional properties of MMSE," in Proc. IEEE Int. Symp. Inf. Theory, Austin, TX, Jun. 2010, pp. 1453-1457.
2. D. Guo, Y. Wu, S. Shamai, and S. Verdú, "Estimation in Gaussian noise: Properties of the minimum mean-square error," IEEE Trans. Inf. Theory, vol. 57, no. 4, pp. 2371-2385, Apr. 2011.
3. C. Villani, Optimal Transport: Old and New. Berlin, Germany: Springer-Verlag, 2008.
4. S. Yüksel and T. Linder, "Optimization and convergence of observation channels in stochastic control," 2010 [Online]. Available: arxiv. org/abs/1009.3824
5. D. Guo, S. Shamai, and S. Verdú, "Mutual information and minimum mean-square error in Gaussian channels," IEEE Trans. Inf. Theory, vol. 51, no. 4, pp. 1261-1283, Apr. 2005.
6. V. M. Zolotarev, One-Dimensional Stable Distributions. Providence, RI: American Mathematical Society, 1986.
7. Y. A. Brychkov and A. P. Prudnikov, Integral Transforms of Generalized Functions. New York: Gordon and Breach, 1989.
8. P. Dupuis and R. S. Ellis, AWeak Convergence Approach to the Theory of Large Deviations. New York: Wiley-Interscience, 1997.
9. P. J. Huber, Robust Statistics. New York, NY: Wiley-Interscience, 1981.
10. G. L. Wise, "A note on a common misconception in estimation," Syst. Control Lett., vol. 5, pp. 355-0356, Apr. 1985.
11. Y. Wu and S. Verdú, "MMSE dimension," IEEE Trans. Inf. Theory, vol. 57, no. 8, pp. 4857-4879, Aug. 2011.
12. L. D. Brown, "Admissible estimators, recurrent diffusions, and insoluble boundary value problems," Ann. Math. Statist., vol. 42, no. 3, pp. 855-903, 1971.
13. S. Vallender, "Calculation of the Wasserstein distance between probability distributions on the line," Theory Probabil. Appl., vol. 18, pp. 784-786, 1974.
14. S. T. Rachev and L. Rüschendorf, Mass Transportation Problems: Vol. I: Theory. Berlin, Germany: Springer-Verlag, 1998.
15. Y. Wu, "Shannon theory for compressed sensing," Ph.D. dissertation, Dept. Elect. Eng., Princeton Univ., Princeton, NJ, 2011.
16. D. Guo, S. Shamai, and S. Verdú, The Interplay Between Information and Estimation Measures 2011, Preprint.
17. R. T. Rockafellar, Convex Analysis. Princeton, NJ: Princeton Univ. Press, 1970.
18. 2-Wasserstein distance," in Proceedings of Distributions with Given Marginals and Statistical Modelling. Norwell, MA: Kluwer, 2002, pp. 51-62.
19. R. M. Gray and D. L. Neuhoff, "Quantization," IEEE Trans. Inf. Theory, vol. 44, no. 6, pp. 2325-2383, Oct. 1998.
20. Y. Wu and S. Verdú, "The impact of constellation cardinality on Gaussian channel capacity," in Proc. 48th Annu. Allerton Conf. Commun., Control, Comput., Monticello, IL, Oct. 1, 2010, pp. 620-628.
21. A. I. Sakhanenko, "Estimates in an invariance principle," Proc. Inst. Math. Novosibirsk, vol. 45, no. 5, pp. 27-44, 1985.
22. E. Rio, "Upper bounds for minimal distances in the central limit theorem," Annales de l'Institut Henri Poincaré-Probabilités et Statistiques, vol. 45, no. 3, pp. 802-817, 2009.
23. A. R. Barron, "Entropy and the central limit theorem," Ann. Probabil., vol. 14, no. 1, pp. 336-342, 1986.
24. S. Artstein, K. Ball, F. Barthe, and A. Naor, "On the rate of convergence in the entropic central limit theorem," Probabil. Theory Related Fields, vol. 129, no. 3, pp. 381-390, 2004.
25. O. Johnson and A. Barron, "Fisher information inequalities and the central limit theorem," Probabil. Theory Related Fields, vol. 129, no. 3, pp. 391-409, 2004.
26. A. A. Borovkov and S. A. Utev, "On an inequality and a related characterization of the normal distribution," Theory Probabil. Appl., vol. 28, pp. 219-228, 1984.
27. S. A. Utev, "An application of integro-differential inequalities in probability theory," Siberian Adv. Math., vol. 2, pp. 164-199, 1992.
28. S. G. Bobkov, G. P. Chistyakov, and F. Götze, Rate of Convergence and Edgeworth-Type Expansion in the Entropic Central Limit Theorem 2011 [Online]. Available: arxiv.org/abs/1104.3994
29. J. M. Stoyanov, Counterexamples in Probability, 2nd ed. Chichester, U.K.: Wiley, 1997.
30. T. Tao, "Szemerédi's regularity lemma revisited," Contributions Discrete Math., vol. 1, no. 1, pp. 8-28, 2006.
31. R. Ahlswede, "The final form of Tao's inequality relating conditional expectation and conditional mutual information," Adv. Math. Commun., vol. 1, no. 2, pp. 239-242, 2007.
32. I. Csiszár and J. Körner, Information Theory: Coding Theorems for Discrete Memoryless Systems. New York: Academic, 1982.
33. R. Zamir, "A proof of the Fisher information inequality via a data processing argument," IEEE Trans. Inf. Theory, vol. 44, no. 3, pp. 1246-1250, May 1998.
34. M. N. Ghosh, "Uniform approximation of minimax point estimates," Ann. Math. Statist., vol. 35, no. 3, pp. 1031-1047, 1964.
35. J. G. Smith, "The information capacity of amplitude and variance-constrained scalar Gaussian channels," Inf. Control, vol. 18, pp. 203-219, 1971.
36. E. Gourdin, B. Jaumard, and B. MacGibbon, "Global optimization decomposition methods for bounded parameter minimax risk evaluation," SIAM J. Sci. Comput., vol. 15, pp. 16-35, 1994.
37. G. Casella and W. E. Strawderman, "Estimating a bounded normal mean," Ann. Statist., vol. 9, pp. 870-878, 1981.
38. M. Raginsky, "On the information capacity of Gaussian channels under small peak power constraints," in Proc. 46th Annu. Allerton Conf. Commun., Control, Comput., 2008, pp. 286-293.
39. P. J. Bickel, "Minimax estimation of the mean of a normal distribution when the parameter space is restricted," Ann. Statist., vol. 9, pp. 1301-1309, 1981.
40. P. J. Huber, "Fisher information and spline interpolation," Ann. Statist., vol. 2, pp. 1029-1033, 1974.
41. E. Uhrmann-Klingen, "Minimal Fisher information distributions with compact-supports," Sankhy-a: Indian J. Statist., Series A, vol. 57, pp. 360-374, 1995.
42. S. S. Shitz and S. Verdú, "Worst-case power-constrained noise for binary-input channels," IEEE Trans. Inf. Theory, vol. 38, no. 5, pp. 1494-1511, Sep. 1992.
43. R. Esposito, "On a relation between detection and estimation in decision theory," Inf. Control, vol. 12, pp. 116-120, 1968.
44. C. Hatsell and L. Nolte, "Some geometric properties of the likelihood ratio," IEEE Trans. Inf. Theory, vol. 17, no. 5, pp. 616-618, Sep. 1971.
45. P. Billingsley, Probability and Measure, 3rd ed. New York: Wiley, 1995.
46. M. Fozunbal, "On regret of parametric mismatch in minimum mean square error estimation," in Proc. 2010 IEEE Int. Symp. Inf. Theory, Austin, TX, Jun. 2010, pp. 1408-1412.