Physics of the Shannon limits

IEEE Transactions on Information Theory

Volumn 56, Issue 9, 2010, Pages 4274-4285

  Merhav, Neri ^a  

^a TECHNION ISRAEL INSTITUTE OF TECHNOLOGY   (Israel)

Author keywords

Data processing theorem; divergence; entropy; Gibbs' inequality; mutual information; relative entropy; second law of thermodynamics

Indexed keywords

DIVERGENCE; GIBBS' INEQUALITY; MUTUAL INFORMATIONS; RELATIVE ENTROPY; SECOND LAW OF THERMODYNAMICS;

ENTROPY; INFORMATION THEORY; PROBABILITY DISTRIBUTIONS; THERMODYNAMICS; TITRATION;

DATA PROCESSING;

EID: 77955751016     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2010.2053867     Document Type: Article

References (40)

1. R. Albert and A.-L. Barabási, Statistical Mechanics of Complex Networks, arXiv:cond-mat/0106096, Jun. 6, 2001.
2. R. J. Baxter, Exactly Solvable Models in Statistical Mechanics. New York: Academic, 1982.
3. J. P. Burg, "Maximum entropy spectral analysis," Ph.D. , Dep. Geophys., Stanford Univ., Stanford, CA, 1975.
4. Y. Ephraim and N. Merhav, "Hidden Markov processes," IEEE Trans. Inf. Theory, vol.48, no.6, pp. 1518-1569, Jun. 2002, Special Issue In Memory of Aaron D. Wyner.
5. D. Guo and S. Verdú, "Multiuser detection and statistical physics," in Communications, Information and Network Security, V. Bhargava, H. V. Poor, V. Tarokh, and S. Yoon, Eds. Norwell, MA: Kluwer, 2002, ch. 13, pp. 229-277.
6. D. Guo, S. Shamai, and S. Verdú, "Mutual information and minimum mean-square error in Gaussian channels," IEEE Trans. Inf. Theory, vol.51, no.4, pp. 1261-1282, Apr. 2005.
7. R. M. Gray, A. H. Gray, G. Rebolledo, and J. E. Shore, "Rate distortion speech coding with a minimum discrimination information distortion measure," IEEE Trans. Inf. Theory, vol.IT-27, no.6, pp. 708-721, Nov. 1981.
8. T. M. Cover and J. A. Thomas, Elements of Information Theory. New York: Wiley, 2006.
9. J. Horowitz and C. Jarzynski, "An illustrative example of the relationship between dissipation and relative entropy," arXiv:0901.0576v1 [cond-mat.stat-mech], Jan. 5, 2009.
10. T. Hosaka and Y. Kabashima, "Statistical mechanical approach to error exponents of lossy data compression," J. Phys. Soc. Japan, vol.74, no.1, pp. 488-497, Jan. 2005.
11. C. Jarzynski, "Nonequilibrium equality for free energy differences," Phys. Rev. Lett., vol.78, no.14, pp. 2690-2693, Apr. 7, 1997.
12. E. T. Jaynes, "Information theory and statistical mechanics," Phys. Rev. A, vol.106, pp. 620-630, May 1957.
13. E. T. Jaynes, "Information theory and statistical mechanics-II," Phys. Rev. A, vol.108, pp. 171-190, Oct. 1957.
14. E. T. Jaynes, "On the rationale of maximum-entropy methods," Proc. IEEE, vol.70, no.9, pp. 939-952, Sep. 1982.
15. Y. Kabashima and T. Hosaka, "Statistical mechanics for source coding with a fidelity criterion," Progr. Theoret. Phys., pp. 197-204, 2005, Suppl. no.157.
16. Y. Kabashima, K. Nakamura, and J. vanMourik, "Statistical mechanics of typical set decoding," Phys. Rev. E, vol.66, 2002.
17. Y. Kabashima and D. Saad, "Statistical mechanics of error correcting codes," Europhys. Lett., vol.45, no.1, pp. 97-103, 1999.
18. Y. Kabashima, N. Sazuka, K. Nakamura, and D. Saad, "Tighter decoding reliability bound for Gallager's error-correcting code," Phys. Rev. E, vol.64, pp. 046113-1-046113-4, 2001.
19. I. Kanter and D. Saad, "Error-correcting codes that nearly saturate Shannon's bound," Phys. Rev. Lett., vol.83, no.13, pp. 2660-2663, Sep. 1999.
20. M. Kardar, Statistical Physics of Particles. Cambridge, U.K.: Cambridge Univ. Press, 2007.
21. R. Kawai, J. M. R. Parrondo, and C. Van den Broeck, "Dissipation: The phase-space perspective," Phys. Rev. Lett., vol.98, 2007, 080602.
22. T. S. Komatsu, N. Nakagawa, S.-I. Sasa, and H. Tasaki, "Representation of nonequilibrium steady states in large mechanical systems," J. Stat. Phys., vol.134, pp. 401-423, 2009.
23. R. Landauer, "Irreversibility and heat generation in the computing process," IBM J. Res. Dev., vol.5, pp. 183-191, 1961.
24. O. J. E. Maroney, The (absence of a) relationship between thermodynamic and logical reversibility, arXiv:physics/0406137, Jun. 27, 2004.
25. N. Merhav, D. Guo, and S. Shamai (Shitz), "Statistical physics of signal estimation in Gaussian noise: Theory and examples of phase transitions," IEEE Trans. Inf. Theory, vol.56, no.3, pp. 1400-1416, Mar. 2010.
26. M. Mézard and A. Montanari, Information, Physics, and Computation. Oxford, U.K.: Oxford Univ. Press, 2009.
27. T. Mora and O. Rivoire, Statistical Mechanics of Error Exponents for Error-Correcting Codes, arXiv:cond-mat/0606696, Jun. 26, 2006.
28. T. Mutayama, "Statistical mechanics of the data compression theorem," J. Phys. A: Math. Gen., vol.35, pp. L95-L100, 2002.
29. P. Pradhan, Y. Kafri, and D. Levine, Non-Equilibrium Fluctuation Theorems in the Presence of Local Heating, arXiv:0712.0339v2 [cond-mat. stat-mech], Apr. 3, 2008.
30. M. B. Plenio and V. Vitelli, "The physics of forgetting: Landauer's erasure principle and information theory," Contemp. Phys., vol.42, no.1, pp. 25-60, 2001.
31. A. Procacci and B. Scoppola, "Statistical mechanics approach to coding theory," J. Statist. Phys., vol. 96, no. 3/4, pp. 907-912, 1999.
32. I. Rojdestvenski and M. C. Cottman, "Mapping of statistical physics to information theory with application to biological systems," J. Theor. Biol., pp. 43-54, 2000.
33. D. Samardzija, "Some analogies between thermodynamics and Shannon theory," in Proc. Conf. Inf. Sci. Syst. (CISS), Baltimore, MD, Mar. 2007, pp. 166-171.
34. J. E. Shore and R. W. Johnson, "Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy," IEEE Trans. Inf. Theory, vol.IT-26, no.1, pp. 26-37, Jan. 1980.
35. N. Sourlas, "Spin-glass models as error-correcting codes," Nature, vol.339, pp. 693-695, Jun. 1989.
36. N. Sourlas, "Spin glasses, error-correcting codes and finite-temperature decoding," Europhys. Lett., vol.25, pp. 159-164, 1994.
37. T. Tanaka, "Statistical mechanics of CDMA multiuser demodulation," Europhys. Lett., vol.54, no.4, pp. 540-546, 2001.
38. T. Tanaka, "A statistical-mechanics approach to large-system analysis of CDMA multiuser detectors," IEEE Trans. Inf. Theory, vol.48, no.11, pp. 2888-2910, Nov. 2002.
39. M. J. Wainwright, T. S. Jaakkola, and S. S. Willsky, "A new class of upper bounds on the log partition function," IEEE Trans. Inf. Theory, vol.51, no.7, pp. 2313-2335, Jul. 2005.
40. R. W. Yeung, A First Course in Information Theory. New York: Kluwer/Plenum, 2002.