Statistical physics and information theory

Foundations and Trends in Communications and Information Theory

Volumn 6, Issue 1-2, 2009, Pages 1-212

  Merhav, Neri ^a  

^a TECHNION ISRAEL INSTITUTE OF TECHNOLOGY   (Israel)

Author keywords

[No Author keywords available]

Indexed keywords

ANALYSIS TOOLS; GRADUATE COURSE; GRADUATE STUDENTS; LECTURE NOTES; PROBLEM AREAS; RESEARCH TRENDS; STATISTICAL PHYSICS; TARGET AUDIENCE; TECHNION;

INFORMATION THEORY; PHYSICS; TEACHING;

STUDENTS;

EID: 79953147200     PISSN: 15672190     EISSN: 15672328     Source Type: Journal    
DOI: 10.1561/0100000052     Document Type: Article

References (127)

1. L.-P. Arguin, "Spin glass computations and Ruelle's probability cascades," arXiv:math-ph/0608045v1, August 17 2006.
2. A. Barg and G. D. Forney, Jr., "Random codes: minimum distances and error exponents," IEEE Transactions on Information Theory, vol. 48, no. 9, pp. 2568-2573, September 2002. (Pubitemid 35032681)
3. G. B. Baǧ ci, "The physical meaning of Rényi relative entropies," arXiv:condmat/0703008v1, March 1 2007.
4. R. J. Baxter, Exactly Solved Models in Statistical Mechanics. Academic Press, 1982.
5. A. H. W. Beck, Statistical Mechanics, Fluctuations and Noise. Edward Arnold Publishers, 1976.
6. C. H. Bennett, "Notes on Landauer's principle, reversible computation and Maxwell's demon," Studies in History and Philosophy of Modern Physics, vol. 34, pp. 501-510, 2003. (Pubitemid 36795105)
7. G. P. Beretta and E. Zanchini, "Rigorous and general definition of thermodynamic entropy," arXiv:1010.0813v1 [quant-ph], October 5 2010.
8. S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge University Press, 2004.
9. E. Buffet, A. Patrick, and J. V. Pulé, "Directed polymers on trees: A martingale approach," Journal of Physics A: Mathematical and General, vol. 26, pp. 1823-1834, 1993.
10. F. Cousseau, K. Mimura, and M. Okada, "Statistical mechanics of lossy compression for non-monotonic multilayer perceptron," in Proceedings of ISIT 2008, pp. 509-513, Toronto, Canada, July 2008.
11. F. Cousseau, K. Mimura, T. Omori, and M. Okada, "Statistical mechanics of lossy compression for non-monotonic multilayer perceptrons," Physical Review E, vol. 78, 021124 2008, arXiv:0807.4009v1 [cond-mat.stat-mech], 25 July 2008.
12. T. M. Cover and E. Ordentlich, "Universal portfolios with side information," IEEE Transactions on Information Theory, vol. IT-42, no. 2, pp. 348-363, March 1996. (Pubitemid 126770492)
13. T. M. Cover and J. A. Thomas, Elements of Information Theory. John Wiley & Sons, Second ed., 2006.
14. G. E. Crooks, "Beyond Boltzmann-Gibbs statistics: Maximum entropy hyperensembles out of equilibrium," Physical Review E, vol. 75, 041119, 2007.
15. I. Csiszár, "A class of measures of informativity of observation channels," Periodica Mathematica Hungarica, vol. 22, no. 1-4, pp. 191-213, 1972.
16. I. Csiszár and J. Körner, Information Theory: Coding Theorems for Discrete Earlier Memoryless Systems. New York: Academic Press, 1981.
17. B. Dacorogna and P. Maréchal, "The role of perspective functions in convexity, polyconvexity, rank-one convexity and separate convexity," http://caa.epfl.ch/publications/2008-The-role-of-perspective- functions-in-convexity.pdf.
18. W.-S. Dai and M. Xie, "The explicit expression of the fugacity for hard-sphere Bose and Fermi gases," arXiv:0906.0952v1 [cond-mat.stat-mech], 4 June 2009.
19. C. R. Davis and M. E. Hellman, "On tree coding with a fidelity criterion," IEEE Transactions on Information Theory, vol. IT-21, no. 4, pp. 373-378, July 1975.
20. N. G. de Bruijn, Asymptotic Methods in Analysis. Dover Publications, 1981.
21. C. De Dominicis, M. Gabay, T. Garel, and H. Orland, "White and weighted averages over solutions of Thouless Anderson Palmer equations for the Sherrington Kirkpatrick spin glass," Journal de Physique, vol. 41, pp. 923-930, 1980. (Pubitemid 11462177)
22. C. R. de Oliveira and T. Werlang, "Ergodic hypothesis in classical statistical mechanics," Revista Brasileira de Ensino de Fisíca, vol. 29, no. 2, pp. 189-201, Also available on-line: http://www.sbfisica.org.br/ rbef/pdf/060601.pdf, 2007.
23. B. Derrida, "The random energy model," Physics Reports (Review Section of Physics Letters), vol. 67, no. 1, pp. 29-35, 1980.
24. B. Derrida, "Random-energy model: Limit of a family of disordered models," Physical Review Letters, vol. 45, no. 2, pp. 79-82, July 1980.
25. B. Derrida, "Random-energy model: An exactly solvable model for disordered systems," Physical Review B, vol. 24, no. 5, pp. 2613-2626, September 1981.
26. B. Derrida, "A generalization of the random energy model which includes correlations between energies," Journal de Physique - Lettres, vol. 46, pp. L-401-L-407, May 1985.
27. B. Derrida and E. Gardner, "Solution of the generalised random energy model," Jounal of Physics C: Solid State Physics, vol. 19, pp. 2253-2274, 1986.
28. R. J. Dick, T. Berger, and F. Jelinek, "Tree encoding of Gaussian sources," IEEE Transformations on Information Theory, vol. IT-20, no. 3, pp. 332-336, May 1974.
29. M. Doi and S. F. Edwards, The Theory of Polymer Dynamics. Oxford University Press, 1986.
30. V. Dotsenko, "One more discussion on the replica trick: The examples of exact solutions," arXiv:1010.3913v1 [cond-mat.stat-mech], 19 October 2010.
31. R. S. Ellis, Entropy, Large Deviations, and Statistical Mechanics. Springer-Verlag, NY, 1985.
32. R. Etkin, N. Merhav, and E. Ordentlich, "Error exponents of optimum decoding for the interference channel," IEEE Transactions on Information Theory, vol. 56, no. 1, pp. 40-56, January 2010.
33. G. D. Forney, Jr., "Exponential error bounds for erasure, list, and decision feedback schemes," IEEE Transactions on Information Theory, vol. IT-14, no. 2, pp. 206-220, March 1968.
34. G. D. Forney, Jr. and A. Montanari, "On exponential error bounds for random codes on the DMC," manuscript, http://www.stanford.edu/ ̃montanar/PAPERS/FILEPAP/dmc.ps, 2001.
35. R. G. Gallager, Information Theory and Reliable Communication. John Wiley & Sons, 1968.
36. R. G. Gallager, "Tree encoding for symmetric sources with a distortion measure," IEEE Transactions on Information Theory, vol. IT-20, no. 1, pp. 65-76, January 1974.
37. R. M. Gray, Source Coding Theory. Kluwer Academic Publishers, 1990.
38. D. Guo and S. Verdú, "Randomly spread CDMA: Asymptotics via statistical physics," IEEE Transactions on Information Theory, vol. 51, no. 6, pp. 1982-2010, June 2005.
39. M. J. W. Hall, "Universal geometric approach to uncertainty, entropy, and information," Physical Review A, vol. 59, no. 4, pp. 2602-2615, April 1999. (Pubitemid 129690460)
40. J. Honerkamp, Statistical Physics - An Advanced Approach with Applications. Springer-Verlag, 2nd ed., 2002.
41. 4 models," Physical Review E, vol. 77, 041104, 2008.
42. T. Hosaka and Y. Kabashima, "Statistical mechanical approach to error exponents of lossy data compression," Journal of the Physical Society of Japan, vol. 74, no. 1, pp. 488-497, January 2005. (Pubitemid 41284400)
43. P. A. Humblet, "Generalization of Huffman coding to minimize the probability of buffer overflow," IEEE Transactions on Information Theory, vol. 27, pp. 230-232, March 1981.
44. C. Jarzynski, "Nonequilibrium equality for free energy differences," Physical Review Letters, vol. 78, no. 14, pp. 2690-2693, 7 April 1997. (Pubitemid 127655287)
45. E. T. Jaynes, "Information theory and statistical mechanics," Physical Review A, vol. 106, pp. 620-630, May 1957.
46. E. T. Jaynes, "Information theory and statistical mechanics-II," Physical Review A, vol. 108, pp. 171-190, October 1957.
47. F. Jelinek, "Buffer overflow in variable length coding of fixed rate sources," IEEE Transactions on Information Theory, vol. IT-14, no. 3, pp. 490-501, May 1968.
48. F. Jelinek, "Tree encoding of memoryless time-discrete sources with a fidelity criterion," IEEE Transactions on Information Theory, vol. IT-15, no. 5, pp. 584-590, September 1969.
49. F. Jelinek and J. B. Anderson, "Instrumentable tree encoding of information sources," IEEE Transactions on Information Theory, vol. IT-17, no. 1, pp. 118-119, January 1971.
50. Y. Kabashima, "How could the replica method improve accuracy of performance assessment of channel coding?," in Proceedings of the International Workshop on Statistical-Mechanical Informatics, Sendai, Japan. arXiv:0808.0548v1 [cs.IT], September 14-17 2008. 5 August 2008.
51. Y. Kabashima and T. Hosaka, "Statistical mechanics for source coding with a fidelity criterion," Progress of Theoretical Physics, no. 157, pp. 197-204, Supplement, 2005. (Pubitemid 40977634)
52. Y. Kabashima, K. Nakamura, and J. van Mourik, "Statistical mechanics of typical set decoding," Physical Review E, vol. 66, 2002.
53. Y. Kabashima and D. Saad, "Statistical mechanics of error correcting codes," Europhysics Letters, vol. 45, no. 1, pp. 97-103, 1999.
54. Y. Kabashima and D. Saad, "Statistical mechanics of low-density parity check codes," Journal of Physics A: Mathamatical and General, vol. 37, pp. R1-R43, 2004. (Pubitemid 38258507)
55. M. Kardar, Statistical Physics of Particles. Cambridge University Press, 2007.
56. Y. Kaspi and N. Merhav, "Error exponents of optimum decoding for the degraded broadcast channel using moments of type class enumerators," in Proceedings of the ISIT 2009, pp. 2507-2511, Seoul, South Korea, June-July 2009. Full version: available in arXiv:0906.1339.
57. R. Kawai, J. M. R. Parrondo, and C. V. den Broeck, "Dissipation: The phasespace perspective," Physical Review Letters, vol. 98, 080602, 2007.
58. F. P. Kelly, Reversibility and Stochastic Networks. John Wiley & Sons, 1979.
59. K. Kitagawa and T. Tanaka, "Optimal spreading sequences in large CDMA systems: a Proceedings of ISIT 2008," pp. 1373-1377, Toronto, Canada, July 2008.
60. C. Kittel, Elementary Statistical Physics. John Wiley & Sons, 1958.
61. R. Kubo, Statistical Mechanics. North-Holland, 1961.
62. L. D. Landau and E. M. Lifshitz, Course of Theoretical Physics - vol. 5: Statistical Physics, Part 1. Elsevier, 3rd ed., 1980.
63. R. Landauer, "Irreversibility and heat generation in the computing process," IBM Journal of Research and Development, vol. 5, pp. 183-191, 1961.
64. T. D. Lee and C. N. Yang, "Statistical theory of equations of state and phase transitions. II. Lattice gas and Ising model," Physical Review, vol. 87, no. 3, pp. 410-419, August 1952.
65. H. Löwen, "Fun with hard spheres," in Spatial Statistics and Statistical Physics, vol. 554, (K. Mecke and D. Stoyan, eds.), pp. 295-331, Berlin: Springer Lecture Notes in Physics, 2000.
66. N. Merhav, "Another look at the physics of large deviations with application to rate-distortion theory," http://arxiv.org/PS-cache/arxiv/ pdf/0908/0908.3562v1.pdf.
67. N. Merhav, "Universal coding with minimum probability of code word length overflow," IEEE Transactions on Information Theory, vol. 37, no. 3, pp. 556-563, May 1991. (Pubitemid 21646295)
68. N. Merhav, "Error exponents of erasure/list decoding revisited via moments of distance enumerators," IEEE Transactions on Information Theory, vol. 54, no. 10, pp. 4439-4447, October 2008.
69. N. Merhav, "An identity of Chernoff bounds with an interpretation in statistical physics and applications in information theory," IEEE Transactions on Information Theory, vol. 54, no. 8, pp. 3710-3721, August 2008.
70. N. Merhav, "The random energy model in a magnetic field and joint source-channel coding," Physica A: Statistical Mechanics and Its Applications, vol. 387, no. 22, pp. 5662-5674, September 15 2008.
71. N. Merhav, "The generalized random energy model and its application to the statistical physics of ensembles of hierarchical codes," IEEE Transactions on Information Theory, vol. 55, no. 3, pp. 1250-1268, March 2009.
72. N. Merhav, "Relations between random coding exponents and the statistical physics of random codes," IEEE Transactions on Information Theory, vol. 55, no. 1, pp. 83-92, January 2009.
73. N. Merhav, "Data processing theorems and the second law of thermodynamics," submitted to IEEE Transactions on Information Theory, 2010.
74. N. Merhav, "On the physics of rate-distortion theory," in Proceedings of ISIT 2010, pp. 71-75, Austin, Texas, U.S.A., June 2010.
75. N. Merhav, "On the statistical physics of directed polymers in a random medium and their relation to tree codes," IEEE Transactions on Information Theory, vol. 56, no. 3, pp. 1345-1350, March 2010.
76. N. Merhav, "Physics of the Shannon limits," IEEE Transactions on Information Theory, vol. 56, no. 9, pp. 4274-4285, September 2010.
77. N. Merhav, "Rate-distortion function via minimum mean square error estimation," accepted IEEE Transactions on Information Theory, November 2010.
78. N. Merhav, "Threshold effects in parameter estimation as phase transitions in statistical mechanics," submitted to IEEE Transactions on Information Theory, 2010.
79. N. Merhav, D. Guo, and S. S. (Shitz), "Statistical physics of signal estimation in Gaussian noise: Theory and examples of phase transitions," IEEE Transactions on Information Theory, vol. 56, no. 3, pp. 1400-1416, March 2010.
80. M. Mézard and A. Montanari, Information, Physics and Computation. Oxford University Press, 2009.
81. A. Montanari, "Turbo codes: The phase transition," The European Physical Journal B, vol. 18, p. 121, E-print: cond-mat/0003218, 2000.
82. A. Montanari, "The glassy phase of Gallager codes," The European Physical Journal B - Condensed Matter and Complex Systems, vol. 23, no. 1, pp. 121-136, E-print: cond-mat/0104079, 2001. (Pubitemid 33624573)
83. T. Mora and O. Rivoire, "Statistical mechanics of error exponents for errorcorrecting codes," arXiv:cond-mat/0606696, June 2006.
84. T. Murayama, "Statistical mechanics of the data compression theorem," Journal of the Physics A: Mathamatical General, vol. 35, pp. L95-L100, 2002. (Pubitemid 37026677)
85. K. R. Narayanan and A. R. Srinivasa, "On the thermodynamic temperature of a general distribution," arXiv:0711.1460v2 cond-mat.stat-mech], November 10 2007.
86. K. R. Narayanan and A. R. Srinivasa, "A Shannon entropy-based nonequilibrium temperature of a general distribution with application to Langevin dynamics," preprint, May 2009.
87. H. Nishimori, Statistical Physics of Spin Glasses and Information Processing: An Introduction. (International Series of Monographs on Physics, no. 111): Oxford University Press, 2001.
88. L. Onsager, "Crystal statistics. I. A two-dimensional model with an order-disorder transition," Physical Review, vol. 65, no. 2, pp. 117-149, 1944.
89. G. Parisi and F. Zamponi, "Mean field theory of the glass transition and jamming hard spheres," arXiv:0802.2180v2 [cond-mat.dis-nn], 18 December 2008.
90. P. Pradhan, Y. Kafri, and D. Levine, "Non-equilibrium fluctuation theorems in the presence of local heating," arXiv:0712.0339v2 [cond-mat.stat-mech], 3 April 2008.
91. A. Procacci and B. Scoppola, "Statistical mechanics approach to coding theory," Journal of Statistical Physics, vol. 96, no. 3/4, pp. 907-912, 1999.
92. H. Qian, "Relative entropy: Free energy associated with equilibrium fluctuations and nonequilibrium deviations," Physical Review E, vol. 63, 042103, 2001.
93. F. Reif, Fundamentals of Statistical and Thermal Physics. McGraw-Hill, 1965.
94. H. Reiss, "Thermodynamic-like transformations in Information Theory," Journal of Statistical Physics, vol. 1, no. 1, pp. 107-131, 1969.
95. H. Reiss, H. L. Frisch, and J. L. Lebowitz, "Statistical mechanics of rigid spheres," Journal of Chemical Physics, vol. 31, no. 2, pp. 369-380, August 1959.
96. H. Reiss and C. Huang, "Statistical thermodynamic formalism in the solution of information theory problems," Journal of Statistical Physics, vol. 3, no. 2, pp. 191-211, 1971.
97. H. Reiss and P. Schaaf, "Hard spheres: Thermodynamics and geometry," Journal of Chemical Physics, vol. 91, no. 4, pp. 2514-2524, 15 August 1989.
98. T. Richardson and R. Urbanke, Modern Coding Theory. Cambridge University Press, 2008.
99. K. Rose, "A mapping approach to rate-distortion computation and analysis," IEEE Transactions on Information Theory, vol. 40, no. 6, pp. 1939-1952, November 1994.
100. D. Ruelle, Statistical Mechanics: Rigorous Results. Addison-Wesley, 1989.
101. P. Ruján, "Finite temperature error-correcting codes," Physics Review Letters, vol. 70, no. 19, pp. 2968-2971, May 1993.
102. F. W. Sears, M. W. Zemansky, and H. D. Young, University Physics. Addison-Wesley, 1976.
103. J. P. Sethna, Statistical Mechanics: Entropy, Order Parameters, and Complexity. Oxford University Press, 2007.
104. O. Shental and I. Kanter, "Shannon capacity of infinite-range spin-glasses," Technical report, Bar Ilan University, 2005.
105. O. Shental, N. Shental, S. S. (Shitz), I. Kanter, A. J. Weiss, and Y. Weiss, "Discrete input two-dimensional Gaussian channels with memory: Estimation and information rates via graphical models and statistical mechanics," IEEE Transactions on Information Theory, vol. 54, no. 4, April 2008.
106. J. E. Shore and R. W. Johnson, "Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy," IEEE Transactions on Information Theory, vol. IT-26, no. 1, pp. 26-37, January 1980.
107. N. Simanyi, "The Boltzmann-Sinai ergodic hypothesis in full generality," arXiv:1007.1206v2 [math.DS], 10 August 2010.
108. A. Somekh-Baruch and N. Merhav, "Exact random coding exponents for erasure decoding," in Proceedings of ISIT 2010, pp. 260-264, Austin, Texas, U.S.A., June 2010.
109. N. Sourlas, "Spin-glass models as error-correcting codes," Nature, vol. 339, pp. 693-695, June 1989.
110. N. Sourlas, "Spin glasses, error-correcting codes and finite-temperature decoding," Europhysics Letters, vol. 25, pp. 159-164, 1994.
111. K. Takeuchi, M. Vehpaperä, T. Tanaka, and R. R. Müller, "Replica analysis of general multiuser detection in MIMO DS-CDMA channels with imperfect CSI," in Proceedings of ISIT 2008, pp. 514-518, Toronto, Canada, July 2008.
112. T. Tanaka, "Statistical mechanics of CDMA multiuser demodulation," Europhysics Letters, vol. 54, no. 4, pp. 540-546, 2001.
113. T. Tanaka, "A statistical-mechanics approach to large-system analysis of CDMA multiuser detectors," IEEE Transactions on Information Theory, vol. 48, no. 11, pp. 2888-2910, November 2002. (Pubitemid 35327603)
114. H. Touchette, "Methods for calculating nonconcave entropies," arXiv:1003.0382v1 [cond-mat.stat-mech], 1 March 2010.
115. A. M. Tulino and S. Verdú, "Random matrix theory and wireless communications," Foundations and Trends in Communications and Information Theory, vol. 1, no. 1, 2004.
116. J. J. M. Varbaarschot and M. R. Zirnbauer, "Critique of the replica trick," Journal of Physics A: Mathamatical and General, vol. 17, pp. 1093-1109, 1985.
117. A. J. Viterbi and J. K. Omura, Principles of Digital Communication and Coding. McGraw-Hill, 1979.
118. M. J. Wainwright and M. I. Jordan, "Graphical models, exponential families, and variational inference," Foundations and Trends in Machine Learning, vol. 1, no. 1-2, 2008.
119. J. M.Wozencraft and I. M. Jacobs, Principles of Communication Engineering. John Wiley & Sons, 1965. Reissued by Waveland Press, 1990.
120. A. D. Wyner, "On the probability of buffer overflow under an arbitrary bounded input-output distribution," SIAM Journal on Applied Mathematics, vol. 27, no. 4, pp. 544-570, December 1974.
121. C. N. Yang and T. D. Lee, "Statistical theory of equations of state and phase transitions. I. Theory of condensation," Physical Review, vol. 87, no. 3, pp. 404-409, August 1952.
122. J. S. Yedidia, Quenched Disorder: Understanding Glasses Using a Variational Principle and the Replica Method. Lectures delivered at the Santa Fe Summer School on Complex Systems, June 15-19 1992.
123. J. S. Yedidia, W. T. Freeman, and Y.Weiss, "Constructing free energy approximations and generalized belief propagation algorithms," IEEE Transactions on Information Theory, vol. 51, no. 7, pp. 2282-2312, July 2005. (Pubitemid 41136394)
124. R. W. Yeung, "A first course in information theory," in Information Theory and Network Coding, (R. W. Yeung, ed.), New York, 2002: Kluwer Academic/Plenum Publishers, 2008. Springer.
125. M. Zakai and J. Ziv, "A generalization of the rate-distortion theory and applications," in Information Theory New Trends and Open Problems, (G. Longo, ed.), pp. 87-123, Springer-Verlag, 1975.
126. M. R. Zirnbauer, "Another critique of the replica trick," arXiv:cond-mat/9903338v1 [cond-mat.mes-hall], 22 March 1999.
127. J. Ziv and M. Zakai, "On functionals satisfying a data-processing theorem," IEEE Transactions on Information Theory, vol. IT-19, no. 3, pp. 275-283, May 1973.