An identity of Chernoff bounds with an interpretation in statistical physics and applications in information theory

IEEE Transactions on Information Theory

Volumn 54, Issue 8, 2008, Pages 3710-3721

  Merhav, Neri ^a  

^a TECHNION ISRAEL INSTITUTE OF TECHNOLOGY   (Israel)

Author keywords

Additives; Books; Cavity resonators; Channel capacity; Chernoff bound; Distortion measurement; Encoding; Energy conservation; Entropy; Equations; Equipartition; Estimation; Filtering; Finite element methods; Glass; Heating; Information theory; Interconnected systems; Isothermal processes; Large deviations theory; Magnetic materials; Magnetization; Mathematical model; Microscopy; Minimization; Mutual information; Object recognition; Optimization; Phase transitions; Physics; Probability; Process control; Quantization; Quantum mechanics; Random variables; Resource management; Reviews; Signal processing; Speech coding; Statistical physics; Temperature; Temperature dependence; Temperature distribution; Testing; Thermal equilibrium; Thermal factors; Thermodynamics; Time frequency analysis; Upper bound

Indexed keywords

ADDITIVES; APPLICATION EXAMPLES; BOOKS; CAVITY RESONATORS; CHANNEL CAPACITY; CHERNOFF BOUND; CHERNOFF BOUNDS; COMPOSITE SYSTEMS; DISTORTION MEASUREMENT; ENCODING; ENERGY CONSERVATION; ENTROPY; EQUATIONS; EQUIPARTITION; ESTIMATION; FILTERING; FINITE ELEMENT METHODS; GLASS; HEATING; INTERCONNECTED SYSTEMS; ISOTHERMAL PROCESSES; LARGE DEVIATIONS; LARGE DEVIATIONS THEORY; MAGNETIC MATERIALS; MAGNETIZATION; MATHEMATICAL MODEL; MICROSCOPY; MINIMIZATION; MULTIPLE SUBSYSTEMS; MUTUAL INFORMATION; OBJECT RECOGNITION; OPTIMIZATION; PHASE TRANSITIONS; PROCESS CONTROL; QUANTIZATION; QUANTUM MECHANICS; RANDOM VARIABLES; RATE FUNCTIONS; RESOURCE MANAGEMENT; REVIEWS; SIGNAL PROCESSING; SPEECH CODING; STATISTICAL PHYSICS; TEMPERATURE; TEMPERATURE DEPENDENCE; TEMPERATURE DISTRIBUTION; TESTING; THERMAL EQUILIBRIUM; THERMAL FACTORS; THERMODYNAMICS; TIME FREQUENCY ANALYSIS; UPPER BOUND;

COMPUTATIONAL FLUID DYNAMICS; CYBERNETICS; INFORMATION THEORY; PHYSICS; PROBABILITY; RISK ASSESSMENT; STATISTICS;

STATISTICAL METHODS;

EID: 48849114063     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2008.926340     Document Type: Article

References (51)

1. R. Albert and A.-L Barabási, "Statistical mechanics of complex networks," 2001, arXiv:cOnd-mat/0106096.
2. G. B. Baǧci, "The physical meaning of Rényi relative entropies," 2007, arXiv:cOnd-mat/0703008v1.
3. V. B. Balakirsky, "A converse coding theorem for mismatched decoding at the output of binary-input memoryless channels," IEEE Trans. Inf. Theory, vol. 41, no. 6, pp. 1889-1902, Nov. 1995.
4. T. Berger, Rate Distortion Theory: A Mathematical Basis for Data Compression. Engelwood Cliffs, NJ: Prentice-Hall,, 1971.
5. J. P. Burg, "Maximum entropy spectral analysis," Ph.D. dissertation, Stanford Univ., Dept. Geophysics, Stanford, CA, 1975.
6. M. Chiang and S. Boyd, "Geometric programming duals of channel capacity and rate distortion," IEEE Trans. Inf. Theory, vol. 50, no. 2, pp. 245-258, Feb. 2004.
7. T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed. New York: Wiley, 2005.
8. I. Csiszár and J. Körner, Information Theory: Coding Theorems for Discrete Memoryless Systems. New York: Academic, 1981.
9. I. Csiszár and P. Narayan, "Channel capacity for a given decoding metric," IEEE Trans. Inf. Theory, vol. 41, no. 1, pp. 35-43, Jan. 1995.
10. R. S. Ellis, Entropy, Large Deviations, and Statistical Mechanics. New York: Springer-Verlag, 1985.
11. R. S. Ellis, "The theory of large deviations and applications to statistical mechanics," 2006 [Online]. Available: http:// www.math.umass.edu/~rsellis/pdf-files/Dresden-lectures.pdf, Lectures for International Seminar on Extreme Events in Complex Dynamics.
12. R. M. Gray, Source Coding Theory. Norwell, MA: Kluwer Academic, 1990.
13. R. M. Gray, A. H. Gray, G. Rebolledo, and J. E. Shore, "Rate distortionspeech coding with a minimum discrimination information distortion measure," IEEE Trans. Inf. Theory, vol. IT-27, no. 6, pp. 708-721, Nov. 1981.
14. R. M. Gray and D. L. Neuhoff, "Quantization," IEEE Trans. Inf. Theory, vol. 44, no. 6, pp. 2325-2383, Oct. 1998.
15. D. Guo and S. Verdú, "Multiuser detection and statistical physics," in Communications, Information and Network Security, V. Bhargava, H. V. Poor, V. Tarokh, and S. Yoon, Eds. Norwell, MA: Kluwer Academic, 2002, ch. 13, pp. 229-277.
16. T. Hosaka and Y. Kabashima, "Statistical mechanical approach to error exponents of lossy data compression," J. Phys. Soc. Japan, vol. 74, no. 1, pp. 488-497, Jan. 2005.
17. Y. Iba, "The Nishimori line and Bayesian statistics," J. Phys. A: Math. Gen., vol. 32, pp. 3875-3888, 1999.
18. E. T. Jaynes, "Information theory and statistical mechanics," Phys. Rev. A, vol. 106, pp. 620-630, May 1957.
19. E. T. Jaynes, "Information theory and statistical mechanics - II," Phys. Rev. A, vol. 108, pp. 171-190, Oct. 1957.
20. E. T. Jaynes, "On the rationale of maximum-entropy methods," Proc. IEEE, vol. 70, no. 9, pp. 939-952, Sep. 1982.
21. Y. Kabashima and T. Hosaka, "Statistical mechanics for source coding with a fidelity criterion," Progr. Theor. Phys., no. 157, pp. 197-204, 2005.
22. Y. Kabashima, K. Nakamura, and J. van Mourik, "Statistical mechanics of typical set decoding," Phys. Rev. E, vol. 66, 2002.
23. Y. Kabashima and D. Saad, "Statistical mechanics of error correcting codes," Europhysics Lett., vol. 45, no. 1, pp. 97-103, 1999.
24. Y. Kabashima, N. Sazuka, K. Nakamura, and D. Saad, "Tighter decoding reliability bound for Gallager's error-correcting code," Phys. Rev. E, vol. 64, pp. 046113-1-046113-4, 2001.
25. I. Kanter and D. Saad, "Error-correcting codes that nearly saturate Shannon's bound," Phys. Rev. Lett., vol. 83, no. 13, pp. 2660-2663, Sep. 1999.
26. C. Kittel, Elementary Statistical Physics. New York: Wiley, 1958.
27. R. Landauer, "Irreversibility and heat generation in the computing process," IBM J. Res. Devel., vol. 5, pp. 183-191, 1961.
28. A. Lapidoth, "Mismatched decoding and the multiple access channel," Stanford Univ., Stanford, CA, Tech. Rep., 1994.
29. A. Lapidoth and S. Shamai (Shitz), "A lower bound on the bit-error rate resulting from mismatched Viterbi decoding," Dept. Elec. Eng., Technion-Israel Inst. Technol., Haifa, Israel, Tech. Rep., CC Pub No. 163, 1996.
30. O. J. E. Maroney, "The (absence of a) relationship between thermodynamic and logical reversibility," 2004, arXiv:pHysics/0406137.
31. D. McAllester, "A Statistical Mechanics Approach to Large Deviations Theorems," 2006 [Online]. Available: http://citeseer.ist.psu.edu/ 443261.html, preprint
32. N. Merhav, G. Kaplan, A. Lapidoth, and S. Shamai (Shitz), "On information rates for mismatched decoders," IEEE Trans. Inf. Theory vol. 40, no. 6, pp. 1953-1967, Nov. 1994.
33. N. Merhav and I. Kontoyiannis, "Source coding exponents for zero-delay coding with finite memory," IEEE Trans. Inf. Theory, vol. 49, no. 3, pp. 609-625, Mar. 2003.
34. M. Mézard and A. Montanari, "Constraint satisfaction networks in physics and computation," 2006 [Online]. Available: http://www.lptms.u-psud.fr/membres/mezard/
35. T. Mora and O. Rivoire, "Statistical mechanics of error exponents for error-correcting codes," 2006, arXiv:cOnd-mat/0606696.
36. T. Mutayama, "Statistical mechanics of the data compression theorem," J. Phys. A: Math. Gen., vol. 35, pp. L95-L100, 2002.
37. D. L. Neuhoff and R. K. Gilbert, "Causal source codes," IEEE Trans. Inf. Theory, vol. IT-28, no. 5, pp. 701-713, Sep. 1982.
38. H. Nishimori, "Spin glasses and information," Physica A, vol. 384, pp. 94-99, 2007.
39. Y. Oono, "Large deviation and statistical physics," Progr. Theor. Phys. Suppl., no. 99, pp. 165-205, 1989.
40. S. Pereira and T. Weissman, "Denoising and filtering under the probability of excess loss criterion," IEEE Trans. Inf. Theory, vol. 53, no. 4, pp. 1265-1281, Apr. 2007.
41. M. B. Plenio and V. Vitelli, "The physics of forgetting: Landauer's erasure principle and information theory," Contemp. Phys., vol. 42, no. 1, pp. 25-60, 2001.
42. A. Procacci and B. Scoppola, "Statistical mechanics approach to coding theory," J. Statist. Phys., vol. 96, no. 3/4, pp. 907-912, 1999.
43. I. Rojdestvenski and M. C. Cottman, "Mapping of statistical physics to information theorywith application to biological systems," J. Theor. Biol., pp. 43-54, 2000.
44. K. Rose, "A mapping approach to rate-distortion computation and analysis," IEEE Trans. Inf. Theory, vol. 40, no. 6, pp. 1939-1952, Nov. 1994.
45. T. Shinzato, "Statistical Physics and Thermodynamics on Large Deviation," [Online]. Available: http://www.sp.dis.titech.ac.jp/ shinzato/LD.pdf preprint.
46. J. E. Shore and R. W. Johnson, "Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy," IEEE Trans. Inf. Theory, vol. IT-26, no. 1, pp. 26-37, Jan. 1980.
47. N. Sourlas, "Spin-glass models as error-correcting codes," Nature vol. 339, pp. 693-695, Jun. 1989.
48. N. Sourlas, "Spin glasses, error-correcting codes and finite-temperature decoding," Europhysics Lett., vol. 25, pp. 159-164, 1994.
49. T. Tanaka, "Statistical mechanics of CDMA multiuser demodulation," Europhysics Lett., vol. 54, no. 4, pp. 540-546, 2001.
50. T. Tanaka, "A statistical-mechanics approach to large-system analysis of CDMA multiuser detectors," IEEE Trans. Inf. Theory, vol. 48, no. 11, pp. 2888-2910, Nov. 2002.
51. M. J. Wainwright, T. S. Jaakkola, and S. S. Willsky, "A new class of upper bounds on the log partition function," IEEE Trans. Inf. Theory vol. 51, no. 7, pp. 2313-2335, Jul. 2005.