A new look at entropy for solving linear inverse problems

IEEE Transactions on Information Theory

Volumn 45, Issue 5, 1999, Pages 1565-1578

  Le Besnerais, Guy ^a   Bercher, Jean François ^b   Demoment, Guy ^c  

^a ONERA   (France)

^b ESIEE PARIS   (France)

^c UNIV PARIS SUD   (France)

Author keywords

Duality; Entropy; Inverse problems; Kullback Leibler information; Regularization

Indexed keywords

INFORMATION THEORY; INVERSE PROBLEMS; MATHEMATICAL MODELS; MATRIX ALGEBRA; SET THEORY; SPURIOUS SIGNAL NOISE; TRANSFER FUNCTIONS;

DUALITY; KULLBACK-LEIBLER INFORMATION; REGULARIZATION;

SIGNAL RECONSTRUCTION;

EID: 0032673155     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/18.771159     Document Type: Article

References (51)

1. D. Mukherjee and D. C. Hurst, "Maximum entropy revisited," Statistica Neerlandica, vol. 38, no. 1, pp. 1-11, 1984.
2. L. K. Jones and V. Trutzer, "Computationally feasible high-resolution minimum-distance procedures which extend the maximum-entropy method," Inverse Problems, vol. 5, pp. 749-766, 1989.
3. L. K. Jones and C. L. Byrne, "General entropy criteria for inverse problems, with applications to data compression, pattern classification and cluster analysis," IEEE Trans. Inform. Theory, vol. 36, pp. 23-30, Jan. 1990.
4. J. Shore and R. Johnson, "Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy," IEEE Trans. Inform. Theory, vol. IT-26, pp. 26-37, Jan. 1980.
5. I. Csiszár, "Why least-squares and maximum entropy - An axiomatic approach to inference for linear inverse problems," Ann. Statist., vol. 19, no. 4, pp. 2032-2066, 1991.
6. _, "Generalized projection for nonnegative functions," Acta. Math. Hungar., vol. 6, nos. 1/2, pp. 161-185, 1995.
7. J. Navaza, "On the maximum entropy estimate of electron density function," Acta Crystallogr., pp. 232-244, 1985.
8. _, "The use of nonlocal constraints in maximum-entropy electron density reconstruction," Acta Crystallogr., pp. 212-223, 1986.
9. D. Dacunha-Castelle and F. Gamboa, "Maximum d'entropie et problème des moments," Ann. l'Institut Henri Poincaré, vol. 26, no. 4, pp. 567-596, 1990.
10. F. Gamboa and É. Gassiat, "Bayesian methods and maximum entropy for ill posed inverse problems," Ann. Statist., vol. 25, no. 1, Feb. 1997.
11. _, "The maximum entropy method on the mean: Applications to linear programming and superresolution," Math. Programming, no. 66, pp. 103-122, Oct. 1994.
12. F. Gamboa and M. Lavielle, "On two-dimensional spectral realization," IEEE Trans. Inform. Theory, vol. 40, pp. 1603-1608, Sept. 1994.
13. I. Csiszár, F. Gamboa, and É. Gassiat, "MEM pixel correlated solutions for generalized moment and interpolation problems," IEEE Trans. Inform. Theory, submitted for publication, 1998.
14. G. Le Besnerais, J. Navaza, and G. Demoment, "Aperture synthesis in astronomical radio-interferometry using maximum entropy on the mean," in SPIE Conf., Stochastic and Neural Methods in Signal Processing, Image Processing and Computer Vision, S. Chen, Ed. (San Diego, July 1991).
15. J.-F. Bercher, G. Le Besnerais, and G. Demoment, "The maximum entropy on the mean method, noise and sensitivity," in Maximum Entropy and Bayesian Methods, J. Skilling, Ed. Dordrecht, The Netherlands: Kluwer, 1994.
16. C. Heinrich, J.-F. Bercher, G. Le Besnerais, and G. Demoment, "Restoration of spiky signals: A new optimal estimate and a comparison," in Proc. IEEE ICASSP (Detroit, MI, May 1995), pp. 877-880.
17. C. Heinrich, J.-F. Bercher, and G. Demoment, "The maximum entropy on the mean method, correlations and implementation issues," in Workshop Maximum Entropy and Bayesian Methods. Berg-en-Tal, South Africa: Kluwer, 1996, pp. 52-61.
18. P. Maréchal and A. Lannes, "Unification of some deterministic and probabilistic methods for the solution of linear inverse problems via the principle of maximum entropy on the mean," Inverse Probl., vol. 13, pp. 135-151, 1997.
19. B. Urban, "Retrieval of atmospheric thermodynamical parameters using satellite measurements with a maximum entropy method," Inverse Probl., vol. 12, pp. 779-796, 1996.
20. C. E. Shannon, "The mathematical theory of communication," Bell Syst. Tech. J., vol. 27, pp. 379-423, pp. 623-656, 1948.
21. S. Kullback and R. A. Leibler, "On information and sufficiency," Ann. Math. Statist., vol. 22, pp. 79-86, 1951.
22. S. Kullback, Information Theory and Statistics. New York: Wiley, 1959.
23. I. Csiszár, "I - Divergence geometry of probability distributions and minimization problems," Ann. Probab., vol. 3, no. 1, pp. 146-158, 1975.
24. T. M. Cover and J. A. Thomas, Elements of Information Theory. New York: Wiley, 1991.
25. E. T. Jaynes, Papers on Probability, Statistics and Statistical Physics, R. D. Rosenkrantz, Ed., Reidel edition. Dordrecht, The Netherlands: Kluwer, 1982.
26. B. Buck and V. A. Macaulay, Eds., Maximum Entropy in Action. Oxford, U.K.: Clarendon, 1991.
27. M. A. Berger, An Introduction to Probability and Stochastic Processes. New York: Springer-Verlag, 1992.
28. R. S. Ellis, Entropy, Large Deviations, and Statistical Mechanics. New York: Springer-Verlag, 1985.
29. I. Csiszár, "An extended maximum entropy principle and a Bayesian justification," in Bayesian Statistics 2, J. M. Bernardo, M. H. DeGroot, and A. F. M. Smith, Eds. Amsterdam, The Netherlands: North-Holland Elsevier, 1985, pp. 83-98.
30. R. Azencott, "Grandes déviations et applications," in École D'été de Probabilité de Saint Flour VIII-1978, P. L. Hennequin, Ed. Berlin, Germany: Springer-Verlag, 1978, pp. 2-172.
31. A. Dembo and O. Zeitouni, Large Deviations Techniques. Boston, MA: Jones and Bartlett, 1992.
32. I. Csiszàr and J. Körner, Information Theory: Coding Theorems for Discrete Memory-Less Systems. New York: Academic, 1981.
33. C. Robert, "An entropy concentration theorem: Applications in artificial intelligence and descriptive statistics," J. Appl. Probab., vol. 27, pp. 303-313, 1990.
34. J. M. Borwein and A. S. Lewis, "Duality relationships for entropy-like minimization problems," SIAM J. Contr. Optimiz., vol. 29, no. 2, pp. 325-338, 1991.
35. A. Decarreau, D. Hilhorst, C. Lemaréchal, and J. Navaza, "Dual methods in entropy maximization. Application to some problems in crystallography," SIAM J. Optimiz., vol. 2, no. 2, pp. 173-197, May 1992.
36. O. Barndorff-Nielsen, Information and Exponential Model in Statistics. New York: Wiley, 1978.
37. R. T. Rockafellar, Convex Analysis. Princeton, NJ: Princeton Univ. Press, 1970.
38. I. Csiszár, "Sanov property, generalized I-projections and a conditional limit theorem," Ann. Probab., vol. 12, pp. 768-793, 1984.
39. L. M. Bregman, "The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming," USSR Comput. Math, and Math. Phys., vol. 7, pp. 200-217, 1967.
40. F. Itakura and S. Saito, "A statistical method for estimation of speech spectral density and formant frequencies," Electron. and Commun., vol. 53-A, pp. 36-43, 1970.
41. J. P. Burg, "Maximum entropy spectral analysis," in Proc. 37th Meet. Soc. Exploration Geophysicists (Oklahoma City, OK, Oct. 1967), pp. 34-41.
42. J. E. Shore, "Minimum cross-entropy spectral analysis," IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-29, pp. 230-237, Apr. 1981.
43. J. Skilling, "The axioms of maximum entropy," in Maximum Entropy and Bayesian Methods, G. J. Erickson and C. R. Smith, Eds. Dordrecht, The Netherlands: Kluwer, 1988, pp. 173-187.
44. J. Shore and R. Johnson, "Which is the better entropy expression for speech processing: - s log s or log s?," IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-32, Feb. 1984.
45. R. Narayan and R. Nityananda, "Maximum entropy image restoration in astronomy," Ann. Rev. Astron. Astrophys., vol. 24, pp. 127-170, 1986.
46. K. Horne, "Image of accretion disc - I: The eclipse mapping method," Monthly Notes Roy. Astronom. Soc., vol. 213, pp. 129-141, 1985.
47. D. L. Donoho, I. M. Johnstone, J. C. Hoch, and A. S. Stern, "Maximum entropy and the nearly black objects," J. Roy. Statist. Soc. B, vol. 54, no. 1, pp. 41-81, 1992.
48. S. F. Gull, "Developments in maximum entropy data analysis," in Maximum Entropy and Bayesian Methods, J. Skilling, Ed. Dordrecht, The Netherlands: Kluwer, 1989, pp. 53-71.
49. J. A. O'Sullivan, "Divergence penalty for image regularization," in Proc. IEEE ICASSP (Adelaide, Australia, Apr. 1994), pp. 541-544.
50. G. T. Herman, H. K. Tuy, H. Langenberg, and P. C. Sabatier, Basic Methods of Tomography and Inverse Problems. Adams Hilger, 1987.
51. A. R. Thomson, J. M. Moran, and G. W. Swenson, Interferometry and Synthesis in Radioastronomy. New York: Wiley, 1986.