Joint additive Kullback-Leibler residual minimization and regularization for linear inverse problems

Mathematical Methods in the Applied Sciences

Volumn 30, Issue 13, 2007, Pages 1527-1544

  Resmerita, Elena ^a   Anderssen, Robert S ^b  

^a JOHANN RADON INSTITUTE FOR COMPUTATIONAL AND APPLIED MATHEMATICS   (Austria)

^b CSIRO   (Australia)

Author keywords

Banach space adjoint operator; Ill posed problems; Information; Kullback Leibler distance; Radon Nikodym theorem; Regularization

Indexed keywords

APPROXIMATION THEORY; CONVERGENCE OF NUMERICAL METHODS; INVERSE PROBLEMS; MATHEMATICAL OPERATORS; PROBLEM SOLVING; THEOREM PROVING;

BANACH SPACE ADJOINT OPERATORS; ILL-POSED PROBLEMS; KULLBACK-LEIBLER DISTANCES; RADON-NIKODYM THEOREM; REGULARIZATION;

LINEAR SYSTEMS;

EID: 34547616147     PISSN: 01704214     EISSN: 10991476     Source Type: Journal    
DOI: 10.1002/mma.855     Document Type: Article

References (49)

1. Engl HW, Hanke M, Neubauer A. Regularization of Inverse Problems. Kluwer Academic Publishers: Dordrecht, 1996.
2. Tikhonov AN. Regularization of incorrectly posed problems. Soviet Mathematics Doklady 1963; 4:1624-1627.
3. Twomey S. Introduction to the Mathematics of Inversion in Remote Sensing and Indirect Measurement. Elsevier: Amsterdam, 1977.
4. Wahba G. Spline Models for Observational Data. SIAM: Philadelphia, PA, 1990.
5. Latham GA, Anderssen RS. A hyperplane approach to the EMS algorithm. Applied Mathematics Letters 1992; 5:71-74.
6. Vardi Y, Shepp LA, Kaufman L. A statistical model for positron emission tomography. Journal of the American Statistical Association 1985; 80:8-37.
7. Iusem AN, Svaiter BF. A new smoothing-regularization approach for a maximum-likelihood estimation problem. Applied Mathematics Optimization 1994; 29(3):225-241.
8. Besnerais GL, Bercher J-F, Demoment G. A new look at entropy for solving linear inverse problems. IEEE Transactions on Information Theory 1999; 45:1566-1578.
9. Kullback S, Leibler RA. On information and sufficiency. Annals of Mathematical Statistics 1951; 22:79-86.
10. Hall P. Akaike's information criterion and Kullback-Leibler loss for histogram estimation. Problems Theory and Related Fields 1990; 85:449-467.
11. Jeffreys H. An invariant form for the prior probability in estimation problems. Proceedings of the Royal Society of London, Series A 1946; 186:453-461.
12. Halmos PR, Savage LJ. Application of the Radon-Nikodym theorem to the theory of sufficient statistics. Annals of Mathematical Statistics 1949; 20:225-241.
13. Shore JE, Johnson RW. Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy. IEEE Transactions on Information Theory 1980; 26:26-37.
14. Jaynes ET. Information theory and statistical mechanics. Physical Review 1957; 106:620-630;
15. Physical Review 1957; 108:171-190.
16. Rubinstein RY, Kroese DP. The Cross-Entropy Method. Springer: Berlin, 2004.
17. Ali SM, Silvey SD, A general class of coefficients of divergence of one distribution from another. Journal of the Royal Statistical Society, Series B 1966; 28:131-142.
18. Bernado JM, Smith AFM. Bayesian Theory. Wiley: Chichester, 1994.
19. Braess D, Sauer T. Bernstein polynomials and learning theory. Journal of Approximation Theory 2004; 128: 187-206.
20. Braess D, Forster J, Sauer T, Simon HU. How to achieve minimax expected Kullback-Leibler distance from an unknown finite distribution. Lecture Notes in Artificial Intelligence, vol. 2533. Springer: Berlin, 2002; 380-394.
21. Fisher RA. On the mathematical foundations of theoretical statistics. Philosophical Transactions of the Royal Society of London, Series A 1921; 222:309-368.
22. Fisher RA. Theory of statistical estimation. Proceedings of Cambridge Philosophical Society 1925; 22:700-725.
23. Kullback S. Information Theory and Statistics. Wiley: New York, 1959.
24. Bregman LM. The relaxation method for finding common points of convex sets and its application to the solution of problems in convex programming. USSR Computational Mathematics and Mathematical Physics 1967; 7:200-217.
25. Csiszar I. Why least squares and maximum entropy? An axiomatic approach to inference for linear inverse problems. Annals of Statistics 1991; 19:2032-2066.
26. Jaynes ET. In Papers on Probability, Statistics and Statistical Physics, Rosenkrantz RD (eds). Reidel: Dordrecht, 1982.
27. Amato U, Hughes W. Maximum entropy regularization of Fredholm integral equations of the first kind. Inverse Problems 1991; 7:793-803.
28. Engl HW, Landl G. Convergence rates for maximum entropy regularization. SIAM Journal on Numerical Analysis 1993; 30:1509-1536.
29. Eggermont PPB. Maximum entropy regularization for Fredholm integral equations of the first kind. SIAM Journal on Mathematical Analysis 1993; 24:1557-1576.
30. Donoho DL, Johnstone IM, Hoch JC, Stern AS. Maximum entropy and the nearly black object. Journal of the Royal Statistical Society, Series B 1992; 54:41-81.
31. Landl GA, Anderssen RS. Non-negative differentially constrained entropy-like regularization. Inverse Problems 1996; 12:35-53.
32. Bousquet O, Elisseff A. Stability and generalization. Journal of Machine Learning Research 2002; 2:499-526.
33. Hall P. On Kullback-Leibler loss and density-estimation. Annals of Statistics 1987; 15:1491-1519.
34. Hall P. On the use of compactly supported density estimates in problems of discrimination. Journal of Multivariate Analysis 1987; 23:131-158.
35. Resmerita E. Regularization of ill-posed inverse problems in Banach spaces: convergence rates. Inverse Problems 2005; 21:1303-1314.
36. Csiszar I, Tusnady G. Information geometry and alternative minimization procedures. Statistics and Decisions Supplement 1984; 1:205-237.
37. Chen G, Teboule M. A proximal-based decomposition method for convex minimization problems. Mathematical Programming 1994; 64:81-101.
38. Burger M, Osher S. Convergence rates of convex variational regularization. Inverse Problems 2004; 20:1411-1421.
39. OSullivan JA. Roughness penalties on finite domains. IEEE Transactions on Image Processing 1995; 4:1258-1264.
40. Anderssen RS, Latham GA, Westcott M. Statistical methodology for inverse problems. Mathematical and Computer Modelling 1995; 22:10-12.
41. Good IJ, Gaskins RA, Nonparametric roughness penalties for probability densities. Biometriks 1971; 58:255-277.
42. Wand MP, Jones MC. Kernel Smoothing. Chapman & Hall: London, 1995.
43. Eggermont PPB, LaRiccia V. Maximum penalized likelihood estimation and smoothed EM algorithms for positive integral equations of the first kind. Numerical Functional Analysis and Optimization 1996; 17:737-754.
44. Green PJ. On use of the EM algorithm for penalized likelihood estimation. Journal of the Royal Statistical Society, Series B 1990; 52:443-452.
45. Dempster AP, Laird NM, Rubin DB. Maximum likelihood for incomplete data via the EM algorithm (with Discussion). Journal of the Royal Statistical Society, Series B 1977; 39:1-38.
46. Yu S, Latham GA, Anderssen RS. Stabilizing properties of maximum penalized likelihood estimation for additive Poisson regression. Inverse Problems 1994; 10:1199-1209.
47. Yu S, Latham GA, Anderssen RS. Matrix analysis of a two-stage-splitting iteration for maximum penalized likelihood estimation. SIAM Journal on Matrix Analysis 1997; 18:348-359.
48. Lindblad G. Entropy, information and quantum measurements. Communications in Mathematical Physics 1973; 33:305-322.
49. Borwein JM, Lewis AS. Convergence of best entropy estimates. SIAM Journal on Optimization 1991; 1(2): 191-205.