Universal discrete denoising: Known channel

IEEE Transactions on Information Theory

Volumn 51, Issue 1, 2005, Pages 5-28

  Weissman, Tsachy ^a   Ordentlich, Erik ^b   Seroussi, Gadiel ^b   Sergio, Verdú ^c   Weinberger, Marcelo J ^b  

^a Stanford University   (United States)

^b HEWLETT PACKARD LABORATORIES   (United States)

^c Princeton University   (United States)

Author keywords

Context models; Denoising; Discrete filtering; Discrete memoryless channels (DMCs); Individual sequences; Noisy channels; Universal algorithms

Indexed keywords

ALGORITHMS; COMMUNICATION CHANNELS (INFORMATION THEORY); MATHEMATICAL MODELS; PROBABILITY; RANDOM PROCESSES; SIGNAL RECONSTRUCTION; SPURIOUS SIGNAL NOISE; THEOREM PROVING;

CONTEXT MODELS; DISCRETE DENOISING ALGORITHM; DISCRETE FILTERING; DISCRETE MEMORYLESS CHANNEL (DMC); NOISY CHANNELS; UNIVERSAL DISCRETE DENOISING;

SIGNAL FILTERING AND PREDICTION;

EID: 12444304529     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2004.839518     Document Type: Article

References (82)

1. D. Angluin and M. Csürös, "Learning Markov chains with variable memory length from noisy output," in Proc. 10th Annu. Conf. Computational Learning Theory (COLT 1997), Nashville, TN, Jul. 1997.
2. R. J. Ballard, "Extended rules for the sequence compound decision problem with m × n component," Ph.D. dissertation, Michigan State Univ., East Lansing, 1974.
3. 2) convergence to κ-extended Bayes risk in the sequence compound decision problem with m × n component," Statistics and Probability, Michigan State Univ., East Lansing, RM-333, 1974.
4. L. E. Baum, T. Petrie, G. Soules, and N. Weiss, "A maximization technique occurring in the statistical analysis of probabilistic functions of Markov chains," Ann. Math. Statist., vol. 41, pp. 164-171, 1970.
5. T. Berger and J. D. Gibson, "Lossy source coding," IEEE Trans. Inf. Theory, vol. 44, no. 6, pp. 2693-2723, Oct. 1998.
6. H. W. Bode and C. E. Shannon, "A simplified derivation of linear least-squares smoothing and prediction theory," Proc. IRE, vol, 38, pp. 417-425, 1950.
7. L. Breiman, Probability. Philadelphia, PA: SIAM, 1992.
8. A. Bruce, D. L. Donoho, and H. Y. Gao, "Wavelet analysis," IEEE Spectrum, vol. 33, no. 10, pp. 26-35, Oct. 1996.
9. G. Caire, S. Shamai (Shitz), and S. Verdú, "Almost-noiseless joint source-channel coding-decoding of sources with memory," in Proc. 5th Int. ITG Conf. Source and Channel Coding (SCC), Jan 14-16, 2004, pp. 295-303.
10. B. Carpentieri, M. J. Weinberger, and G. Seroussi, "Lossless compression of continuous-tone images," Proc. IEEE, vol. 88, no. 11, pp. 1797-1809, Nov. 2000.
11. G. Chang, B. Yu, and M. Vetterli, "Bridging compression to wavelet thresholding as a denoising method," in Proc. Conf. Information Sciences and Systems, vol. 2, Mar. 1997, pp. 568-573.
12. G. Chang, B. Yu, and M. Vetterli, "Adaptive wavelet thresholding for image denoising and compression," IEEE Trans. Image Process., vol. 9, no. 9, pp. 1532-1546, Sep. 2000.
13. R. W. Chang and J. C. Hancock, "On receiver structures for channels having memory," IEEE Trans. Inf. Theory, vol. IT-12, no. 4, pp. 463-468, Oct. 1966.
14. T. M. Cover and J. A. Thomas, Elements of Information Theory. New York: Wiley, 1991.
15. F. J. Damerau and E. Mays, "An examination of undetected typing errors," Inf. Process. and Management: An Int. J., vol. 25, no. 6, pp. 659-664, 1989.
16. A. Dembo and T. Weissman, "The minimax distortion redundancy in noisy source coding," IEEE Trans. Inf. Theory, vol. 49, no. 11, pp. 3020-3030, Nov. 2003.
17. A. Dembo, and T. Weissman, "Universal denoising for the finite-input-general-output channel," IEEE Trans. Inf. Theory. Available [Online] at http:// www.stanford.edu/~tsachy/ interest.htm, to be published.
18. A. Dembo and O. Zeitouni, Large Deviations Techniques and Applications, 2nd ed. New York: Springer-Verlag, 1998.
19. J. L. Devore, "A note on the observation of a Markov source through a noisy channel," IEEE Trans. Inf. Theory, vol. IT-20, no. 6, pp. 762-764, Nov. 1974.
20. D. L. Donoho, "De-noising by soft-thresholding," IEEE Trans. Inf. Theory, vol. 41, no. 3, pp. 613-627, May 1995.
21. D. L. Donoho, "The Kolmogorov sampler," preprint, Jan. 2002. Available [Online] at http://playfair.stanford.edu/~donoho/Reports/.
22. D. L. Donoho, I. M. Johnstone, G. Keryacharian, and D. Picard, " Wavelet shrinkage: Asymptopia?," J. Roy. Statist. Soc., vol. 57, no. 2, pp. 301-369, 1995.
23. R. Durret, Probability: Theory and Examples. Belmont, CA: Duxbury, 1991.
24. Y. Ephraim and N. Merhav, "Hidden Markov processes," IEEE Trans. Inf. Theory, vol. 48, no. 6, pp. 1518-1569, Jun. 2002.
25. M. Feder, N. Merhav, and M. Gutman, "Universal prediction of individual sequences," IEEE Trans. Inf. Theory, vol. 38, no. 4, pp. 1258-1270, Jul. 1992.
26. W. Feller, An Introduction to Probability Theory and its Applications New York: Wiley, 1968.
27. G. Gemelos, S. Sigurjònsson, and T. Weissman, "Universal discrete denoising under channel uncertainty," in Proc. IEEE Int. Symp. Information Theory, Chicago, IL, Jun./Jul. 2004, p. 199.
28. G. H. Golub and C. F. Van Loan, Matrix Computations, 3rd ed. Baltimore, MD: Johns Hopkins Univ. Press, 1996.
29. R. C. Gonzalez and R. E. Woods. Digital Image Processing. Reading, MA: Addison Wesley, 1992.
30. J. Hannan, "Approximation to Bayes risk in repeated play," in Contributions to the Theory of Games. Princeton, NJ: Princeton Univ. Press, 1957, vol. III, pp. 97-139.
31. J. Hannan and H. Robbins, "Asymptotic solutions of the compound decision problem for two completely specified distributions," Ann. Math. Statist., vol. 26, pp. 37-51, 1955.
32. W. Hoeffding, "Probability inequalities for sums of bounded random variables," Amer. Statist. Assoc., vol. 58, pp. 13-30, 1963.
33. M. V. Johns, Jr., "Two-action compound decision problems," in Proc. 5th Berkeley Symp. Mathematical and Statistical Probability, Berkeley, CA, 1967, pp. 463-478.
34. R. E. Kalman, "A new approach to linear filtering and prediction problems," Trans. ASME - J. Basic Eng., ser. D, vol. 82, pp. 35-45, 1960.
35. R. Khasminskii and O. Zeitouni, "Asymptotic filtering for finite state Markov chains," Stochastic Processes and Their Applications, vol. 63, pp. 1-10, 1996.
36. S. P. Lalley, "Beneath the noise, chaos," Ann. Statist., vol. 27, pp. 461-479, 1999.
37. S. P. Lalley, "Removing the noise from chaos plus noise," in Nonlinear Dynamics and Statistics, A. I. Mees, Ed. Basel, Switzerland: Birkhauser, 1999, pp. 233-244.
38. S. P. Lalley and A. B. Nobel, "Denoising deterministic time series," preprint, 2002. Available [Online] at http:// www.stat.unc.edu/faculty/nobel/links/denoise.pdf.
39. P. Lancaster and M. Tismenetsky, The Theory of Matrices. Orlando, FL: Academic, 1985.
40. E. Mays, F. J. Damerau, and R. L. Mercer, "Context based spelling correction," in Proc. IBM Natural Language ITL, Paris, France, 1990, pp. 517-522.
41. N. Merhav and M. Feder, "Universal prediction," IEEE Trans. Inf. Theory, vol. 44, no. 6, pp. 2124-2147, Oct. 1998.
42. B. Natarajan, "Filtering random noise via data compression," in Proc. Data Compression Conf. (DCC '93), Snowbird, UT, Mar. 1993, pp. 60-69.
43. B. Natarajan, "Filtering random noise from deterministic signals via data compression," IEEE Trans. Signal Process., vol. 43, no. 11, pp. 2595-2605, Nov. 1995.
44. B. Natarajan, K. Konstantinides, and C. Herley, "Occam filters for stochastic sources with application to digital images," IEEE Trans. Signal Process., vol. 46, no. 11, pp. 1434-1438, Nov. 1998.
45. E. Ordentlich, G. Seroussi, S. Verdú, K. Viswanathan, M. J. Weinberger, and T. Weissman, "Channel decoding of systematically encoded unknown redundant sources," in Proc. IEEE Int. Symp. Information Theory, Chicago, IL, Jun./Jul. 2004, p. 165.
46. E. Ordentlich, G. Seroussi, S. Verd-acu, M. J. Weinberger, and T. Weissman, "A universal discrete image denoiser and its application to binary images," in Proc. IEEE Int. Conf. Image Processing, vol. 1, Barcelona, Catalonia, Spain, Sep. 2003, pp. 117-120.
47. E. Ordentlich and T. Weissman, "On the optimality of symbol by symbol filtering and denoising," IEEE Trans. Inf. Theory. Available: [Online] at http://www.stanford.edu/~tsachy/interest.htm, to be published.
48. E. Ordentlich, T. Weissman, M. J. Weinberger, A. Somekh-Baruch, and N. Merhav, "Discrete universal filtering through incremental parsing," in Proc. 2004 Data Compression Conf. (DCC'04), Snowbird, UT, Mar. 2004, pp. 352-361.
49. J. Rissanen, Stochastic Complexity in Statistical Inquiry, Singapore: World Scientific, 1989.
50. J. Rissanen, "MDL denoising," IEEE Trans. Inf. Theory, vol. 46, no. 7, pp. 2537-2543, Nov. 2000.
51. H. Robbins, "Asymptotically subminimax solutions of compound decision problems," in Proc. 2nd Berkeley Symp. Mathematical and Statistical Probability, Berkeley, CA, 1951, pp. 131-148.
52. H. Robbins and E. Samuel, "Testing statistical hypotheses; the compound approach," in Recent Developments in Information anal Decision Processes. New York: Macmillan, 1962, pp. 63-70.
53. J. Van Ryzin, "The compound decision problem with m x n finite loss matrix," Ann. Math. Statist., vol. 37, pp. 412-424, 1966.
54. J. Van Ryzin, "The sequential compound decision problem with m x n finite loss matrix," Ann. Math. Statist., vol. 37, pp. 954-975, 1966.
55. E. Samuel, "Asymptotic solutions of the sequential compound decision problem," Ann. Math. Statist., vol. 34, no. 3, pp. 1079-1095, 1963.
56. E. Samuel, "An empirical Bayes approach to the testing of certain parametric hypotheses," Ann. Math. Statist., vol. 34, no. 4, pp. 1370-1385, 1963.
57. E. Samuel, "Convergence of the losses of certain decision roles for the sequential compound decision problem," Ann. Math. Statist., vol. 35, no. 4, pp. 1606-1621, 1964.
58. E. Samuel, "On simple rules for the compound decision problem," J. Roy. Statist. Soc., ser. B, vol. 27, pp. 238-244, 1965.
59. C. E. Shannon, "A mathematical theory of communication," Bell Syst. Tech. J., vol. 27, pp. 379-423, 1948.
60. J. W. Shavlik, "Finding genes by case-based reasoning in the presence of noisy case boundaries," in Proc. DARPA Cased-Based Reasoning Workshop, vol. 1, 1991, pp. 327-338.
61. L. Shue, B. D. O. Anderson, and F. De Bruyne, "Asymptotic smoothing error for hidden Markov models," IEEE Trans. Signal Process.,vol. 48, no. 12, pp. 3289-3302, Dec. 2000.
62. P. Soille, Morphological Image Analysis: Principles and Applications Berlin, Germany: Springer-Verlag, 1999.
63. I. Tabus, J. Rissanen, and J. Astola, "Normalized maximum likelihood models for boolean regression with application to prediction and classification in genomics," in Computational and Statistical Approaches to Genomics, W. Zhang and I. Shmulevich, Eds. Boston, MA: Kluwer Acadmic, 2002.
64. I. Tabus, J. Rissanen, and J. Astola, "Classification and feature gene selection using the normalized maximum likelihood model for discrete regression," Signal Process., Special Issue: Genomic Signal Processing, vol. 83, no. 4, pp. 713-727, Apr. 2003.
65. S. B. Vardeman, "Admissible solutions of finite state sequence compound decision problems," Ann. Statist., vol. 6, no. 3, pp. 673-679, 1978.
66. S. B. Vardeman, "Admissible solutions of κ-extended finite state set and sequence compound decision problems," J. Multiv. Anal., vol. 10, pp. 426-441, 1980.
67. S. B. Vardeman, "Approximation to minimum κ-extended Bayes risk in sequences of finite state decision problems and games," Bull. Inst. Math. Academia Sinica, vol. 10, no. 1, pp. 35-52, Mar. 1982.
68. M. J. Weinberger, J. J. Rissanen, and M. Feder, "A universal finite-memory source," IEEE Trans. Inf. Theory, vol. 41, no. 3, pp. 643-652, May 1995.
69. T. Weissman, "Universally attainable error-exponents for rate-distortion coding of noisy sources," IEEE Trans. Inf. Theory vol. 50, no. 6, pp. 1229-1246, Jun. 2004.
70. T. Weissman and N. Merhav, "Universal prediction of individual binary sequences in the presence of noise," IEEE Trans. Inf. Theory, vol. 47, no. 6, pp. 2151-2173, Sep. 2001.
71. T. Weissman, "Finite-delay lossy coding and filtering of individual sequences corrupted by noise," IEEE Trans. Inf. Theory, vol. 48, no. 3, pp. 721-733, Mar. 2002.
72. T. Weissman, "Universal prediction of random binary sequences in a noisy environment," Ann. Appl. Probab., vol. 14, no. 1, pp. 54-89, Feb. 2004.
73. T. Weissman, N. Merhav, and A. Baruch, "Twofold universal prediction schemes for achieving the finite-state predictability of a noisy individual binary sequence," IEEE Trans. Inf. Theory, vol. 47, no. 5, pp. 1849-1866, Jul. 2001.
74. T. Weissman and E. Ordentlich, "The empirical distribution of rate-constrained codes," IEEE Trans. Inf. Theory. Available [Online] at http://www.stanford.edu/~tsachy/interest.htm, to be published.
75. N. Wiener, The Extrapolation, Interpolation and Smoothing of Stationary Time Series. New York: Wiley, 1949.
76. F. M. J. Willems, Y. M. Shtarkov, and T. J. Tjalkens, "The context-tree weighting method: Basic properties," IEEE Trans. Inf. Theory, vol. 41, no. 3, pp. 653-664, May 1995.
77. C. H. Zhang, "Compound decision theory and empirical Bayes methods," Ann. Statist., vol. 31, no. 2, pp. 379-390, 2003.
78. J. Ziv, "Distortion-rate theory for individual sequences," IEEE Trans. Inf. Theory, vol. IT-26, no. 2, pp. 137-143, Mar. 1980.
79. J. Ziv and A. Lempel, "Compression of individual sequences via variable-rate coding," IEEE Trans. Inf. Theory, vol. IT-24, no. 5, pp. 530-536, Sep. 1978.
80. E. Ordentilich, M.J. Weinberger, and T. Weissman, "Efficient pruning of bi-directional context trees with applications to universal denoising and compression," in Proc. 2004 Information Theory Workshop, San Antonio, TX, Oct. 2004.
81. G. Gemelos, S. Sigurjønsson, and T. Weissman, "Universal minimax binary image denoising under channel uncertainty," in Proc. IEEE Int. Conf. Image Processing, Singapore, Sep. 2004, pp. 997-1000.
82. R. Zhang and T. Weissman, "On discrete denoising for the burst noise channel," in Proc. 42nd Annu. Allerton Conf. Communication, Control, and Computing, vol. 1, Monticello, IL, Sep. 2004.