On the optimality of symbol-by-symbol filtering and denoising

IEEE Transactions on Information Theory

Volumn 52, Issue 1, 2006, Pages 19-40

  Ordentlich, Erik ^a   Weissman, Tsachy ^b  

^a HEWLETT PACKARD LABORATORIES   (United States)

^b Stanford University   (United States)

Author keywords

Asymptotic entropy; Denoising; Discrete memoryless channels (DMCs); Entropy rate; Estimation; Filtering; Hidden Markov processes (HMP); Large deviations performance; Noisy channels; Singlet decoding; Smoothing; Symbol by symbol schemes

Indexed keywords

CODES (SYMBOLS); COMMUNICATION CHANNELS (INFORMATION THEORY); DECODING; ESTIMATION; MARKOV PROCESSES;

ASYMPTOTIC ENTROPY; DENOISING; DISCRETE MEMORYLESS CHANNELS (DMCS); ENTROPY RATE; HIDDEN MARKOV PROCESSES (HMP); LARGE DEVIATIONS PERFORMANCE; NOISY CHANNELS; SINGLET DECODING; SYMBOL-BY-SYMBOL SCHEMES;

SIGNAL FILTERING AND PREDICTION;

EID: 33144485966     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIT.2005.860432     Document Type: Article

References (40)

1. F. Alajaji, N. Phamdo, N. Farvardin, and T. E. Fuja, "Detection of binary Markov sources over channels with additive Markov noise," IEEE Trans. Inf. Theory, vol. 42, no. 1, pp. 230-239, Jan. 1996.
2. L. Arnold, L. Demetrius, and M. Gundlach, "Evolutionary formalism for products of positive random matrices," Ann. Appl. Probab., no. 4, pp. 859-901, 1994.
3. R. Atar and O. Zeitouni, "Exponential stability for nonlinear filtering," Annales de l'Institut H. Poincare Probabilites et Statistique, no. 33, pp. 697-725, 1997.
4. T. Berger, Rate-Distortion Theory: AMathematical Basis for Data Compression. Englewood Cliffs, N.J.: Prentice-Hall, 1971.
5. A. Bhatt, A. Budhiraja, and R. Karandikar, "Markov property and ergodicity of the nonlinear filter," SIAM J. Control and Optimiz., no. 39, pp. 928-949, 2000.
6. D. Blackwell, "The entropy of functions of finite-state Markov chains," in Trans. 1st Prague Conf. Information Theory, Statistical Decision Functions, Random Processes, Prague, Czechoslovakia, 1957, pp. 13-20.
7. T. M. Cover and J. A. Thomas, Elements of Information Theory. New York: Wiley, 1991.
8. I. Csiszár and J. Körner, Information Theory: Coding Theorems for Discrete Memoryless Systems. New York: Academic, 1981.
9. A. Dembo and I. Kontoyiannis, "The asymptotics of waiting times between stationary processes, allowing distortion," Ann. Appl. Probab., no. 9, pp. 413-429, 1999.
10. A. Dembo and T. Weissman, "Universal denoising for the finite-inputcontinuous- output channel," IEEE Trans. Inf. Theory, vol. 51, no. 4, pp. 1507-1517, Apr. 2005.
11. A. Dembo and O. Zeitouni, Large Deviations Techniques and Applications, 2nd ed. New York: Springer-Verlag, 1998.
12. J. L. Devore, "A note on the observation of a Markov source through a noisy channel," IEEE Trans. Inf. Theory, vol. IT-20, no. 6, pp. 762-764, Nov. 1974.
13. A.W. Drake, "Observation of aMarkov source through a noisy channel," in Proc. IEEE Symp. Signal Transmission and Processing, Columbia Univ., New York, 1965, pp. 12-18.
14. Y. Ephraim and N. Merhav, "Hidden Markov processes," IEEE Trans. Inf. Theory, vol. 48, no. 6, pp. 1518-1569, Jun. 2002.
15. H. O. Georgii, Gibbs Measures and Phase Transitions. Berlin, Germany: Walter de Gruyter, 1988.
16. A. Goldsmith and P. Varaiya, "Capacity, mutual information, and coding for finite state Markov channels," IEEE Trans. Inf. Theory, vol. 42, no. 3, pp. 868-886, May 1996.
17. R. M. Gray, "Information rates of autoregressive processes," IEEE Trans. Inf. Theory, vol. IT-16, no. 4, pp. 412-421, Jul. 1970.
18. R. M. Gray, "Rate distortion functions for finite-state finite-alphabet Markov sources," IEEE Trans. Inf. Theory, vol. IT-17, no. 2, pp. 127-134, Mar. 1971.
19. X. Guyon, Random Fields on a Network. New York: Springer-Verlag, 1995.
20. B. E. Hajek and T. Berger, "A decomposition theorem for binary Markov random fields," Ann. Probab., no. 15, pp. 1112-1125, 1987.
21. J. Hannan, "Approximation to Bayes risk in repeated play," in Contributions to the Theory of Games, Ann. Math. Study. Princeton, NJ: Princeton Univ. Press, 1957, vol. III, pp. 97-139.
22. T. Holliday, P. Glynn, and A. Goldsmith. (2003, Submitted for publication.) On Entropy and Lyapunov Exponents for Finite State Channels. [Online]. Available: http://wsl.stanford.edu/Publications/ THolliday/Lyapunov.pdf
23. P. Jacquet, G. Seroussi, and W. Szpankowski, "On the entropy of a hidden Markov process," in Proc. Data Compression Conf., Snowbird, UT, Mar. 2004, pp. 362-371.
24. G. Kallianpur, Stochastic Filtering Theory. New York: Springer-Verlag, 1980.
25. R. Khasminskii and O. Zeitouni, "Asymptotic filtering for finite state Markov chains," Stochastic Processes and their Applications, vol. 63, pp. 1-10, 1996.
26. H. Kunita, "Asymptotic behavior of the nonlinear filtering errors of Markov processes," J. Multivariate Anal., no. 1, pp. 365-393, 1971.
27. N. Merhav and M. Feder, "Universal prediction," IEEE Trans. Inf. Theory, vol. 44, no. 6, pp. 2124-2147, Oct. 1998.
28. N. Merhav and I. Kontoyiannis, "Source coding exponents for zerodelay coding with finite memory," IEEE Trans. Inf. Theory, no. 3, pp. 609-625, Mar. 2003.
29. M. Mushkin and I. Bar-David, "Capacity and coding for the Gilbert-Elliot channel," IEEE Trans. Inf. Theory, vol. 35, no. 6, pp. 1277-1290, Nov. 1989.
30. E. Ordentlich and T. Weissman, "New bounds on the entropy rate of hidden Markov processes," in Proc. IEEE Information Theory Workshop, San Antonio, TX, Oct. 2004.
31. Y. Peres, "Analytic dependence of Lyapunov exponents on transition probabilities," in Proc. Oberwolfack Conf. (Lecture Notes in Mathematics). Berlin, Germany: Springer-Verlag, 1991, vol. 1486, pp. 64-80.
32. N. Phamdo and N. Farvardin, "Optimal detection of discrete Markov sources over discrete memoryless channels-Applications to combined source-channel coding," IEEE Trans. Inf. Theory, vol. 40, no. 1, pp. 186-193, Jan. 1994.
33. D. Sagalowicz, "Hypothesis testing with finite memory," Ph.D. dissertation, Elec. Eng. Dept., Stanford Univ., Stanford, CA, 1970.
34. E. Samuel, "An empirical Bayes approach to the testing of certain parametric hypotheses," Ann. Math. Statist., vol. 34, no. 4, pp. 1370-1385, 1963.
35. L. Shue, S. Dey, B. D. O. Anderson, and F. De Bruyne, "On state-estimation of a two-state hidden Markov model with quantization," IEEE Trans. Signal Process., vol. 49, no. 1, pp. 202-208, Jan. 2001.
36. T.Weissman, "Universally attainable error-exponents for rate-distortion coding of noisy sources," IEEE Trans. Inf. Theory, vol. 50, no. 6, pp. 1229-1246, Jun. 2004.
37. T. Weissman and N. Merhav, "Tradeoffs between the excess-code-length exponent and the excess-distortion exponent in lossy source coding," IEEE Trans. Inf. Theory, vol. 48, no. 2, pp. 396-415, Feb. 2002.
38. T. Weissman and N. Merhav, "On competitive prediction and its relationship to rate-distortion theory," IEEE Trans. Inf. Theory, vol. 49, no. 12, pp. 3185-3194, Dec. 2003.
39. T. Weissman, E. Ordentlich, G. Seroussi, S. Verdú, and M. Weinberger, "Universal discrete denoising: Known channel," IEEE Trans. Inf. Theory, vol. 51, no. 1, pp. 5-28, Jan. 2005.
40. W. M.Wonham, "Some applications of stochastic differential equations to optimal nonlinear filtering," SIAM J. Control Optimiz., vol. 2, pp. 347-368, 1965.