Universal prediction

IEEE Transactions on Information Theory

Volumn 44, Issue 6, 1998, Pages 2124-2147

  Merhav, Neri ^a,b   Feder, Meir ^a,c  

^a IEEE   (Israel)

^b TECHNION ISRAEL INSTITUTE OF TECHNOLOGY   (Israel)

^c TEL AVIV UNIVERSITY   (Israel)

Author keywords

Bayes envelope; Entropy; Finite state machine; Linear prediction; Loss function; Probability assignment; Redundancy capacity; Stochastic complexity; Universal coding; Universal prediction

Indexed keywords

DATA COMPRESSION; ENCODING (SYMBOLS); FINITE AUTOMATA; INFORMATION THEORY; PROBABILITY; RANDOM PROCESSES; REDUNDANCY;

BAYES ENVELOPE; LINEAR PREDICTION; LOSS FUNCTION; PROBABILITY ASSIGNMENT; STOCHASTIC COMPLEXITY; UNIVERSAL CODING; UNIVERSAL PREDICTION;

FORECASTING;

EID: 0032184197     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/18.720534     Document Type: Article

References (129)

1. R. Agrawal, D. Teneketzis, and V. Anantharam, "Asymptotic adaptive allocation schemes for controlled i.i.d. processes: Finite parameter case," IEEE Trans. Automat. Contr.. vol. 34, pp. 258-267, Mar. 1989.
2. "Asymptotic adaptive allocation schemes for controlled Markov chains: Finite parameter case," IEEE Trans. Automat. Contr., vol. 34, pp. 1249-1259, Mar. 1989.
3. P. H. Algoet, "Universal schemes for prediction, gambling and portfolio selection," A««. Probab., vol. 20, pp. 901-941, Apr. 1992.
4. "The strong law of large numbers for sequential decision under uncertainty," IEEE Trans. Inform. Theory, vol. 40, pp. 609-633, May 1994.
5. P. H. Algoet and T. M. Cover, "Asymptotic optimality and asymptotic equipartition properties of log-optimal investment," Ann. Probab., vol. 16, no. 2, pp. 876-898, 1988.
6. D. H. Bailey, "Sequential schemes for classifying and predicting ergodic processes," Ph.D. dissertation, Stanford Univ., Stanford, CA, 1976.
7. J. M. Bernardo, "Reference posterior distributions for Bayesian infer-ence," J. Roy. Statist. Soc. B, vol. 41, no. 2, pp. 113-147, 1979.
8. D. Blackwell, "An analog to the minimax theorem for vector payoffs," Pac. J. Math., vol. 6, pp. 1-8, 1956.
9. "Controlled random walks," in P roc. Int. Congress Math., vol. 3. Amsterdam, The Netherlands: North Holland, 1956, pp. 336-338.
10. A. Blum, "On-line algorithms in machine learning." [Online.] Available WWW: http://www.cs.cmu.edu/afs/cs.cmu.edu/user/ avrim/www/ Papers/pubs.html.
11. A. C. Blumer, "Minimax universal noiseless coding for unifilar and Markov sources," IEEE Trans. Inform. Theory, vol. IT-33, pp. 925-930, Nov. 1987.
12. N. Cesa-Bianchi, Y. Freund, D. P. Helmbold, D. Haussler, R. E. Schapire, and M. K. Warmuth, "How to use expert advice," in Annu. ACM Symp. Theory of Computing, 1993, pp. 382-391.
13. N. Cesa-Bianchi, Y. Freund, D. P. Helmbold, and M. K. Warmuth, "On-line prediction and conversion strategies," in Proc. EUROCOLT'93, (Oxford, U.K., 1993), pp. 205-216.
14. N. Cesa-Bianchi and G. Lugosi, "On sequential prediction of individual sequences relative to a set of experts," 1998, preprint.
15. B. S. Clarke and A. R. Barron, "Information-theoretic asymptotics of Bayesian methods," IEEE Trans. Inform. Theory, vol. 36, pp. 453-471, May 1990.
16. B. S. Clarke and A. R. Barron, "Jeffreys' prior is asymptotically least favorable under entropy risk," J. Statist. Plan. Inform., vol. 41, pp. 37-60, Aug. 1994.
17. T. M. Cover, "Universal gambling schemes and the complexity measures of Kolmogorov and Chaitin," Tech. Rep. 12, Dept. Statist., Stanford Univ., Stanford, CA, Oct. 1974.
18. "Open problems in information theory," in Proc. Moscow Information Theory Workshop. New York: IEEE Press, 1975, pp. 35-36.
19. T. M. Cover and R. King, "A convergent gambling estimate of the entropy of English," IEEE Trans. Inform. Theory, vol. IT-24, pp. 413-121, July 1978.
20. T. M. Cover, "On the competitive optimality of Huffman code," IEEE Trans. Inform. Theory, vol. 37, pp. 172-174, Jan. 1991.
21. "Universal portfolios," Math. Finance, vol. 1, no. 1, pp. 1-29, Jan. 1991.
22. T. M. Cover and E. Ordentlich, "Universal portfolios with side in-formation," IEEE Trans. Inform. Theory, vol. 42, pp. 348-363, Mar. 1996.
23. T. M. Cover and J. A. Thomas, Elements of Information Theory. New York: Wiley, 1991.
24. I. Csiszâr and J. Körner, Information Theory-Coding Theorems for Discrete Memoryless Systems. New York: Academic, 1981.
25. I. Csiszâr and P. C. Shields, "Redundancy rates for renewal and other processes," IEEE Trans. Inform. Theory, vol. 42, pp. 2065-2072, Nov. 1996.
26. L. D. Davisson, "The prediction error of stationary Gaussian time series of unknown covariance," IEEE Trans. Inform. Theory, vol. IT-11, pp. 527-532, Oct. 1965.
27. "Universal noiseless coding," IEEE Trans. Inform. Theory, vol. IT-19, pp. 783-795, Nov. 1973.
28. "Minimax noiseless universal coding for Markov sources," IEEE Trans. Inform. Theory, vol. IT-29, pp. 211-215, Mar. 1983.
29. L. D. Davisson and A. L. Garcia, "A source matching approach to finding minimax codes," IEEE Trans. Inform. Theory, vol. IT-26, pp. 166-174, Mar. 1980.
30. L. D. Davisson, R. J. McEliece, M. B. Pursley, and M. S. Wallace, "Ef-ficient universal noiseless source codes," IEEE Trans. Inform. Theory, vol. 27, pp. 269-278, May 1981.
31. A. P. Dawid, "Present position and potential developments: Some personal views on statistical theory the prequential approach (with discussion)," J. Roy. Statist. Soc. A, vol. 147, pt. 2, pp. 278-292, 1984.
32. "Fisherian inference in likelihood and prequential frames of reference (with discussion)," J. Roy. Statist. Soc. B, vol. 53, pp. 79-109, 1991.
33. "Prequential data analysis," in Current Issues in Statistical Inference (IMS Lecture Notes-Monograph Series 17), M. Ghosh and P. K. Pathak, Eds., 1992, pp. 113-126.
34. A. P. Dawid and V. G. Vovk, "Prequential probability: Principles and properties." [Online.] Available WWW: http://www-stat.wharton.upenn. edu/Seq96/members/vovk/index. html.
35. M. H. DeGroot, Optimal Statistical Decisions. New York: McGraw-Hill, 1970.
36. A. DeSantis, G. Markowsky, and M. Wegman, "Learning probabilis-tic prediction functions," in Proc. 29th IEEE Symp. Foundations of Computer Science, 1988, pp. 110-119.
37. P. Elias, "Minimax optimal universal codeword sets," IEEE Trans. Inform. Theory, vol. IT-29, pp. 491-502, July 1983.
38. M. Feder, "Gambling using a finite state machine," IEEE Trans. Inform. Theory, vol. 37, pp. 1459-1465, Sept. 1991.
39. M. Feder and N. Merhav, "Hierarchical universal coding," IEEE Trans. Inform. Theory, vol. 42, pp. 1354-1364, Sept. 1996.
40. M. Feder, N. Merhav, and M. Gutman, "Universal prediction of indi-vidual sequences," IEEE Trans. Inform. Theory, vol. 38, pp. 1258-1270, July 1992.
41. M. Feder and A. Singer, "Universal data compression and linear prediction," in Proc. DCC'98, 1998, pp. 511-520.
42. B. Fittingoff, "Universal methods of coding for the case of unknown statistics," in Proc. 5th Symp. Information Theory (Moscow/Gorky, USSR, 1972), pp. 129-135.
43. Y. Freund and R. Schapire, "Game theory, on-line prediction and boosting," in Proc. 9th Ann. Workshop Computational Learning Theory, 1996. pp. 89-98.
44. R. G. Gallager, Information Theory and Reliable Communications. New York: Wiley, 1968.
45. "Source coding with side information and universal coding," unpublished manuscript; also presented at the Int. Symp. Information Theory, Oct. 1974.
46. D. C. Gilliland, "Sequential compound estimation," Ann. Math. Statist., vol. 39, no. 6, pp. 1890-1904, 1968.
47. R. M. Gray, Entropy and Information Theory. New York: Springer-Verlag, 1990.
48. L. Györfi, I. Pali, and E. C. van der Meulen, "There is no universal source code for an infinite source alphabet," IEEE Trans. Inform. Theory, vol. 40, pp. 267-271, Jan. 1994.
49. J. F. Hannan, "Approximation to Bayes risk in repeated plays," in Contributions to the Theory of Games, Ann. Math. Studies (Princeton Univ., Princeton, NJ), vol. 3, no. 39, pp. 97-139, 1957.
50. J. F. Hannan and H. Robbins, "Asymptotic solutions of the compound decision problem for two completely specified distributions," Ann. Math. Statist., vol. 26, pp. 37-51, 1955.
51. D. Haussier, "A general minimax result for relative entropy," IEEE Trans. Inform. Theory, vol. 43, pp. 1276-1280, July 1997.
52. D. Haussler, J. Kivinen, and M. K. Warmuth, "Sequential prediction of individual sequences under general loss functions," IEEE Trans. Inform. Theory, to be published.
53. D. Haussler and M. Opper, "Mutual information, metric entropy and cumulative relative entropy risk," Ann. Statist., vol. 25, no. 6, Dec. 1997. to be published
54. "General bounds on the mutual information between a pa-rameter and n conditionally independent observations," in Proc. 8th Annu. Workshop Computational Learning Theory (COLT'95), 1995, pp. 402-111.
55. D. P. Helmbold and R. E. Schapire, "Predicting nearly as well as the best pruning of a decision tree," Mach. Learn., vol. 27, pp. 51-68, 1997.
56. Y. Hershkovitz and J. Ziv, "On fixed-database universal data compres-sion with limited memory," IEEE Trans. Inform. Theory, vol. 43, pp. 1966-1976, Nov. 1997.
57. H. Jeffreys, "An invariant form of for the prior probability in estimation problems," Proc. Roy. Soc. London, vol. 186, pt. A, pp. 453161, 1946.
58. D. Kazakos, "Robust noiseless source coding through a game theoretic approach," IEEE Trans. Inform. Theory, vol. IT-29, pp. 576-583, July 1983.
59. J. L. Kelly, Jr., "A new interpretation of information rate," Bell Syst. Tech. J., vol. 35, pp. 917-926, 1956.
60. J. C. Kieffer, "A unified approach to weak universal source coding," IEEE Trans. Inform. Theory, vol. IT-24, pp. 674-682, Nov. 1978.
61. "An ergodic theorem for constrained sequences of functions," Bull. Amer. Math. Soc., vol. 21, pp. 249-253, 1989.
62. R. E. Krichevskiy, "Laplace's law of succession and universal encoding," IEEE Trans. Inform. Theory, vol. 44, pp. 296-303, Jan. 1998.
63. R. E. Krichevski and V. E. Trofimov, "The performance of universal encoding," IEEE Trans. Inform. Theory, vol. IT-27, pp. 199-207, Mar. 1981.
64. T. L. Lai and H. Robbins, "Asymptotically efficient adaptive allocation rules," Adv. Appl. Math., vol. 6, pp. 4-22, 1985.
65. G. G. Langdon, "A note on the Lempel-Ziv model for compressing individual sequences," IEEE Trans. Inform. Theory, vol. IT-29, pp. 284-287, 1983.
66. P. S. Laplace, "Mémoire sur la probabilité des causes par les evenemens," Mémoires de l'Académie Royale del Sciences, no. 6, pp. 612-656, 1774, reprinted in
67. Laplace Complete Work, vol. 8. Paris, France: Gauthier-Villars, pp. 27-65. English translation by S. M. Stigler, 1986.
68. "Mémoire sur les approximations des formulas qui sont functions de très grands nombres et sur leur application aux probabilies," Mémoires de l'Académie des Sciences de Paris, pp. 353-415, 559-565, 1810; reprinted in
69. Laplace Complete Work, vol. 12. Paris, France: Gauthier-Villars, pp. 301-353; English translation by S. M. Stigler, 1986.
70. A. Lempel and J. Ziv, "On the complexity of finite sequences," IEEE Trans. Inform. Theory, vol. IT-22, pp. 75-81, Jan. 1976.
71. N. Littlestone, P. Long, and M. K. Warmuth, "On-line learning of linear functions," in Proc. 23rd Annu. ACM Symp. Theory of Computing, 1991, pp. 382-391.
72. N. Littlestone and M. K. Warmuth, "The weighted majority algorithm," Inform. Comput., vol. 108, no. 2, pp. 212-261, 1994.
73. G. Louchard and W. Szpankowski, "On the average redundancy rate of the Lempel-Ziv code," IEEE Trans. Inform. Theory, vol. 43, pp. 2-8, Jan. 1997.
74. T. Matsushima, H. Inazumi, and S. Hirawasa, "A class of distortionless codes designed by Bayes decision theory," IEEE Trans. Inform. Theory, vol. 37, pp. 1288-1293, Sept. 1991.
75. R. Meir, "Performance bounds for nonlinear time-series prediction," 1997, preprint.
76. R. Meir and N. Merhav, "On the stochastic complexity of learning realizable and unrealizable rules," Mach. Learn., vol. 19, no. 3, pp. 241-261, 1995.
77. N. Merhav and M. Feder, "Universal schemes for sequential decision from individual sequences," IEEE Trans. Inform. Theory, vol. 39, pp. 1280-1292, July 1993.
78. "A strong version of the redundancy-capacity theorem of uni-versal coding," IEEE Trans. Inform. Theory, vol. 41, pp. 714-722, May 1995.
79. N. Merhav, M. Feder, and M. Gutman, "Some properties of sequential predictors of binary Markov sources," IEEE Trans. Inform. Theory, vol. 39, pp. 887-892, May 1993.
80. J. W. Miller, R. Goodman, and P. Smyth, "On loss functions which minimize to conditional expected values and posterior probabilities," IEEE Trans. Inform. Theory, vol. 39, pp. 1404-1408, July 1993.
81. D. S. Modha and E. Masry, "Universal prediction of stationary random processes," 1996, preprint.
82. G. Morvai, S. J. Yakowitz, and L. Györfi, "Nonparametric inference for ergodic, stationary time series," A««. Statist., vol. 24, pp. 370-379, 1996.
83. G. Morvai, S. J. Yakowitz, and P. H. Algoet, "Weakly convergent non-parametric forecasting of stationary time series," IEEE Trans. Inform. Theory, vol. 43, pp. 483-497, Mar. 1997.
84. Y. Nogami, "The fc-extended set-compound estimation problem in nonregular family of distributions," Ann. Inst. Stat. Math., vol. 31A, pp. 169-176, 1979.
85. M. Opper and D. Haussler, "Bounds for predictive errors in the statistical mechanics of supervised learning," Phys. Rev. Lett., vol. 75, pp. 3772-3775, 1995.
86. "Worst case prediction over sequences under log loss," in The Mathematics of Information Coding, Extraction and Distribution, G. Cybenko, D. O'Leary, and J. Rissanen, Eds. New York: Springer-Verlag, 1997.
87. D. S. Orenstein, "Guessing the next output of a stationary process," Israel J. Math., vol. 30, pp. 292-296, 1978.
88. A. Papoulis, Probability, Random Variables, and Stochastic Processes, 3rd ed. (McGraw-Hill Series in Electrical Engineering). New York: McGraw-Hill, 1991.
89. E. Plotnik, M. J. Weinberger, and J. Ziv, "Upper bounds on the probability of sequences emitted by finite-state sources and on the redundancy of the Lempel-Ziv algorithm," IEEE Trans. Inform. Theory, vol. 38, pp. 66-72, Jan. 1992.
90. J. Rissanen, "Generalized Kraft' s inequality and arithmetic coding," IBM J. Res. Develop., vol. 20, no. 3, pp. 198-203, 1976.
91. "Modeling by shortest data description," Automatica, vol. 14,pp. 465, 1978.
92. "Universal coding, information, prediction, and estimation," IEEE Trans. Inform. Theory, vol. IT-30, pp. 629-636, July 1984.
93. "Complexity of strings in the class of Markov sources," IEEE Trans. Inform. Theory, vol. IT-32, pp. 526-532, 1986.
94. "Fisher information and stochastic complexity," IEEE Trans. Inform. Theory, vol. 42, pp. 40-47, Jan. 1996.
95. J. Rissanen and G. G. Langdon, "Universal modeling and coding," IEEE Trans. Inform. Theory, vol. IT-27, pp. 12-23, Jan. 1984.
96. H. Robbins, "Asymptotically subminimax solutions of compound statis-tical decision problems," in Proc. 2nd Berkeley Symp. Math. Stat. Prob., 1951, pp. 131-148.
97. R. T. Rockafeller, Convex Analysis. Princeton, NJ: Princeton Univ. Press, 1970.
98. B. Ya. Ryabko, "Encoding a source with unknown but ordered proba-bilities," Probl. Inform. Transm., pp. 134-138, Oct. 1979.
99. "Twice-universal coding," Probl. Inform. Transm., vol. 20, no. 3, pp. 173-177, July/Sept. 1984.
100. "Prediction of random sequences and universal coding," Probl. Inform. Transm., vol. 24, no. 2, pp. 87-96, Apr./June 1988.
101. E. Samuel, "Asymptotic solution of the sequential compound decision problem," Ann. Math. Statist., pp. 1079-1095, 1963.
102. "Convergence of the losses of certain decision rules for the sequential compound decision problem," Ann. Math. Statist., pp. 1606-1621, 1964.
103. S. A. Savari, "Redundancy of the Lempel-Ziv incremental parsing rule," IEEE Trans. Inform. Theory, vol. 43, pp. 9-21, Jan. 1997.
104. B. Scarpellini, "Conditional expectations of stationary processes," Z. Wahrscheinlichkeitstheorie Verw. Gebiete, vol. 56,pp. 427-441, 1981.
105. G. Schwarz, "Estimating the dimension of a model," A««. Statist., vol. 6, no. 2, pp. 46164, 1978.
106. C. E. Shannon, "Prediction and entropy of printed English," Bell Syst. Tech. ]., vol. 30, pp. 5-64, 1951.
107. "The mind reading machine," in Shannon's Collected Papers, A. D. Wyner and N. J. A. Sloane, Eds. New York: IEEE Press, 1993, pp. 688-689.
108. P. C. Shields, "Uniform redundancy rates do not exist," IEEE Trans. Inform. Theory, vol. 39, pp. 520-524, Mar. 1993.
109. P. C. Shields and B. Weiss, "Universal redundancy rates for the class of B-processes do not exist," IEEE Trans. Inform. Theory, vol. 41, pp. 508-512, Mar. 1995.
110. N. Shimkin, "Dynamic decision problems in multi-user systems," Ph.D. dissertation, Technion-Israel Inst. Technol., Haifa, Isreal, Nov. 1991.
111. Y. M. Shtar'kov, "Universal sequential coding of single messages," Problems of Inform. Trans., vol. 23, no. 3, pp. 175-186, July/Sept. 1987.
112. A. Singer and M. Feder, "Universal linear prediction over parameters and model orders," IEEE Trans. Signal Processing, to be published.
113. J.-I. Takeuchi and A. R. Barron, "Asymptotically minimax regret for exponential and curved exponential families," 1998, preprint.
114. J. van Ryzin, "The sequential compound decision problem with m x n finite loss matrix," Ann. Math. Statist., vol. 37, pp. 954-975, 1966.
115. S. B. Vardeman, "Admissible solutions of fc-extended finite state set and the sequence compound decision problems," J. Multiv. Anal., vol. 10, pp. 426-441, 1980.
116. J. S. Vitter, "Optimal prefetching via data cxompression," in Proc. Conf. Foundations of Computer Science, 1991, pp. 121-130.
117. V. G. Vovk, "Aggregating strategies," in Proc. 3rd Anna. Workshop Computational Learning Theory (San Mateo, CA, 1990), pp. 371-383.
118. "A game of prediction with expert advice," in Proc. 3rd Annu. Workshop Computational Learning Theory (New York, NY, 1995), pp. 51-60.
119. M. J. Weinberger, N. Merhav, and M. Feder, "Optimal sequential probability assignment for individual sequences," IEEE Trans. Inform. Theory, vol. 40, pp. 384-396, Mar. 1994.
120. M. J. Weinberger, J. Rissanen, and M. Feder, "A universal finite memory source," IEEE Trans. Inform. Theory, vol. 41, pp. 643-652, May 1995.
121. N. Wiener, Extrapolation, Interpolation, and Smoothing of Stationary Time Series. Cambridge, MA: MIT Press, 1949.
122. F. M. J. Willems, Y. M. Shtarkov, and T. Tjalkens, "The context-tree weighting method: Basic properties," IEEE Trans. Inform. Theory, vol. 41, pp. 653-664, May 1995
123. Q. Xie and A. R. Barron, "Asymptotic minimax regret for data com-pression, gambling, and prediction," IEEE Trans. Inform. Theory, 1996, submitted for publication.
124. "Minimax redundancy for the class of memoryless sources," IEEE Trans. Inform. Theory, vol. 43, pp. 646-657, Mar. 1997.
125. B. Yu, "Lower bounds on expected redundancy for nonparametric classes," IEEE Trans. Inform. Theory, vol. 42, pp. 272-275, Jan. 1996.
126. J. Ziv, "Coding of sources with unknown statistics-Part I: Probability of encoding error," IEEE Trans. Inform. Theory, vol. IT-18, pp. 384-394, May 1972.
127. J. Ziv and A. Lempel, "A universal algorithm for sequential data compression," IEEE Trans. Inform. Theory, vol. IT-23, pp. 337-343, July 1977.
128. "Compression of individual sequences via variable-rate coding," IEEE Trans. Inform. Theory, vol. IT-24, pp. 530-536, Sept. 1978.
129. "Universal coding of two-dimensional data," IEEE Trans. In-form. Theory, vol. IT-32, pp. 2-8, Jan. 1986.