Memory-Universal Prediction of Stationary Random Processes

IEEE Transactions on Information Theory

Volumn 44, Issue 1, 1998, Pages 117-133

  Modha, Dharmendra S ^a  

^a IBM ALMADEN RESEARCH CENTER   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL COMPLEXITY; DATA STORAGE EQUIPMENT; LEAST SQUARES APPROXIMATIONS; MATHEMATICAL MODELS; NEURAL NETWORKS; POLYNOMIALS; TIME SERIES ANALYSIS;

BERNSTEIN INEQUALITY; LEGENDRE POLYNOMIALS; MEMORY UNIVERSAL PREDICTION;

MARKOV PROCESSES;

EID: 0031647695     PISSN: 00189448     EISSN: None     Source Type: Journal    
DOI: 10.1109/18.650998     Document Type: Article

References (63)

1. H. Akaike, "A new look at the statistical model identification," IEE Trans. Automat. Contr., vol. AC-19, pp. 716-723, 1974.
2. P.H. Algoet, "Universal schemes for prediction, gambling, and portfoli selection," A. Probab., vol. 20, no. 2, pp. 901-941, 1992. Correction ibid., vol. 23, pp. 474-478, 1995.
3. -, "The strong law of large numbers for sequential decisions unde uncertainty," IEEE Trans. Inform. Theory, vol. 40, pp. 609-633, Ma 1994.
4. K.B. Athreya and S. G. Pantula, "Mixing properties of Harris chain and autoregressive processes," J. Appl. Probab., vol. 23, pp. 880-892 1986.
5. B. Auestad and D. Tj0stheim, "Identification of nonlinear time series First order characterization and order determination," Biometrika, vol 77, no. 4, pp. 669-687, 1990.
6. -, "Functional identification in nonlinear time series," in Proceedings NATO Advanced Study Institute on Nonparametric Functional Estimation, G. Roussas, Ed. Dordrecht, The Netherlands: Kluwer, 1991
7. A.D. Back and A. C. Tsoi, "FIR and IIR synapses, a new neural network architecture for time series modeling," Neural Comput., vol. 3, pp. 375-385, 1991
8. A.R. Barron, "Complexity regularization," in Proceedings NATO Advanced Study Institute on Nonparametric Functional Estimation, G. Roussas, Ed. Dordrecht, The Netherlands: Kluwer, 1991, pp. 561-576.
9. __, "Universal approximation bounds for superpositions of a sigmoidal function," IEEE Trans. Inform. Theory, vol. 39, pp. 930-945, May 1993
10. __, "Approximation and estimation bounds for artificial neural networks," Mach. Learn., vol. 14, pp. 115-133, 1994
11. __, "Asymptotically optimal model selection and neural nets," in P roc. 1994 IEEE-IMS Workshop on Information Theory and Statistics. New York: IEEE Press, Oct. 1994, p. 35
12. A.R. Barron, L. Birgé, and P. Massart, "Risk bounds for model selection via penalization," 1995, preprint
13. A.R. Barron and T. M. Cover, "Minimum complexity density estimation," IEEE Trans. Inform. Theory, vol. 37, pp. 1034-1054, July 1991
14. R.J. Bhansali, "Order selection for linear time series models: A review," in Developments in Time Series Analysis: In Honour of M. B. Priestley, T. Subba Rao, Ed. New York: Chapman and Hall, 1993, pp. 50-66
15. D. Bosq, Nonparametric Statistics for Stochastics Processes: Estimation and Prediction. New York: Springer-Verlag, 1996
16. C. Canuto and A. Quarteroni, "Approximation results for orthogonal polynomials in Sobolev spaces," Math, of Computation, vol. 38, pp. 67-86, 1982
17. B. Cheng and H. Tong, "On consistent nonparametric order determination and chaos," J. Roy. Statist. Soc., B, vol. 54, no. 2, pp. 427-4149, 1992
18. T.M. Cover, "Open problems in information theory," in Proc. 1975 IEEE-USSR Joint Workshop on Information Theory, (Moscow, USSR, Dec. 15-19, 1975). New York: IEEE Press, 1975
19. Y.A. Davydov, "Mixing conditions for Markov chains," Theory Probab. Applications, vol. XVIII, no. 2, pp. 312-328, 1973
20. L. Devroye, L. Györfi, and G. Lugosi, A Probabilistic Theory of Pattern Recognition. New York: Springer-Verlag, 1996
21. P. Doukhan, Mixing: Properties and Examples. New York: SpringerVerlag, 1994
22. R. Durrett, Probability: Theory and Examples. Pacific Grove, CA: Wadsworth & Brooks, 1991
23. A. Farago and G. Lugosi, "Strong universal consistency of neural network classifiers," IEEE Trans. Inform. Theory, vol. 39, pp. 1146-1151, July 1993
24. P. Hall and C. C. Heyde, Martingale Limit Theory and Its Application. New York: Academic, 1980
25. K. Hornik, M. B. Stinchcombe, H. White, and P. Auer, "Degree of approximation results for feedforward networks approximating unknown mappings and their derivatives," Neural Comput., vol. 6, pp. 1262-1275 1994.
26. J.M. Hutchinson, A. W. Lo, and T. Poggio, "A nonparametric approac to pricing and hedging derivative securities via learning networks," J Finance, vol. XLIX, no. 3, July 1994.
27. L.K. Jones, "The computational intractability of training sigmoida neural networks," IEEE Trans. Inform. Theory, vol. 43, pp. 167-173 Jan. 1997.
28. G. Lugosi and A. Nobel, "Adaptive model selection using empirica complexities," 1995, preprint.
29. G. Lugosi and K. Zeger, "Nonparametric estimation via empirical ris minimization," IEEE Trans. Inform. Theory, vol. 41, pp. 677-687, Ma 1995.
30. __, "Concept learning using complexity regularization," IEEE Trans Inform. Theory, vol. 42, pp. 48-54, Jan. 1996.
31. P. Masani and N. Wiener, "Non-linear prediction," in Probability an Statistics: The Harald Cramer Volume, U. Grenander, Ed. Stockholm Sweden: Almqvist and Wiksell, 1959, pp. 190-212.
32. D.F. McCaffrey and A. R. Gallant, "Convergence rates for single hidde layer feedforward networks," Neural Net., vol. 7, no. 1, pp. 147-158 1994.
33. R.W. Means, B. Wallach, D. Busby, and R. Lengel, Jr., "Bispectru signal processing on HNC's SIMD numerical array processor (SNAP),"
34. in 1993 P roc. Supercomputing (Portland, OR, Nov. 15-19, 1993), pp. 535-537; Los Alamitos, ÇA: IEEE Comput. Soc. Press, 1993
35. D.S. Modha and E. Masry, "Minimum complexity regression estimation with weakly dependent observations," IEEE Trans. Inform. Theory, vol. 42, pp. 2133-2145, Nov. 1996
36. G. Morvai, S. Yakowitz, and P. Algoet, "Weakly convergent nonparametric forecasting of nonparametric stationary time series," 1996, submitted for publication
37. G. Morvai, S. Yakowitz, and L. Györfi, "Nonparametric inferences for ergodic, stationary time series," A. Statist., vol. 24, no. 1, pp. 370-379, 1996
38. J. Rissanen, "A universal data compression system," IEEE Trans. Inform. Theory, vol. IT-29, pp. 656-664, Sept. 1983
39. __, "Complexity of strings in the class of Markov sources," IEEE Trans. Inform. Theory, vol. IT-32, pp. 526-532, July 1986
40. __, Stochastic Complexity in Statistical Inquiry. Teaneck, NJ: World Scientific, 1989
41. P.M. Robinson, "Nonparametric estimators for time series," J. Time Series Anal., vol. 4, pp. 185-297, 1983
42. G.G. Roussas, "Nonparametric regression estimation under mixing conditions," Stochastic Process. Appl, vol. 36, pp. 107-116, 1990
43. M. Rosenblatt, "A central limit theorem and strong mixing conditions," Proc. Nat. Acad. Sei, vol. 4, pp. 43-417, 1956
44. B.Y. Ryabko, "Twice-universal coding," Probl. Inform. Transm., vol. 20, pp. 173-177, July-Sept. 1984
45. __, "Prediction of random sequences and universal coding," Probl. Inform. Transm., vol. 24, pp. 87-96, Apr.-June 1988
46. E. Sackinger and H. P. Graf, "A board system for high-speed image analysis and neural networks," IEEE Trans. Neural Net., vol. 7, pp. 214-221, Jan. 1996
47. C.-H. Sheu, "Density estimation with Kullback-Leibler loss," Ph.D. dissertation, Dept. Statistics, Univ. Illinois at Urbana-Champaign, 1990
48. C.-Y. Sin and H. White, "Information criteria for selecting possibly misspecified parametric models," 1995, preprint
49. C.J. Stone, "Consistent nonparametric regression (with discussion)," Ann. Statist., vol. 5, pp. 549-645, 1977
50. G. Szegö, Orthogonal Polynomials. New York: American Math. Soc., 1939
51. D. Tj0stheim, "Non-linear time series: A selective review," Scandinavian J. Statist., vol. 21, pp. 97-130, 1994
52. V. Vapnik, Estimation of Dependences Based on Empirical Data. New York: Springer-Verlag, 1982
53. -, The Nature of Statistical Learning Theory. New York: Springer-Verlag, 1995
54. E.A. Wan, "Discrete time neural networks," J. Appl. Intel!., vol. 3, pp. 91-105, 1993
55. A.S. Weigend and N. A. Gershenfeld, "The future of time series: Learning and understanding," in Time Series Prediction: Forecasting the Future and Understanding the Past: Proc. NATO Advanced Research Workshop on Comparative Time Series Analysis, A. S. Weigend and N. A. Gershenfeld, Eds. Reading, MA: Addison-Wesley, 1994
56. H. White, "Connectionist nonparametric regression: Multilayer feedforward networks can learn arbitrary mappings," Neural Net., vol. 3, pp. 535-549, 1989
57. -, "Nonparametric estimation of conditional quantités using neural networks," in Computing Science and Statistics, Statistics of Many Parameters: Curves, Images, Spatial Models, Proc. 22nd Symp. on the Interface, C. Page and R. LePage, Eds. New York: Springer-Verlag, 1992, pp. 190-199
58. H. White and J. M. Wooldridge, "Some results on sieve estimation with dependent observations," in Nonparametric and Semiparametric Methods in Econometrics and Statistics: Proc. 5th Int. Symp. in Economic Theory and Econometrics, W. A. Barnett, J. Powell, and G. Tauchen, Eds. New York: Cambridge Univ. Press, 1991
59. B. Widrow and S. D. Stearns, Adaptive Signal Processing. Englewood Cliffs, NJ: Prentice-Hall, 1985
60. C.S. Withers, "Conditions for linear processes to be strong-mixing," Z. Wahrscheinlichkeitstheorie verw. Gebiete, vol. 57, pp. 477-4180, 1981
61. L. Wu, M. Niranjan, and F. Fallside, "Fully vector-quantized neural network-based code-excited nonlinear predictive speech coding," IEEE Trans. Speech Audio Processing, vol. 2, pp. 482-4189, Oct. 1994
62. Y. Yang and A. R. Barron, "An asymptotic property of model selection criteria," 1995, preprint
63. J.E. Yukich, M. B. Stinchcombe, and H. White, "Sup-norm approximation bounds for networks through probabilistic methods," IEEE Trans. Inform. Theory, vol. 41, pp. 1021-1027, July 1995.