On the averaged stochastic approximation for linear regression

SIAM Journal on Control and Optimization

Volumn 34, Issue 1, 1996, Pages 31-61

  Györfi, László ^a   Walk, Harro ^b  

^a BUDAPEST UNIVERSITY OF TECHNOLOGY AND ECONOMICS   (Hungary)

^b UNIVERSITY OF STUTTGART   (Germany)

Author keywords

Adaptive filtering; Almost sure convergence; Asymptotic normality; Averaged stochastic approximation; Constant gains; Ergodicity; Linear regression; Martingales; Mixing

Indexed keywords

ADAPTIVE FILTERING; ALGORITHMS; CONVERGENCE OF NUMERICAL METHODS; EIGENVALUES AND EIGENFUNCTIONS; FUNCTIONS; MATRIX ALGEBRA; PARAMETER ESTIMATION; REGRESSION ANALYSIS; VECTORS;

ASYMPTOTIC NORMALITY; AVERAGED STOCHASTIC APPROXIMATION; CONSTANT GAIN; ERGODICITY; LINEAR REGRESSION; MARTINGALES;

APPROXIMATION THEORY;

EID: 0029771427     PISSN: 03630129     EISSN: None     Source Type: Journal    
DOI: 10.1137/S0363012992226661     Document Type: Article

References (31)

1. P. BILLINGSLEY, Convergence of Probability Measures, Wiley, New York, 1968.
2. G. COLLOMB, Propriétés de convergence presque complète du prédicteur à noyau, Z. Wahrscheinlichkeitstheorie verw. Gebiete, 66 (1984), pp. 441-460.
3. J. FRITZ, Learning from ergodic training sequence, in Limit Theorems of Probability Theory, P. Révész, ed., North-Holland, Amsterdam, 1974, pp. 79-91.
4. H. FURSTENBERG AND H. KESTEN, Products of random matrices, Ann. Math. Stat., 31 (1960), pp. 457-469.
5. L. GYÖRFI, Stochastic approximation from ergodic sample for linear regression, Z. Wahrscheinlichkeitstheorie verw. Gebiete, 54 (1980), pp. 47-55.
6. _, Adaptive linear procedures under general conditions, IEEE Trans. Inform. Theory, IT-30 (1984), pp. 262-267.
7. L. GYÖRFI, W. HÄRDLE, P. SARDA, AND PH. VIEU, Nonparametric Curve Estimation from Time Series, Springer-Verlag, Berlin, 1989.
8. I. A. IBRAGIMOV, Some limit theorems for stationary processes, Theory Probab. Appl., 7 (1962), pp. 349-382.
9. I. A. IBRAGIMOV AND YU. V. LINNIK, Independent and Stationary Sequences of Random Variables, Wolters-Noordhoff, Groningen, 1971.
10. T. Y. KIM, A note on moment bounds for strong mixing sequences, Statist. Probab. Lett., 16 (1993), pp. 163-168.
11. U. KRENGEL, Ergodic Theorems, de Gruyter, Berlin, New York, 1985.
12. A. KRIEGER AND E. MASRY, Convergence analysis of adaptive linear estimation for dependent stationary processes, IEEE Trans. Inform. Theory, IT-34 (1988), pp. 642-654.
13. H. J. KUSHNER AND A. SHWARTZ, Weak convergence and asymptotic properties of adaptive filters with constant gains, IEEE Trans. Inform. Theory, IT-30 (1984), pp. 177-182.
14. H. J. KUSHNER AND J. YANG, Stochastic approximation with averaging of the iterates: Optimal asymptotic rate of convergence for general processes, SIAM J. Control Optim. 31 (1993), pp. 1045-1062.
15. L. LJUNG, Analysis of stochastic gradient algorithms for linear regression problems, IEEE Trans. Inform. Theory, IT-30 (1984), pp. 151-160.
16. O. MACCHI AND E. EWEDA, Second-order convergence analysis of stochastic adaptive linear filtering, IEEE Trans. Automat. Control, AC-28 (1983), pp. 76-85.
17. D. L. MCLEISH, Invariance principles for dependent variables, Z. Wahrscheinlichkeitstheorie verw. Gebiete, 32 (1975), pp. 165-178.
18. M. MÉTIVIER AND P. PRIOURET, Applications of a Kushner and Clark lemma to general classes of stochastic algorithms, IEEE Trans. Informat. Theory, IT-30 (1984), pp. 140-151.
19. M. PELIGRAD, The convergence of moments in the central limit theorem for p-mixing sequences of random variables, Proc. Amer. Math. Soc., 101 (1987), pp. 142-148.
20. G. CH. PFLUG, Non-asymptotic confidence bounds for stochastic approximation algorithms with constant step size, Monatsh. Math., 110 (1990), pp. 297-314.
21. B. T. POLYAK, New stochastic approximation type procedures, Avtomat. i Telemekh., 51 (1990), pp. 98-107 (in Russian), translated in Automat. Remote Control, 51 (1990), pp. 937-946.
22. B. T. POLYAK, New stochastic approximation type procedures, Avtomat. i Telemekh., 51 (1990), pp. 98-107 (in Russian), translated in Automat. Remote Control, 51 (1990), pp. 937-946.
23. B. T. POLYAK AND A. B. JUDITSKY, Acceleration of stochastic approximation by averaging, SIAM J. Control Optim., 30 (1992), pp. 838-855.
24. H. ROBBINS AND D. SIEGMUND, A convergence theorem for nonnegative almost supermartingales and some applications, in Optimizing Methods in Statistics, J. S. Rustagi, ed., Academic Press, New York, 1971, pp. 233-257.
25. D. RUPPERT, Efficient estimators from a slowly convergent Robbins-Monro process, Technical report no. 781, School of Oper. Res. and Ind. Eng., Cornell University, Ithaca, N.Y., 1988 (see §2.8. of D. Ruppert, Stochastic Approximation, in Handbook of Sequential Analysis, B. K. Ghosh and P. K. Sen, eds., Marcel Dekker, New York, 1991, pp. 503-529).
26. D. RUPPERT, Efficient estimators from a slowly convergent Robbins-Monro process, Technical report no. 781, School of Oper. Res. and Ind. Eng., Cornell University, Ithaca, N.Y., 1988 (see §2.8. of D. Ruppert, Stochastic Approximation, in Handbook of Sequential Analysis, B. K. Ghosh and P. K. Sen, eds., Marcel Dekker, New York, 1991, pp. 503-529).
27. E. SANCHEZ-PALENCIA, Méthode de centrage-estimation de l'erreur et comportement des trajectoires dans l'espace des phases, Internat. J. Non-Linear Mechanics, 11 (1976), pp. 251-263.
28. J. A. SANDERS AND F. VERHULST, Averaging Methods in Nonlinear Dynamical Systems, Springer-Verlag, New York, Berlin, 1985.
29. H. WALK, Almost sure convergence of stochastic approximation processes, Statist. Decisions, supplement issue 2 (1985), pp. 137-141.
30. H. WALK AND L. ZSIDÓ, Convergence of the Robbins-Monro method for linear problems in a Banach space, J. Math. Anal. Appl., 139 (1989), pp. 152-177.
31. G. YIN, On extensions of Polyak's averaging approach to stochastic approximation, Stochastics Stochastics Rep., 36 (1991), pp. 245-264.