On the use of cross-validation for time series predictor evaluation

Information Sciences

Volumn 191, Issue , 2012, Pages 192-213

  Bergmeir, Christoph ^a   Benítez, José M ^a  

^a UNIVERSITY OF GRANADA   (Spain)

Author keywords

Cross validation; Error measures; Machine learning; Predictor evaluation; Regression; Time series

Indexed keywords

CROSS VALIDATION; ERROR MEASURES; MACHINE-LEARNING; PREDICTOR EVALUATION; REGRESSION;

FORECASTING; LEARNING SYSTEMS; REGRESSION ANALYSIS;

TIME SERIES;

EID: 84857652807     PISSN: 00200255     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.ins.2011.12.028     Document Type: Article

References (53)

1. N. Amjady, and F. Keynia Application of a new hybrid neuro-evolutionary system for day-ahead price forecasting of electricity markets Applied Soft Computing Journal 10 3 2010 784 792
2. R.D.A. Araujo Swarm-based translation-invariant morphological prediction method for financial time series forecasting Information Sciences 180 24 2010 4784 4805
3. S. Arlot, and A. Celisse A survey of cross-validation procedures for model selection Statistics Surveys 4 2010 40 79
4. J.S. Armstrong Significance tests harm progress in forecasting International Journal of Forecasting 23 2 2007 321 327
5. J.S. Armstrong Statistical significance tests are unnecessary even when properly done and properly interpreted: Reply to commentaries International Journal of Forecasting 23 2 2007 335 336
6. J.L. Aznarte, and J.M. Benítez Equivalences between neural-autoregressive time series models and fuzzy systems IEEE Transactions on Neural Networks 21 9 2010 1434 1444
7. S.D. Balkin, and J.K. Ord Automatic neural network modeling for univariate time series International Journal of Forecasting 16 4 2000 509 515
8. G. Box, and G. Jenkins Time Series Analysis: Forecasting and Control 1970 Holden-Day
9. Prabir Burman, Edmond Chow, and Deborah Nolan A cross-validatory method for dependent data Biometrika 81 2 1994 351 358
10. P.S. Carmack, W.R. Schucany, J.S. Spence, R.F. Gunst, Q. Lin, and R.W. Haley Far casting cross-validation Journal of Computational and Graphical Statistics 18 4 2009 879 893
11. Chih-Chung Chang, Chih-Jen Lin, LIBSVM: a library for support vector machines, 2001 < http://www.csie.ntu.edu.tw/cjlin/libsvm >.
12. C. Chatfield Model uncertainty and forecast accuracy Journal of Forecasting 15 7 1996 495 508 (Pubitemid 126704487)
13. B.-J. Chen, M.-W. Chang, and C.-J. Lin Load forecasting using support vector machines: a study on eunite competition 2001 IEEE Transactions on Power Systems 19 4 2004 1821 1830
14. Y. Chen, B. Yang, Q. Meng, Y. Zhao, and A. Abraham Time-series forecasting using a system of ordinary differential equations Information Sciences 181 1 2011 106 114
15. Zhuo Chen, Yuhong Yang, Assessing forecast accuracy measures. Technical Report 2004-10, Iowa State University, Department of Statistics & Statistical Laboratory, 2004.
16. R.B. Cleveland, W.S. Cleveland, J.E. McRae, and I. Terpenning Stl: a seasonal-trend decomposition procedure based on loess (with discussion) Journal of Official Statistics 6 1990 3 73
17. W.J. Conover, Mark E. Johnson, and Myrle M. Johnson Comparative study of tests for homogeneity of variances, with applications to the outer continental shelf bidding data Technometrics 23 4 1981 351 361 (Pubitemid 12456651)
18. S.F. Crone, M. Hibon, and K. Nikolopoulos Advances in forecasting with neural networks? Empirical evidence from the nn3 competition on time series prediction International Journal of Forecasting 27 3 2011 635 660
19. Jonathan D. Cryer, and Kung-Sik Chan Time Series Analysis With Applications in R 2008 Springer ISBN 978-0-387-75958-6
20. C.G. da Silva Time series forecasting with a non-linear model and the scatter search meta-heuristic Information Sciences 178 16 2008 3288 3299
21. J.G. De Gooijer, and R.J. Hyndman 25 Years of time series forecasting International Journal of Forecasting 22 3 2006 443 473 (Pubitemid 44057710)
22. G. Deco, R. Neuneier, and B. Schürmann Non-parametric data selection for neural learning in non-stationary time series Neural Networks 10 3 1997 401 407 (Pubitemid 27241130)
23. B. Efron, T. Hastie, I. Johnstone, R. Tibshirani, H. Ishwaran, K. Knight, J.-M. Loubes, P. Massart, D. Madigan, G. Ridgeway, S. Rosset, J.I. Zhu, R.A. Stine, B.A. Turlach, S. Weisberg, T. Hastie, I. Johnstone, and R. Tibshirani Least angle regression Annals of Statistics 32 2 2004 407 499 (Pubitemid 41250302)
24. M. Gan, H. Peng, X. Peng, X. Chen, and G. Inoussa A locally linear rbf network-based state-dependent ar model for nonlinear time series modeling Information Sciences 180 22 2010 4370 4383
25. S. García, A. Fernández, J. Luengo, and F. Herrera Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: experimental analysis of power Information Sciences 180 10 2010 2044 2064
26. P. Goodwin, and R. Lawton On the asymmetry of the symmetric mape International Journal of Forecasting 15 4 1999 405 408
27. C. Hamzaebi Improving artificial neural networks' performance in seasonal time series forecasting Information Sciences 178 23 2008 4550 4559
28. Jeffrey D. Hart Automated kernel smoothing of dependent data by using time series cross- validation Journal of the Royal Statistical Society: Series B (Methodological) 56 3 1994 529 542
29. J.S.U. Hjorth Computer Intensive Statistical Methods Validation, Model Selection and Bootstrap 1994 Chapman and Hall
30. Urban Hjorth Model selection and forward validation Scandinavian Journal of Statistics 9 2 1982 95 105
31. R.J. Hyndman, and A.B. Koehler Another look at measures of forecast accuracy International Journal of Forecasting 22 4 2006 679 688 (Pubitemid 44524717)
32. A. Inoue, and L. Kilian On the selection of forecasting models Journal of Econometrics 130 2 2006 273 306 (Pubitemid 43144865)
33. T.Y. Kim, K.J. Oh, C. Kim, and J.D. Do Artificial neural networks for non-stationary time series Neurocomputing 61 1-4 2004 439 447 (Pubitemid 39618391)
34. R. Kunst Cross validation of prediction models for seasonal time series by parametric bootstrapping Austrian Journal of Statistics 37 2008 271 284
35. Robert M. Kunst, and Adusei Jumah Toward a theory of evaluating predictive accuracy Economics Series vol. 162 2004 Institute for Advanced Studies
36. S. Makridakis, and M. Hibon The m3-competition: results, conclusions and implications International Journal of Forecasting 16 4 2000 451 476
37. Allan D.R. McQuarrie, and Chih-Ling Tsai Regression and Time Series Model Selection 1998 World Scientific Publishing
38. J. Opsomer, Y. Wang, and Y. Yang Nonparametric regression with correlated errors Statistical Science 16 2 2001 134 153 (Pubitemid 33636345)
39. N.M. Pindoriya, S.N. Singh, and S.K. Singh An adaptive wavelet neural network-based energy price forecasting in electricity markets IEEE Transactions on Power Systems 23 3 2008 1423 1432
40. R Development Core Team, R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2009, ISBN 3-900051-07-0.
41. J. Racine Consistent cross-validatory model-selection for dependent data: hv-block cross-validation Journal of Econometrics 99 1 2000 39 61
42. S.E. Said, and D.A. Dickey Testing for unit roots in autoregressive- moving average models of unknown order Biometrika 71 1984 599 607
43. J. Shao An asymptotic theory for linear model selection Statistica Sinica 7 1997 221 264 (Pubitemid 127473663)
44. Jun Shao Linear model selection by cross-validation Journal of the American Statistical Association 88 422 1993 486 494
45. T.A.B. Snijders, On cross-validation for predictor evaluation in time series, in: T.K. Dijkstra (Ed.), On Model Uncertainty and its Statistical Implications, 1988, pp. 56-69.
46. M. Stone Cross-validatory choice and assessment of statistical predictions Journal of the Royal Statistical Society: Series B (Methodological) 36 2 1974 111 147
47. L.J. Tashman Out-of-sample tests of forecasting accuracy: an analysis and review International Journal of Forecasting 16 4 2000 437 450
48. H. Tong Non-Linear Time Series: A Dynamical System Approach 1990 Clarendon Press Oxford
49. W.N. Venables, and B.D. Ripley Modern Applied Statistics with S fourth ed. 2002 Springer New York ISBN 0-387-95457-0
50. E.-J. Wagenmakers, P. Grünwald, and M. Steyvers Accumulative prediction error and the selection of time series models Journal of Mathematical Psychology 50 2 2006 149 166 (Pubitemid 43414940)
51. A.S Weigend, N.A. Gershenfeld, (Eds.), Time Series Prediction: Forecasting the Future and Understanding the Past, 1994.
52. G.P. Zhang A neural network ensemble method with jittered training data for time series forecasting Information Sciences 177 23 2007 5329 5346 (Pubitemid 47409984)
53. S.T. Ziliak, and D.N. McCloskey Size matters: the standard error of regressions in the american economic review Journal of Socio-Economics 33 5 2004 527 546 (Pubitemid 39615256)