The importance of prewhitening in change point analysis under persistence

Stochastic Environmental Research and Risk Assessment

Volumn 30, Issue 2, 2016, Pages 763-777

  Serinaldi, Francesco ^a,b   Kilsby, Chris G ^a,b  

^a NEWCASTLE UNIVERSITY   (United Kingdom)

^b Towers Watson   (United Kingdom)

Author keywords

Autoregressive process; Change point analysis; Fractional Gaussian noise; Hurst parameter; Pettitt test; Prewhitening

Indexed keywords

MONTE CARLO METHODS;

AUTO REGRESSIVE PROCESS; CHANGE-POINT ANALYSIS; FRACTIONAL GAUSSIAN NOISE; HURST PARAMETER; PETTITT TEST; PREWHITENING;

GAUSSIAN NOISE (ELECTRONIC);

CORRELATION; GAUSSIAN METHOD; HYDROMETEOROLOGY; MONTE CARLO ANALYSIS; PARAMETERIZATION; REGRESSION ANALYSIS;

EID: 84958111949     PISSN: 14363240     EISSN: 14363259     Source Type: Journal    
DOI: 10.1007/s00477-015-1041-5     Document Type: Article

References (66)

1. Basseville M, Nikiforov IV (1993) Detection of abrupt changes: theory and application. Prentice Hall, New Jersey
2. Bayazit M, Önöz B (2004) Comment on “Applicability of prewhitening to eliminate the influence of serial correlation on the Mann–Kendall test” by Sheng Yue and Chun Yuan Wang. Water Resour Res 40(8):W08801
3. Bayazit M, Önöz B (2007) To prewhiten or not to prewhiten in trend analysis? Hydrol Sci J 52(4):611–624
4. Bayazit M, Önöz B (2008) Reply to discussion of “To prewhiten or not to prewhiten in trend analysis?”. Hydrol Sci J 53(3):669–669
5. Bayley GV, Hammersley JM (1946) The “effective” number of independent observations in an autocorrelated time series. Suppl J R Stat Soc 8(2):184–197
6. Beran J, Feng Y, Ghosh S, Kulik R (2013) Long-memory processes: probabilistic properties and statistical methods. Springer, New York
7. Bronaugh D, Werner A (2013) zyp: Zhang + Yue-Pilon trends package. http://www.CRAN.R-project.org/package=zyp,rpackageversion0.10-1
8. Busuioc A, von Storch H (1996) Changes in the winter precipitation in Romania and its relation to the large-scale circulation. Tellus A 48(4):538–552
9. Cochrane D, Orcutt GH (1949) Application of least squares regression to relationships containing auto-correlated error terms. J Am Stat Assoc 44(245):32–61
10. Constantine W, Percival D (2014) Fractal: fractal time series modeling and analysis. http://www.CRAN.R-project.org/package=fractal,rpackageversion2.0-0
11. Ferguson CR, Villarini G (2012) Detecting inhomogeneities in the Twentieth Century Reanalysis over the central United States. J Geophys Res Atmos 117(D5):D05–123
12. Fraley C (2012) fracdiff: fractionally differenced ARIMA aka ARFIMA(p, d, q) models. URL http://www.CRAN.R-project.org/package=fracdiff,rpackageversion1.4--2 (S original by Chris Fraley and U. Washington and Seattle. R port by Fritz Leisch at TU Wien; since 2003–2012: Martin Maechler; fdGPH and fdSperio and etc by Valderio Reisen and Artur Lemonte.)
13. Guerreiro SB, Kilsby CG, Serinaldi F (2014) Analysis of time variation of rainfall in transnational basins in Iberia: abrupt changes or trends? Int J Climatol 34(1):114–133
14. Gurevich G (2009) Asymptotic distribution of Mann–Whitney type statistics for nonparametric change point problems. Comput Model New Technol 13(3):18–26
15. Gurevich G, Raz B (2010) Monte Carlo analysis of change point estimators. J Appl Quant Methods 5(4):659–669
16. Hamed KH (2008a) To prewhiten or not to prewhiten in trend analysis? Hydrol Sci J 53(3):667–668
17. Hamed KH (2008b) Trend detection in hydrologic data: the Mann–Kendall trend test under the scaling hypothesis. J Hydrol 349(3–4):350–363
18. Hamed KH (2009) Enhancing the effectiveness of prewhitening in trend analysis of hydrologic data. J Hydrol 368(1–4):143–155
19. Hamed KH, Rao AR (1998) A modified Mann–Kendall trend test for autocorrelated data. J Hydrol 204(1–4):182–196
20. Hawkins DM (1977) Testing a sequence of observations for a shift in location. J Am Stat Assoc 72(357):180–186
21. Kang W, Shin DW, Lee Y (2003) Biases of the restricted maximum likelihood estimators for ARMA processes with polynomial time trend. J Stat Plan Inference 116(1):163–176
22. Katz RW (1988) Statistical procedures for making inferences about climate variability. J Clim 1(11):1057–1064
23. Khaliq MN, Ouarda TBMJ, Gachon P, Sushama L, St-Hilaire A (2009) Identification of hydrological trends in the presence of serial and cross correlations: a review of selected methods and their application to annual flow regimes of Canadian rivers. J Hydrol 368(1–4):117–130
24. Koutsoyiannis D (2003) Climate change, the Hurst phenomenon, and hydrological statistics. Hydrol Sci J 48(1):3–24
25. Koutsoyiannis D, Montanari A (2007) Statistical analysis of hydroclimatic time series: uncertainty and insights. Water Res Res 43(5):W05–429
26. Kulkarni A, von Storch H (1995) Monte Carlo experiments on the effect of serial correlation on the Mann–Kendall test of trend. Meteorol Z 4(2):82–85
27. Kundzewicz ZW, Robson AJ (2004) Change detection in hydrological records-a review of the methodology. Hydrol Sci J 49(1):7–19
28. Lenton RL, Schaake JC (1973) Comments on ‘Small sample estimation of (Formula presented.) ′ by James R. Wallis and P. Enda O’Connell. Water Resour Res 9(2):503–504
29. Mallakpour I, Villarini G (2015) A simulation study to examine the sensitivity of the Pettitt test to detect abrupt changes in mean. Hydrol Sci J. doi:10.1080/02626667.2015.1008482
30. Marriott FHC, Pope JA (1954) Bias in the estimation of autocorrelations. Biometrika 41(3–4):390–402
31. Matalas NC, Sankarasubramanian A (2003) Effect of persistence on trend detection via regression. Water Resour Res 39(12):1342
32. McLeod AI, Hipel KW (1978) Preservation of the rescaled adjusted range: 1. A reassessment of the Hurst Phenomenon. Water Resour Res 14(3):491–508
33. McLeod AI, Veenstra J (2012) FGN: fractional Gaussian noise, estimation and simulation. http://www.CRAN.R-project.org/package=FGN,rpackageversion2.0
34. McLeod AI, Yu H, Krougly ZL (2007) Algorithms for linear time series analysis: with R Package. J Stat Softw 23(5):1–26
35. Mudelsee M (2001) Note on the bias in the estimation of the serial correlation coefficient of AR(1) processes. Stat Pap 42(4):517–527
36. Önöz B, Bayazit M (2003) The power of statistical tests for trend detection. Turk J Eng and Environ Sci 27(4):247–251
37. Orcutt GH (1948) A study of the autoregressive nature of the time series used for Tinbergen’s model of the economic system of the United States, 1919–1932. J R Stat Soc Ser B 10(1):1–53
38. Pettitt AN (1979) A non-parametric approach to the change-point problem. J R Stat Soc Ser C 28(2):126–135
39. R Development Core Team (2014) R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna. http://www.R-project.org/, ISBN3-900051-07-0
40. Rodionov SN (2004) A sequential algorithm for testing climate regime shifts. Geophys Res Lett 31(9):L09204
41. Rodionov SN (2006) Use of prewhitening in climate regime shift detection. Geophys Res Lett 33(12):L12707
42. Rougé C, Ge Y, Cai X (2013) Detecting gradual and abrupt changes in hydrological records. Adv Water Resour 53:33–44
43. Rybski D, Neumann J (2011) A review on the Pettitt test. In: Kropp J, Schellnhuber HJ (eds) In extremis. Springer, Dordrecht, pp 202–213
44. Sagarika S, Kalra A, Ahmad S (2014) Evaluating the effect of persistence on long-term trends and analyzing step changes in streamflows of the continental United States. J Hydrol 517:36–53
45. Sen PK (1968) Estimates of the regression coefficient based on Kendall’s tau. J Am Stat Assoc 63(324):1379–1389
46. Serinaldi F (2010) Use and misuse of some Hurst parameter estimators applied to stationary and non-stationary financial time series. Phys A Stat Mech Appl 389(14):2770–2781
47. Shumway RH, Stoffer DS (2011) Time series analysis and its applications: with R examples, 3rd edn. Springer, New York
48. Tramblay Y, El Adlouni S, Servat E (2013) Trends and variability in extreme precipitation indices over Maghreb countries. Nat Hazards Earth Syst Sci 13(12):3235–3248
49. Tyralis H, Koutsoyiannis D (2011) Simultaneous estimation of the parameters of the Hurst–Kolmogorov stochastic process. Stoch Environ Res Risk Assess 25(1):21–33
50. van Giersbergen NPA (2005) On the effect of deterministic terms on the bias in stable AR models. Econ Lett 89(1):75–82
51. Villarini G, Serinaldi F, Smith JA, Krajewski WF (2009) On the stationarity of annual flood peaks in the continental United States during the 20th century. Water Resour Res 45(8):W08417
52. Villarini G, Smith JA, Serinaldi F, Ntelekos AA (2011) Analyses of seasonal and annual maximum daily discharge records for central Europe. J Hydrol 399(3–4):299–312
53. von Storch H (1999) Misuses of statistical analysis in climate research. In: von Storch H, Navarra A (eds) Analysis of climate variability. Springer, Dordrecht, pp 11–26
54. Wallis JR, O’Connell PE (1972) Small sample estimation of (Formula presented.). Water Resour Res 8(3):707–712
55. Wang XL, Swail VR (2001) Changes of extreme wave heights in northern hemisphere oceans and related atmospheric circulation regimes. J Clim 14(10):2204–2221
56. Wuertz D, many others, see the SOURCE file (2013) fArma: ARMA Time Series Modelling. http://www.CRAN.R-project.org/package=fArma, rpackageversion3010.79
57. Xie H, Li D, Xiong L (2014) Exploring the ability of the Pettitt method for detecting change point by Monte Carlo simulation. Stoch Environ Res Risk Assess 28(7):1643–1655
58. Yue S, Pilon P (2004) A comparison of the power of the t test, Mann–Kendall and bootstrap tests for trend detection. Hydrol Sci J 49(1):21–37
59. Yue S, Wang C (2002) Applicability of prewhitening to eliminate the influence of serial correlation on the Mann–Kendall test. Water Resour Res 38(6):41–47
60. Yue S, Wang C (2004a) Reply to comment by Mehmetcik Bayazit and Bihrat Önöz on “Applicability of prewhitening to eliminate the influence of serial correlation on the Mann–Kendall test”. Water Resour Res 40(8):W08802
61. Yue S (2004b) Reply to comment by Xuebin Zhang and Francis W. Zwiers on “Applicability of prewhitening to eliminate the influence of serial correlation on the Mann–Kendall test”. Water Resour Res 40(3):W03806
62. Yue S, Wang CY (2004c) The Mann–Kendall test modified by effective sample size to detect trend in serially correlated hydrological series. Water Resour Manag 18(3):201–218
63. Yue S, Pilon P, Cavadias G (2002a) Power of the Mann–Kendall and Spearman’s rho tests for detecting monotonic trends in hydrological series. J Hydrol 259(1–4):254–271
64. Yue S, Pilon P, Phinney B, Cavadias G (2002b) The influence of autocorrelation on the ability to detect trend in hydrological series. Hydrol Process 16(9):1807–1829
65. Zhang X, Zwiers FW (2004) Comment on “Applicability of prewhitening to eliminate the influence of serial correlation on the Mann–Kendall test” by Sheng Yue and Chun Yuan Wang. Water Resour Res 40(3):W03805
66. Zhang X, Vincent LA, Hogg WD, Niitsoo A (2000) Temperature and precipitation trends in Canada during the 20th century. Atmos Ocean 38(3):395–429