How do I know if my forecasts are better? Using benchmarks in hydrological ensemble prediction

Journal of Hydrology

Volumn 522, Issue , 2015, Pages 697-713

  Pappenberger, F ^a,e,f   Ramos, M H ^b   Cloke, H L ^d   Wetterhall, F ^a   Alfieri, L ^a,c   Bogner, K ^a   Mueller, A ^a,d   Salamon, P ^c  

^a EUROPEAN CENTRE FOR MEDIUM RANGE WEATHER FORECASTS   (United Kingdom)

^b Procédés biotechnologiques au service de l'environnement   (France)

^c JOINT RESEARCH CENTRE   (Italy)

^d UNIVERSITY OF READING   (United Kingdom)

^e UNIVERSITY OF BRISTOL   (United Kingdom)

^f HOHAI UNIVERSITY   (China)

Author keywords

Benchmark; Evaluation; Forecast performance; Hydrological ensemble prediction; Probabilistic forecasts; Verification

Indexed keywords

BENCHMARKING; CATCHMENTS; FORECASTING; VERIFICATION;

CONTINUOUS RANKED PROBABILITY SCORES; ENSEMBLE PREDICTION; ENSEMBLE PREDICTION SYSTEMS; EVALUATION; FORECAST PERFORMANCE; HYDROLOGICAL FORECASTING; METEOROLOGICAL OBSERVATION; PROBABILISTIC FORECASTS;

WEATHER FORECASTING;

BENCHMARKING; CATCHMENT; CLIMATOLOGY; DISCRIMINANT ANALYSIS; ENSEMBLE FORECASTING; HYDROLOGICAL MODELING; METEOROLOGY; OBSERVATIONAL METHOD; PRECIPITATION (CLIMATOLOGY); PREDICTION;

EID: 84922567620     PISSN: 00221694     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jhydrol.2015.01.024     Document Type: Article

References (73)

1. Alfieri L., Salamon P., Bianchi A., Neal J., Bates P., Feyen L. Advances in pan-European flood hazard mapping. Hydrol. Process. 2013, 10.1002/hyp.9947.
2. Alfieri L., Pappenberger F., Wetterhall F., Haiden T., Richardson D., Salamon P. Evaluation of ensemble streamflow predictions in Europe. J. Hydrol. 2014, 517:913-922. 10.1016/j.jhydrol.2014.06.035.
3. Baruth, B., Genovese, G., Leo, O., Boogard, H., te Roller, J.A., van Diepen, K., 2007. CGMS version 9.2 User Manual and Technical Documentation, JRC Scientific and Technical Report, ISBN 978-92-79-06995-6, OPOCE, Luxembourg.
4. Berthet L., Andréassian V., Perrin C., Javelle P. How crucial is it to account for the antecedent moisture conditions in flood forecasting? Comparison of event-based and continuous approaches on 178 catchments. Hydrol. Earth Syst. Sci. 2009, 13(6):819-831.
5. Bogner K., Pappenberger F. Multiscale error analysis, correction, and predictive uncertainty estimation in a flood forecasting system. Water Resour. Res. 2010, 47(7):W07524. 10.1029/2010WR009137.
6. Bogner K., Pappenberger F. Multiscale error analysis, correction, and predictive uncertainty estimation in a flood forecasting system. Water Resour. Res. 2011, 47.
7. Brown, J., 2013. Comments to HEPEX Science and Challenges: Verification of Ensemble Forecasts. http://hepex.irstea.fr/hepex-science-and-challenges-verification-of-ensemble-forecasts-44/.
8. Brown J.D., Seo D.-J. A nonparametric postprocessor for bias correction of hydrometeorological and hydrologic ensemble forecasts. J. Hydrometeorol. 2010, 11(3):642-665.
9. Brown J.D., Demargne J., Seo D.-J., Liu Y. The ensemble verification system (EVS): a software tool for verifying ensemble forecasts of hydrometeorological and hydrologic variables at discrete locations. Environ. Model. Softw. 2010, 25(7):854-872.
10. Cloke H.L., Pappenberger F. Evaluating forecasts of extreme events for hydrological applications: an approach for screening unfamiliar performance measures. Meterol. Appl. 2008, 15(1):181-197.
11. Cloke H.L., Pappenberger F. Ensemble flood forecasting: a review. J. Hydrol. 2009, 375(3-4):613-626.
12. Cloke H.L., Pappenberger F., van Andel S.J., Schaake J., Thielen J., Ramos M.-H. Hydrological ensemble prediction systems preface. Hydrol. Process. 2013, 27(1):1-4.
13. Cloke H.L., Wetterhall F., He Y., Freer J.E., Pappenberger F. Modelling climate impact on floods with ensemble climate projections. Q. J. R. Meteorol. Soc. 2013, 139(671):282-297.
14. Dawson C.W., Abrahart R.J., See L.M. HydroTest: a web-based toolbox of evaluation metrics for the standardised assessment of hydrological forecasts. Environ. Modell. Softw. 2007, 22:1034-1052.
15. Demargne, J., Brown, J., 2013. HEPEX Science and Challenges: Verification of Ensemble Forecasts.. http://hepex.irstea.fr/hepex-science-and-challenges-verification-of-ensemble-forecasts/.
16. Demargne J., Mullusky M., Werner K., Adams T., Lindsey S., Schwein N., Marosi W., Welles E. Application of forecast verification science to operational river forecasting in the U.S. national weather service. Bull. Am. Meteorol. Soc. 2009, 90(6):779-784.
17. Demargne J., Wu L., Regonda S.K., Brown J.D., Lee H., He M., Seo D.-J., Hartman R., Herr H.D., Fresch M., Schaake J., Zhu Y. The science of NOAA's operational hydrologic ensemble forecast service. Bull. Amer. Meteor. Soc. 2014, 95:79-98.
18. Demeritt D., Nobert S., Cloke H.L., Pappenberger F. The European flood alert system and the communication, perception, and use of ensemble predictions for operational flood risk management. Hydrol. Process. 2013, 27(1):147-157.
19. Ebert E., Wilson L., Weigel A., Mittermaier M., Nurmi P., Gill P., Göber M., Joslyn S., Brown B., Fowler T., Watkins A. Progress and challenges in forecast verification. Met. Apps. 2013, 20:130-139. 10.1002/met.1392.
20. Ewen J. Hydrograph matching method for measuring model performance. J. Hydrol. 2011, 408(1-2):178-187.
21. Fewtrell T.J., Duncan A., Sampson C.C., Neal J.C., Bates P.D. Benchmarking urban flood models of varying complexity and scale using high resolution terrestrial LiDAR data. Phys. Chem. Earth 2011, 36(7-8):281-291.
22. Fundel F., Jörg-Hess S., Zappa M. Long-range hydrometeorological ensemble predictions of drought parameters. Hydrol. Earth Syst. Sci. Discuss. 2012, 9:6857-6887.
23. Fundel F., Jörg-Hess S., Zappa M. Monthly hydrometeorological ensemble prediction of streamflow droughts and corresponding drought indices. Hydrol. Earth Syst. Sci. 2013, 17(1):395-407.
24. Garrick M., Cunnane C., Nash J.E. Criterion of efficiency for rainfall-runoff models. J. Hydrol. 1978, 36(3-4):375-381.
25. Gneiting T., Ranjan R. Comparing density forecasts using threshold- and quantile-weighted scoring rules. J. Business Econ. Statist. 2011, 29(3):411-422.
26. Gordon N., Shaykewich J., World Meteorological Organization Guidelines on Performance Assessment of Public Weather Services 2000, WMO.
27. Hamill T.M., Juras J. Measuring forecast skill: is it real skill or is it the varying climatology?. Q. J. R. Meteorol. Soc. 2006, 132(621C):2905-2923.
28. Hersbach H. Decomposition of the continuous ranked probability score for ensemble prediction systems. Weather Forecasting 2000, 15(5):559-570.
29. Hopson T.M., Webster P.J. A 1-10-day ensemble forecasting scheme for the major river basins of Bangladesh: forecasting severe floods of 2003-07. J. Hydrometeor 2010, 11:618-641. 10.1175/2009JHM1006.1.
30. Jaun S., Ahrens B. Evaluation of a probabilistic hydrometeorological forecast system. Hydrol. Earth Syst. Sci. 2009, 13(7):1031-1043.
31. Forecast Verification: A Practioner's Guide in Atmospheric Science 2011, Wiley. J.T. Jolliffe, D.B. Stephenson (Eds.).
32. Kachroo R.K. River flow forecasting: 1 a discussion of the principles. J. Hydrol. 1992, 133(1-2):1-15.
33. Kiese S., Pappenberger A., Friess W., Mahler H.C. Equilibrium studies of protein aggregates and homogeneous nucleation in protein formulation. J. Pharm. Sci. 2010, 99(2):632-644.
34. Krause P., Boyle D.P., Bäse F. Comparison of different efficiency criteria for hydrological model assessment. Adv. Geosci. 2005, 5:89-97.
35. Lerch S. Verification of probabilistic forecasts for rare and extreme events 2012, Ruprecht-Karls-Universitaet Heidelberg, Heidelberg.
36. Lerch S., Thorarinsdottir T.L. Comparison of nonhomogeneous regression models for probabilistic wind speed forecasting. Tellus A 2013, 65:21206.
37. Luo Y.Q., Randerson J.T., Abramowitz G., Bacour C., Blyth E., Carvalhais N., Ciais P., Dalmonech D., Fisher J.B., Fisher R., Friedlingstein P., Hibbard K., Hoffman F., Huntzinger D., Jones C.D., Koven C., Lawrence D., Li D.J., Mahecha M., Niu S.L., Norby R., Piao S.L., Qi X., Peylin P., Prentice I.C., Riley W., Reichstein M., Schwalm C., Wang Y.P., Xia J.Y., Zaehle S., Zhou X.H. A framework for benchmarking land models. Biogeosciences 2012, 9(10):3857-3874.
38. Martinec J., Rango A. Merits of statistical criteria for the performance of hydrological models. Water Resour. Bull. 1989, 25(2):421-432.
39. Marty R., Zin I., Obled C., Bontron G., Djerboua A. Toward real-time daily PQPF by an analogue sorting approach: application to flash-flood catchments. J. Appl. Meteorol. Clim. 2012, 51(3):505-520.
40. Murphy A.H., Winkler R.L. A general framework for forecasts verification. Mon. Weather Rev. 1987, 115(7):1330-1338.
41. Nicolle P., Pushpalatha R., Perrin C., François D., Thiéry D., Mathevet T., Le Lay M., Besson F., Soubeyroux J.M., Viel C., Regimbeau F., Andréassian V., Maugis P., Augeard B., Morice E. Benchmarking hydrological models for low-flow simulation and forecasting on French catchments. Hydrol. Earth Syst. Sci. Discuss. 2013, 10(11):13979-14040.
42. Obled C., Bontron G., Garçon R. Quantitative precipitation forecasts: a statistical adaptation of model outputs through an analogues sorting approach. Atmos. Res. 2002, 63:303-324.
43. Pagano T.C. Evaluation of mekong river commission operational flood forecasts, 2000-2012. Hydrol. Earth Syst. Sci. Discuss. 2013, 10(11):14433-14461.
44. Pagano T.C., Shrestha D.L., Wang Q.J., Robertson D.E., Hapuarachchi P. Ensemble dressing for hydrological applications. Hydrol. Process. 2013, 27(1):106-116.
45. Pappenberger F., Beven K. Functional classification and evaluation of hydrographs based on multicomponent mapping. J. River Basin Manage. 2004, 2(2):89-100.
46. Pappenberger F., Buizza R. The skill of ECMWF precipitation and temperature predictions in the Danube basin as forcings of hydrological models. Weather Forecasting 2009, 24(3):749-766.
47. Pappenberger F., Bartholmes J., Thielen J., Cloke H.L., Buizza R., de Roo A. New dimensions in early flood warning across the globe using grand-ensemble weather predictions. Geophys. Res. Lett. 2008, 35(10).
48. Pappenberger F., Bogner K., Wetterhall F., He Y., Cloke H.L., Thielen J. Forecast convergence score: a forecaster's approach to analysing hydro-meteorological forecast systems. Adv. Geosci. 2011, 29:27-32.
49. Pappenberger F., Thielen J., Del Medico M. The impact of weather forecast improvements on large scale hydrology: analysing a decade of forecasts of the European flood alert system. Hydrol. Process. 2011, 25(7):1091-1113.
50. Pappenberger F., Wetterhall F., Dutra E., Di Giuseppe F., Bogner K., Alfieri L., Cloke H.L. Seamless forecasting of extreme events on a global scale. Climate and Land Surface Changes in Hydrology 2013, 3-10. IAHS Publication. E. Boegh (Ed.).
51. Perrin C., Andreassian V., Michel C. Simple benchmark models as a basis for model efficiency criteria. Arch. Fuer Hydrobiol. Suppl. 2006, 161(1-2):221-244.
52. Pushpalatha R., Perrin C., Le Moine N., Andreassian V. A review of efficiency criteria suitable for evaluating low-flow simulations. J. Hydrol. 2012, 420:171-182.
53. Radanovics S., Vidal J.P., Sauquet E., Ben Daoud A., Bontron G. Optimising predictor domains for spatially coherent precipitation downscaling. Hydrol. Earth Syst. Sci. Discuss. 2013, 10(4):4015-4061.
54. Ramos M.-H., Bartholmes J., Thielen-del Pozo J. Development of decision support products based on ensemble forecasts in the European flood alert system. Atmos. Sci. Lett. 2007, 8(4):113-119.
55. Ramos M.H., Mathevet T., Thielen J., Pappenberger F. Communicating uncertainty in hydro-meteorological forecasts: mission impossible?. Meteorol. Appl. 2010, 17(2):223-2258.
56. Ramos M.H., van Andel S.J., Pappenberger F. Do probabilistic forecasts lead to better decisions?. Hydrol. Earth Syst. Sci. 2013, 17:2219-2232. 10.5194/hess-17-2219-2013.
57. Randrianasolo A., Ramos M.H., Thirel G., Andréassian V., Martin E. Comparing the scores of hydrological ensemble forecasts issued by two different hydrological models. Atmos. Sci. Lett. 2010, 11(2):100-107.
58. Ritter A., Munoz-Carpena R. Performance evaluation of hydrological models: statistical significance for reducing subjectivity in goodness-of-fit assessments. J. Hydrol. 2013, 480:33-45.
59. Rodwell M.J., Richardson D.S., Hewson T.D., Haiden T. A new equitable score suitable for verifying precipitation in numerical weather prediction. Q. J. R. Meteorol. Soc. 2010, 136(650):1344-1363.
60. Romanowicz R.J., Young P.C., Beven K.J., Pappenberger F. A data based mechanistic approach to nonlinear flood routing and adaptive flood level forecasting. Adv. Water Resour. 2008, 31(8):1048-1056.
61. Rykiel E.J. Testing ecological models: the meaning of validation. Ecol. Model. 1996, 90(3):229-244.
62. Schaefli B., Gupta H.V. Do Nash values have value?. Hydrol. Process. 2007, 21(15):2075-2080.
63. Seibert J. On the need for benchmarks in hydrological modelling. Hydrol. Process. 2001, 15(6):1063-1064.
64. Stedinger J.R., Kim Y.-O. Probabilities for ensemble forecasts reflecting climate information. J. Hydrol. 2010, 391:9-23.
65. Thielen J., Bartholmes J., Ramos M.H., de Roo A. The European flood alert system - Part 1: concept and development. Hydrol. Earth Syst. Sci. 2009, 13(2):125-140.
66. Thielen J., Bogner K., Pappenberger F., Kalas M., del Medico M., de Roo A. Monthly-, medium-, and short-range flood warning: testing the limits of predictability. Meteorol. Appl. 2009, 16(1):77-90.
67. Trinh B.N., Thielen-del Pozo J., Thirel G. The reduction continuous rank probability score for evaluating discharge forecasts from hydrological ensemble prediction systems. Atmos. Sci. Lett. 2013, 14(2):61-65.
68. van der Knijff J.M., Younis J., de Roo A.P.J. LISFLOOD: a GIS-based distributed model for river basin scale water balance and flood simulation. Int. J. Geogr. Inf. Sci. 2010, 24:189-212.
69. Węglarczyk S. The interdependence and applicability of some statistical quality measures for hydrological models. J. Hydrol. 1998, 206(1-2):98-103.
70. Weijs S.V., van de Giesen N. An information-theoretical perspective on weighted ensemble forecasts. J. Hydrol. 2013, 498:177-190.
71. Weijs S.V., Schoups G., van de Giesen N. Why hydrological predictions should be evaluated using information theory. Hydrol. Earth Syst. Sci. 2010, 14:2545-2558. 10.5194/hess-14-2545-2010.
72. Wetterhall F., Pappenberger F., Cloke H.L., Thielen-del Pozo J., Balabanova S., Daňhelka J., Vogelbacher A., Salamon P., Carrasco I., Cabrera-Tordera A.J., Corzo-Toscano M., Garcia-Padilla M., Garcia-Sanchez R.J., Ardilouze C., Jurela S., Terek B., Csik A., Casey J., Stankūnavičius G., Ceres V., Sprokkereef E., Stam J., Anghel E., Vladikovic D., Alionte Eklund C., Hjerdt N., Djerv H., Holmberg F., Nilsson J., Nyström K., Sušnik M., Hazlinger M., Holubecka M. Forecasters priorities for improving probabilistic flood forecasts. Hydrol. Earth Syst. Sci. Discuss. 2013, 10(2):2215-2242.
73. Zalachori I., Ramos M.H., Garçon R., Mathevet T., Gailhard J. Statistical processing of forecasts for hydrological ensemble prediction: a comparative study of different bias correction strategies. Adv. Sci. Res. 2012, 8:135-141.