The Ensemble Verification System (EVS): A software tool for verifying ensemble forecasts of hydrometeorological and hydrologic variables at discrete locations

Environmental Modelling and Software

Volumn 25, Issue 7, 2010, Pages 854-872

  Brown, James D ^a,b   Demargne, Julie ^a,b   Seo, Dong Jun ^a,b   Liu, Yuqiong ^a,c  

^a NOAA EARTH SYSTEM RESEARCH LABORATORY   (United States)

^b UNIVERSITY CORPORATION FOR ATMOSPHERIC RESEARCH   (United States)

^c RIVERSIDE TECHNOLOGY INC   (United States)

Author keywords

Ensemble forecasting; Forecast quality; Java; Probability forecasts; Uncertainty; Verification

Indexed keywords

DISCRETE LOCATION; ENSEMBLE FORECASTING; ENSEMBLE FORECASTS; FORECAST PROBABILITIES; FORECAST QUALITY; LEADTIME; NATIONAL WEATHER SERVICES; PREDICTIVE ERROR; PREPROCESSORS; PROBABILITY FORECASTS; SOFTWARE TOOL; SPACE AND TIME; STREAMFLOW PREDICTION; VERIFICATION SYSTEMS;

APPLICATION PROGRAMMING INTERFACES (API); COMPUTER SOFTWARE SELECTION AND EVALUATION; ERRORS; GRAPHICAL USER INTERFACES; PROBABILITY; STREAM FLOW; SYSTEMATIC ERRORS; VERIFICATION; WATER;

WEATHER FORECASTING;

ENSEMBLE FORECASTING; HYDROLOGY; HYDROMETEOROLOGY; PRECIPITATION (CLIMATOLOGY); PROBABILITY; SOFTWARE; STREAMFLOW; UNCERTAINTY ANALYSIS; WEATHER FORECASTING;

EID: 77949571388     PISSN: 13648152     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.envsoft.2010.01.009     Document Type: Article

References (70)

1. Anderson J.L. A method for producing and evaluating probabilistic forecasts from ensemble model integrations. Journal of Climate 9 (1996) 1518-1530
2. Anderson T.W. On the distribution of the two-sample Cramer-von Mises criterion. The Annals of Mathematical Statistics 33 3 (1962) 1148-1159
3. Araújo M.B., and New M. Ensemble forecasting of species distributions. Trends in Ecology and Evolution 22 (2007) 42-47
4. Bonadonna C., Connor C.B., Houghton B.F., Connor L., Byrne M., Laing A., and Hincks T. Probabilistic modeling of tephra dispersion: hazard assessment of a multi-phase eruption at Tarawera, New Zealand. Journal of Geophysical Research 110 B3 (2005) B03203
5. Bradley A.A., Hashino T., and Schwartz S.S. Distributions-oriented verification of probability forecasts for small data samples. Weather and Forecasting 18 (2003) 903-917
6. Bradley A.A., Schwartz S.S., and Hashino T. Distributions-oriented verification of ensemble streamflow predictions. Journal of Hydrometeorology 5 3 (2004) 532-545
7. Bradley A.A., Schwartz S.S., and Hashino T. Sampling uncertainty and confidence intervals for the brier score and brier skill score. Weather and Forecasting 23 5 (2008) 992-1006
8. Brier G.W. Verification of forecasts expressed in terms of probability. Monthly Weather Review 78 (1950) 1-3
9. Bröcker J., and Smith L.A. Increasing the reliability of reliability diagrams. Weather and Forecasting 22 3 (2007) 651-661
10. Bröcker J., and Smith L.A. Scoring probabilistic forecasts: on the importance of being proper. Weather and Forecasting 22 2 (2007) 382-388
11. Bröcker J. On reliability analysis of multi-categorical forecasts. Nonlinear Processes in Geophysics 15 4 (2008) 661-673
12. Brooks H.E., and Doswell C.A. A comparison between measures-oriented and distributions-oriented verification methods in forecast verification. Weather and Forecasting 11 (1996) 288-303
13. Brown J.D., and Heuvelink G. Assessing uncertainty propagation through physically based models of soil water flow and solute transport. In: Anderson M. (Ed). The Encyclopedia of Hydrological Sciences (2005), John Wiley and Sons, Chichester 1181-1195
14. Brown J.D., and Heuvelink G. The Data Uncertainty Engine (DUE): a software tool for assessing and simulating uncertain environmental variables. Computers and Geosciences 33 2 (2007) 172-190
15. Brown J.D., Demargne J., Liu Y., Wu L., and Seo D.-J. Ensemble Verification System, Version 2.0, User's Manual (2009). http://www.nws.noaa.gov/oh/hrl/hsmb/docs/hydrologic_ensembles/EVS_2.0_MANUAL.pdf (accessed 01.11.10), 91 pp
16. Brown, J.D., Seo, D.-J. 2010. A nonparametric post-processor for bias-correction of hydrometeorological and hydrologic ensemble forecasts. Journal of Hydrometeorology, in press, doi:10.1175/2009JHM1188.1.
17. Casella G., and Berger R.L. Statistical Inference (1990), Duxbury Press 650 pp
18. Clark M., Gangopadhyay S., Hay L., Rajagopalan B., and Wilby R. The Schaake shuffle: a method for reconstructing space-time variability in forecasted precipitation and temperature fields. Journal of Hydrometeorology 5 (2004) 243-262
19. Dawson C.W., Abrahart R.J., and See L.M. HydroTest: a web-based toolbox of evaluation metrics for the standardised assessment of hydrological forecasts. Environmental Modelling and Software 22 (2007) 1034-1052
20. Demargne J., Wu L., Seo D.-J., and Schaake J. Experimental hydrometeorological and hydrologic ensemble forecasts and their verification in the U.S. National Weather Service. Quantification and Reduction of Predictive Uncertainty for Sustainable Water Resources Management (Proceedings of Symposium HS2004 at IUGG2007, Perugia, July 2007) vol. 313 (2007), IAHS Publication 177-187
21. Demargne J., Mullusky M., Werner K., Adams T., Lindsey S., Schwein N., Marosi W., and Welles E. Application of forecast verification science to operational river forecasting in the U.S. national weather service. Bulletin of the American Meteorological Society 90 6 (2009) 779-784
22. de Elìa R., and Laprise R. Distribution-oriented verification of limited-area model forecasts in a perfect-model framework. Monthly Weather Review 131 (2003) 2492-2509
23. de Lannoy G.J.M., Houser P.R., Pauwels V.R.N., and Verhoest N.E.C. Assessment of model uncertainty for soil moisture through ensemble verification. Journal of Geophysical Research 111 (2006) D10101 10.1029/2005JD006367
24. Ehrendorfer M., and Murphy A.H. Comparative evaluation of weather forecasting systems: sufficiency, quality, and accuracy. Monthly Weather Review 116 (1988) 1757-1770
25. Elmore K.L. Alternatives to the chi-square test for evaluating rank histograms from ensemble forecasts. Weather and Forecasting 20 (2005) 789-795
26. Fawcett T. An introduction to ROC analysis. Pattern Recognition Letters 27 (2006) 861-874
27. Flanagan D. Java in a Nutshell. fifth ed. (2005), O'Reilly and Associates, North Sebastopol, CA 1252 pp
28. Franz K.J., Hartmann H.C., Sorooshian S., and Bales R. Verification of national weather service ensemble streamflow predictions for water supply forecasting in the Colorado River Basin. Journal of Hydrometeorology 4 6 (2003) 1105-1118
29. Gneiting T.A., Raftery E., Westveld III A.H., and Goldman T. Calibrated probabilistic forecasting using ensemble model output statistics and minimum CRPS estimation. Monthly Weather Review 133 (2005) 1098-1118
30. Gneiting T., Balabdaoui F., and Raftery A.E. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society Series B: Statistical Methodology 69 2 (2007) 243-268
31. Good I.J. Rational decisions. Journal of the Royal Statistical Society 14 (1952) 107-114
32. Green D.M., and Swets J.M. Signal Detection Theory and Psychophysics (1966), John Wiley and Sons Inc., New York 455 pp
33. Gupta H.V., Beven K.J., and Wagener T. Model calibration and uncertainty estimation. In: Anderson M. (Ed). The Encyclopedia of Hydrological Sciences (2005), John Wiley & Sons, Chichester 2015-2032
34. Hamill T.M. Reliability diagrams for multicategory probabilistic forecasts. Weather and Forecasting 12 (1997) 736-741
35. Hamill T.M., Whittaker J.S., and Mullen S.L. Reforecasts: an important data set for improving weather predictions. Bulletin of the American Meteorological Society 87 1 (2006) 33-46
36. Hamill T.M., and Colucci S.J. Verification of Eta-RSM short-range ensemble forecasts. Monthly Weather Review 125 (1997) 1312-1327
37. Hashino T., Bradley A.A., and Schwartz S.S. Evaluation of bias-correction methods for ensemble streamflow volume forecasts. Hydrology and Earth System Sciences Discussions 3 (2006) 561-594
38. Hersbach H. Decomposition of the continuous ranked probability score for ensemble prediction systems. Weather and Forecasting 15 (2000) 559-570
39. Hou D., Kalnay E., and Droegemeier K.K. Objective verification of the SAMEX-98 ensemble forecasts. Monthly Weather Review 129 1 (2001) 73-91
40. Hsu W.-R., and Murphy A.H. The attributes diagram: a geometrical framework for assessing the quality of probability forecasts. International Journal of Forecasting 2 (1986) 285-293
41. Jakeman A.J., Letcher R.A., and Norton J.P. Ten iterative steps in development and evaluation of environmental models. Environmental Modelling and Software 21 (2006) 602-614
42. In: Jolliffe I.T., and Stephenson D.B. (Eds). Forecast Verification: A Practitioner's Guide in Atmospheric Science (2003), John Wiley and Sons, Chichester 240 pp
43. Karssenberg D., Schmitz O., Salamon P., de Jong K., and Bierkens M.F.P. A software framework for construction of process-based stochastic spatio-temporal models and data assimilation. Environmental Modelling and Software 25 4 (2010) 489-502 10.1016/j.envsoft.2009.10.004
44. Kennedy E.J. Techniques of Water-Resources Investigations of the United States Geological Survey, Book 3. Chapter A13: Computation of Continuous Records of Streamflow (1983), US Government Printing Office. http://pubs.usgs.gov/twri/twri3-a13/pdf/TWRI_3-A13.pdf (accessed 05.07.09), 52pp.
45. Kruger A., Khandewal S.G., and Bradley A. A web-based system for hydrologic forecast verification. Computers and Geosciences 33 6 (2006) 739-748
46. Mason S.J., and Graham N.E. Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation. Quarterly Journal of the Royal Meteorological Society 30 (2002) 291-303
47. Mason S.J. Understanding forecast verification statistics. Meteorological Applications 15 (2008) 31-40
48. Matheson J.E., and Winkler R.L. Scoring rules for continuous probability distributions. Management Science 22 (1976) 1087-1095
49. Murphy A.H., and Winkler R.L. A general framework for forecast verification. Monthly Weather Review 115 (1987) 1330-1338
50. Murphy A.H., and Winkler R.L. Diagnostic verification of probability forecasts. International Journal of Forecasting 7 (1992) 435-455
51. Murphy A.H. Forecast verification: its complexity and dimensionality. Weather and Forecasting 119 (1991) 1590-1601
52. Murphy A.H. General decompositions of MSE-based skill scores: measures of some basic aspects of forecast quality. Monthly Weather Review 124 (1996) 2353-2369
53. National Research Council of the National Academies (NRC). Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts (2006). http://www.nap.edu (accessed 05.07.09)
54. National Weather Service (NWS). National Weather Service River Forecast System (NWSRFS) User Manual Documentation (2005), National Weather Service Documentation, Silver Spring, Maryland, USA. http://www.nws.noaa.gov/oh/hrl/nwsrfs/users_manual/htm/xrfsdocpdf.php (accessed 05.07.09)
55. Pagano T.C., Garen D.C., and Sorooshian S. Evaluation of official Western US seasonal water supply outlooks, 1922-2002. Journal of Hydrometeorology 5 5 (2004) 683-698
56. Park S.K., and Xu L. Data Assimilation for Atmospheric, Oceanic and Hydrologic Applications (2009), Springer-Verlag 495pp
57. R. Development Core Team. R: A Language and Environment for Statistical Computing, Reference Index Version 2.7.2 (2008), R Foundation for Statistical Computing, Vienna, Austria. 3-900051-07-0. http://www.R-project.org URL:
58. Refsgaard J.C., van der Sluijs J.P., Højberg A.L., and Vanrolleghem P.A. Uncertainty in the environmental modeling process - a framework and guidance. Environmental Modelling and Software 22 (2007) 1543-1556
59. Schaake J., Demargne J., Hartman R., Mullusky M., Welles E., Wu L., Herr H., Fan X., and Seo D.J. Precipitation and temperature ensemble forecasts from single-value forecasts. Hydrology and Earth Systems Sciences 4 (2007) 655-717
60. Seo D.-J., Herr H.D., and Schaake J.C. A statistical post-processor for accounting of hydrologic uncertainty in short-range ensemble streamflow prediction. Hydrology and Earth System Sciences 3 (2006) 1987-2035
61. Shorack G.R., and Wellner J.A. Empirical Processes with Applications to Statistics (1986), John Wiley and Sons Inc. 976 pp.
62. Stensrud D.J., Brooks H.E., Du J., Tracton M.S., and Rogers E. Using ensembles for short-range forecasting. Monthly Weather Review 127 (1999) 433-446
63. Talagrand O. Assimilation of observations, an introduction. Journal of the Meteorological Society of Japan 75 (1997) 191-209
64. Toth Z., Kalnay E., Tracton S.M., Wobus R., and Irwin J. A synoptic evaluation of the NCEP ensemble. Weather and Forecasting 12 (1997) 140-153
65. Verbunt M., Zappa M., Gurtz J., and Kaufmann P. Verification of a coupled hydrometeorological modelling approach for alpine tributaries in the Rhine basin. Journal of Hydrology 324 (2006) 224-238
66. Wei M., Toth Z., Wobus R., and Zhu Y. Initial perturbations based on the ensemble transform (ET) technique in the NCEP global operational forecast system. Tellus 60A (2008) 62-79
67. Welles E., Sorooshian S., Carter G., and Olsen B. Hydrologic verification: a call for action and collaboration. Bulletin of the American Meteorological Society 88 (2007) 503-511
68. Wilks D.S. Statistical Methods in the Atmospheric Sciences. second ed. (2006), Academic Press 627pp
69. Wilks D.S. Diagnostic verification of the climate prediction center long-lead outlooks, 1995-98. Journal of Climate 13 (2000) 2389-2403
70. Wilson L.J., Burrows W.R., and Lanzinger A. A strategy for verification of weather element forecasts from an ensemble prediction system. Monthly Weather Review 127 (1999) 956-970