Evaluating uncertainty estimates in hydrologic models: Borrowing measures from the forecast verification community

Hydrology and Earth System Sciences

Volumn 15, Issue 11, 2011, Pages 3367-3382

  Franz, K J ^a   Hogue, T S ^b  

^a IOWA STATE UNIVERSITY   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ATMOSPHERIC SCIENCE; COMMUNITY IS; CONDITIONAL STATISTICS; ENSEMBLE DATA ASSIMILATION; HYDROLOGIC MODELS; MODEL EVALUATION; MODEL PERFORMANCE; MODEL SIMULATION; MODEL VALIDATION; MULTI-MODELING; NATIONAL WEATHER SERVICES; PARAMETER UNCERTAINTY; PROBABILISTIC ESTIMATES; PROBABILISTIC FORECASTS; PROBABILISTIC OUTPUT; ROBUST ANALYSIS; SHUFFLE COMPLEX EVOLUTIONS; STREAMFLOW PREDICTION; UNCERTAINTY ESTIMATES;

DATA PROCESSING; ESTIMATION; GEOLOGIC MODELS; MATHEMATICAL MODELS; REACTOR CORES; SOIL MOISTURE; STREAM FLOW; UNCERTAINTY ANALYSIS; WEATHER FORECASTING;

COMPUTER SIMULATION;

ACCURACY ASSESSMENT; DATA ASSIMILATION; ERROR ANALYSIS; ESTIMATION METHOD; FORECASTING METHOD; HYDROLOGICAL MODELING; IMPLEMENTATION PROCESS; NUMERICAL MODEL; SOIL MOISTURE; STREAMFLOW; UNCERTAINTY ANALYSIS;

UNITED STATES;

EID: 81355141811     PISSN: 10275606     EISSN: 16077938     Source Type: Journal    
DOI: 10.5194/hess-15-3367-2011     Document Type: Article

References (77)

1. Ajami, N. K., Duan, Q., and Sorooshian, S.: An integrated hy-drologic Bayesian multimodel combination framework: Confronting input, parameter, and model structural uncertainty in hydrologic prediction, Water Resour. Res., 43, W01403, doi:10.1029/2005WR004745, 2007. (Pubitemid 46364281)
2. Bartholmes, J. C., Thielen, J., Ramos, M. H., and Gentilini, S.: The european flood alert system EFAS-Part 2: Statistical skill assessment of probabilistic and deterministic operational forecasts, Hydrol. Earth Syst. Sci., 13, 141-153, doi:10.5194/hess-13-141-2009, 2009.
3. Beven, K.: A manifesto for the equifinality thesis, J. Hydrol., 320, 18-36, 2006.
4. Beven, K. and Binley, A.: Future of distributed models: Model calibration and uncertainty prediction, Hydrol. Process., 6, 279-298, 1992.
5. Beven, K. and Freer, J.: Equifinality, data assimilation, and uncertainty estimation in mechanistic modelling of complex environmental systems using the GLUE methodology, J. Hydrol., 249, 11-29, doi:10.1016/S0022-1694(01)00421-8, 2001. (Pubitemid 32664993)
6. Box, G. E. P. and Tiao, G. C.: Bayesian Inference in Statistical Analysis, Addison-Wesley, 1973.
7. Bradley, A. A. and Schwartz, S. S.: Summary verification measures and their interpretation for ensemble forecasts, Mon. Weather Rev., 3075-3089, 2011.
8. Bradley, A. A., Hashino, T., and Schwartz, S. S.: Distributions-oriented verification of probability forecast for small data samples, Weather Forecast., 18, 903-917, 2003. (Pubitemid 37450300)
9. Bradley, A. A., Schwartz, S. S., and Hashino, T.: Distributions-oriented verification of ensemble streamflow predictions, J. Hy-drometeorol., 5, 532-545, 2004. (Pubitemid 38978532)
10. Brazil, L. E. and Hudlow, M. D.: Calibration procedures used with the National Weather Service Forecast System, in: Water and Realted Land Resource Systems, edited by: Haimes, Y. Y. and Kindler, J., Pergamon, Tarrytown, N.Y., 457-466, 1981. (Pubitemid 12497724)
11. Brier, G. W.: Verification of forecasts expressed in terms of probabilities, Mon. Weather Rev., 78, 1-3, 1950.
12. Brown, J., Demargne, J., Seo, D.-J., and Liu, Y.: The Ensemble Verification System (EVS): a software tool for verifying ensemble forecasts of hydrometeorological and hydrologic variables at discrete locations, Environ. Modell. Softw., 25, 854-872, doi:10/1016/j.envsoft.2010.01.009, 2010.
13. Burnash, R. J., Ferral, R. L., and McGuire, R. A.: A Generalized Streamflow Simulation System Conceptual: Modeling for Digital Computers, Joint Federal-State River Forecast Center, Sacramento, CA, 1973.
14. Clark, M. P. and Kavetski, D.: Ancient numerical daemons of conceptual hydrological modeling: 1. Fidelity and efficiency of time stepping schemes, Water Resour. Res., 46, W10510, doi:10.1029/2009WR008894, 2010.
15. Coccia, G. and Todini, E.: Recent developments in predictive uncertainty assessment based on the model conditional processor approach, Hydrol. Earth Syst. Sci., 15, 3253-3274, doi:10.5194/hess-15-3253-2011, 2011.
16. Cooke, W. E.: Forecasts and verifications in Western Australia, Mon. Weather Rev., 34, 23-24, 1906.
17. Christoffersen, P. F.: Evaluating interval forecasts, Int. Econ. Rev., 39, 841-862, 1998. (Pubitemid 128470721)
18. Day, G. N.: Extended streamflow forecasting using NWSRFS, J. Water Resour. Plann. Manage., 111, 157-170, 1985.
19. De Finetti, B.: Foresight: its logical laws, its subjective sources, in: Studies in Subjective Probability, edited y: Kyburg Jr., H. E. and Smokler, H. E., Wiley, New York, 1964, 94-158, 1937.
20. De Lannoy, G. J. M., Houser, P. R., Pauwels, V. R. N., and Verhoest, N. E. C.: Assessment of model uncertainty for soil moisture through ensemble verification, J. Geophys. Res., 111, D10101, doi:10.1029/2005JD006367, 2006. (Pubitemid 44043240)
21. Demargne, J., Brown, J., Liu, Y., Seo, D. J., Wu, L., Toth, Z., and Zhu, Y.: Diagnostic verification of hydrometerological ensembles, Atmos. Sci. Lett., 11, 114-122, 2010.
22. Duan, Q., Sorooshian, S., and Gupta, V.: Effective and efficient global optimization for conceptual rainfall-runoff models, Water Resour. Res., 28, 1015-1031, 1992.
23. Duan, Q., Gupta, V. K., and Sorooshian, S.: A Shuffled Complex Evolution Approach for Effective and Efficient Global Optimization, J. Opt. Theory App., 76, 501-521, 1993. (Pubitemid 23643003)
24. Duan, Q., Ajami, N. K., Gao, X., and Sorooshian, S.: Multi-model ensemble hydrologic prediction using Bayesian model averaging, Adv. Water Resour., 30, 1371-1386, 2007. (Pubitemid 46323445)
25. Evensen, G.: Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics, J. Geophys. Res., 99, 10143-10162, 1994.
26. Faber, B. A. and Stedinger, J. R.: Reservoir optimization using sampling SDP with ensemble streamflow prediction (ESP) forecasts, J. Hydrol., 249, 113-133, 2001. (Pubitemid 32664999)
27. Finley, J. P.: Tornado predictions, Am. Meteorol. J., 1, 85-88, 1884.
28. Franz, K. J., Hartmann, H. C., Sorooshian, S., and Bales, R.: Verification of National Weather Service Ensemble Streamflow Predictions for water supply forecasting in the Colorado River basin, J. Hydrometeorol., 4, 1105-1118, 2003. (Pubitemid 38162558)
29. Franz, K. J., Hogue, T., and Sorooshian, S.: Snow model verification using ensemble prediction and operational benchmarks, J. Hydrometeorol., 9, 1402-1415, 2008.
30. Franz, K. J., Butcher, P., and Ajami, N. K.: Addressing snow model uncertainty for hydrologic prediction, Adv. Water Resour., 33, 820-832, 2010.
31. Gelman, A. And Rubin, D. B.: Inference from iterative simulation using multiple sequences, Stat. Sci., 7, 457-472, 1992.
32. Gupta, H. V., Beven, K. J., and Wagener, T.: Model calibration and uncertainty estimation, in: Encyclopedia of Hydrologic Sciences, edited by: Anderson, M. G. and Mcconnell, J. J., John Wiley, New York, 2015-2031, 2006.
33. Hamill, T. M.: Interpretation of the rank histogram for verifying ensemble forecasts, Mon. Weather Rev., 129, 550-560, 2001. (Pubitemid 32418375)
34. Hamill, T. M. and Collucci, S. J.: Verification of Eta-RSM short-range ensemble forecasts, Mon. Weather Rev., 125, 1312-1327, 1997. (Pubitemid 127592870)
35. Hastings, W. K.: Monte-Carlo sampling methods using Markov Chains and their applications, Biometrika, 57, 97-109, 1970.
36. Jolliffe, I. T. and Stephenson, D. B.: Forecast Verification: A Prac-tioner's Guide in Atmospheric Science, John Wiley and Sons, Chichester, 240 pp., 2003.
37. Kavetski, D. and Clark, M. P.: Ancient numerical daemons of conceptual hydrological modeling: 2. Impact of time stepping schemes on model analysis and prediction, Water Resour. Res., 46, W10511, doi:10.1029/2009WR008896, 2010.
38. Kitanidis, P. K. and Bras, R. L.: Real-time forecasting with a conceptual hydrologic model I. analysis of uncertainty, Water Re-sour. Res., 16, 1025-1033, 1980a.
39. Kitanidis, P. K. and Bras, R. L.: Real-Time Forecasting with a conceptual hydrologic model 2. applications and results, Water Re-sour. Res., 16, 1034-1044, 1980b.
40. Krzysztofowicz, R.: The case for probabilistic forecasting in hydrology, J. Hydrol., 249, 2-9, 2001. (Pubitemid 32664992)
41. Krzysztofowicz, R. and Kelly, K. S.: Hydrologic uncertainty processor for probabilistic river stage forecasting, Water Resour. Res., 36, 3265-3277, 2000.
42. Laio, F. and Tamea, S.: Verification tools for probabilistic forecasts of continuous hydrological variables, Hydrol. Earth Syst. Sci., 11, 1267-1277, doi:10.5194/hess-11-1267-2007, 2007. (Pubitemid 46707033)
43. Margulis, S. A., McLaughlin, D., Entekhabi, D., and Dunne, S.: Land data assimilation and estimation of soil moisture using measurements from the Southern Great Plains 1997 Field Experiment, Water Resour. Res., 38, 1299, doi:10.1029/2001WR001114, 2002.
44. Mason, S. J. and Graham, N. E.: Conditional probabilities, relative operating characteristics, and relative operating levels, Weather Forecast., 14, 713-725, 1999. (Pubitemid 30065843)
45. Metropolis, N., Rosenbluth, A. W., Rosenbluth, M. N., Teller, A. H., and Teller, E.: Equations of state calculations by fast computing machines, J. Chem. Phys., 21, 1087-1091, 1953.
46. Montanari, A. and Brath, A.: A stochastic approach for assessing the uncertainty of rainfall-runoff simulations, Water Resour. Res., 40, W01106, doi:10.1029/2003WR002540, 2004.
47. Moriasi, D. N., Arnold, J. G., Van Liew, M. W., Bingner, R. L., Harmel, R. D., and Veith, T. L.: Model evaluation guidelines for systematic quantification of accuracy in watershed simulations, Trans, ASABE, 50, 885-900, 2007. (Pubitemid 47074809)
48. Muleta, M. K. and Nicklow, J. W.: Sensitivity and uncertainty analysis coupled with automatic calibration for a distributed watershed model, J. Hydrol., 306, 127-145, 2005. (Pubitemid 40678746)
49. Murphy, A. H.: A note on the utility of probabilistic predictions and the probability score in the cost-loss ratio decision situation, J. Appl. Meteorol., 5, 534-537, 1966.
50. Murphy, A. H.: Forecast verification: Its complexity and dimensionality, Mon. Weather Rev., 119, 1590-1601, 1991.
51. Murphy, A. H.: A coherent method of stratification within a general framework for forecast verification, Mon. Weather Rev., 123, 1582-1588, 1995.
52. Murphy, A. H.: The Finley affair: A signal event in the history of forecast verification, Weather Forecast., 11, 3-20, 1996.
53. Murphy, A. H.: Forecast Verification, in: Economic value of weather and climate forecasts, edited by: Katz, R. W. and Murphy, A. H., Cambridge University Press, Cambridge, MA, 240 pp., 1997.
54. Murphy, A. H. and Epstein, E. S.: A note on probability forecasts and "Hedging", J. Appl. Meteorol., 6, 1002-1004, 1967.
55. Murphy, A. H. and Wilks, D. S.: A case study of the use of statistical models in forecast verification: Precipitation probability forecasts, Weather Forecast., 13, 795-810, 1998.
56. Murphy, A. H. and Winkler, R. L.: A general framework for forecast verification, Mon. Weather Rev., 115, 1330-1338, 1987.
57. Murphy, A. H. and Winkler, R. L.: Diagnostic verification of probability forecasts, Hydrol. Process., 7, 435-455, 1992.
58. Murphey, A. H., Brown, B. G., and Chen, Y.: Diagnostic verification of temperature forecasts, Weather Forecast., 4, 485-501, 1989.
59. NWS: National Weather Service River Forecast Verification Plan, Report of the Hydrologic Verification System Requirements Team, October 2006, http://nws.noaa.gov/oh/rfcdev/docs/Final Verification Report.pdf, last access: October 2010, US Department of Commerce, NOAA/NWS, Silver Spring, Maryland, 2006.
60. Ramsey, F. P.: Truth and Probability, in: The Foundations of Mathematics and Other Logical Essays, edited by: Braithwaite, R. B.,Humanities Press, New York, 1950, 156-198, 1926.
61. Randrianasolo, A., Ramos, M. H., Thirel, G., Andréassian, V., and Martin, E.: Comparing the Scores of hydrologic ensemble forecasts issued by two different hydrological models, Atmos. Sci. Lett., 11, 100-107, 2010.
62. Renner, M., Werner, M. G. F., Rademacher, S., and Sprokkereef, E.: Verification of ensemble flow forecasts for the River Rhine, J. Hydrol., 376, 463-475, doi:10.1016/j.jhydrol.2009.07.059, 2009.
63. Schoups, G., Vrugt, J. A., Fenicia, F., and van de Giesen, N. C.: Corruption of accuracy and efficiency of Markov chain Monte Carlo simulation by inaccurate numerical implementation of conceptual hydrologic models, Water Resour. Res., 46, W10530, doi:10.1029/2009WR008648, 2010.
64. Seo, D. J., Koren, V., and Cajina, N.: Real-time variational assimilation of hydrologic and hydrometeorological data into operational hydrologic forecasting, J. Hydrometeorol., 4, 627-641, 2003. (Pubitemid 36906494)
65. Seo, D. J., Cajina, L., Corby, R., and Howieson, T.: Automatic state updating for operational streamflow forecasting via variational data assimilation, J. Hydrol., 367, 255-275, 2009.
66. Shrestha, D. L., Kayastha, N., and Solomatine, D. P.: A novel approach to parameter uncertainty analysis of hydrological models using neural networks, Hydrol. Earth Syst. Sci., 13, 1235-1248, doi:10.5194/hess-13-1235-2009, 2009.
67. Thirel, G., Rousset-Regimbeau, F., Martin, E., and Habets, F.: On the impact of short-range meteorological forecasts for Ensemble Streamflow Predictions, J. Hydrometeorol., 9, 1301-1317, 2008.
68. Verbunt, M., Zappa, M., Gurtz, J., and Kaufmann, P.: Verification of a coupled hydrometeorological modeling approach for alpine tributaries in the Rhine basin, J. Hydrol., 324, 224-238, doi:10.1016/j.jhydrol.2005.09.036, 2006. (Pubitemid 43729378)
69. Vrugt, J. A. and Robinson, B. A.: Treatment of uncertainty using ensemble methods: Comparison of sequential data assimilation and Bayesian model averaging, Water Resour. Res., 43, W01411, doi:10.1029/2005WR004838, 2007. (Pubitemid 46364289)
70. Vrugt, J. A., Gupta, H. V., Bouten, W., and Sorooshian, S.: A shuffled complex evolution Metropolis algorithm for optimization and uncertainty assessment of hydrological model parameters, Water Resour. Res., 39, 1201, 2003.
71. Vrugt, J. A., Gupta, H. V., Nuallain, B., and Bouten, W.: Real-time data assimilation for operational ensemble streamflow forecasting, J. Hydrometeorol., 7, 548-565, 2006. (Pubitemid 44087847)
72. Weerts, A. H., Winsemius, H. C., and Verkade, J. S.: Estimation of predictive hydrological uncertainty using quantile regression: examples from the National Flood Forecasting System (England and Wales), Hydrol. Earth Syst. Sci., 15, 255-265, doi:10.5194/hess-15-255-2011, 2011.
73. Wilks, D. S.: Resampling hypothesis tests for autocorrelated fields, J. Climate, 10, 65-82, 1997. (Pubitemid 28000974)
74. Wilks, D. S.: Multisite generalizations of a daily stochastic precipitation generation model, J. Hydrol., 210, 178-191, 1998. (Pubitemid 28504056)
75. Wilks, D. S.: Statistical Methods in the Atmospheric Sciences, 2nd Edn., Academic Press, Amsterdam, 627 pp., 2006.
76. Xiong, L. and O'Connor, K. M.: An empirical method to improve the prediction limits of the GLUE methodology in rainfall-runoff modeling, J. Hydrol., 349, 115-124, 2008.
77. Zak, S. and Beven, K.: Equifinality, sensitivity and predictive uncertainty in the estimation of critical loads, Sci. Total Environ., 236, 191-214, 1999. (Pubitemid 29435964)