Progress and challenges in forecast verification

Meteorological Applications

Volumn 20, Issue 2, 2013, Pages 130-139

  Ebert, E ^a   Wilson, L ^b   Weigel, A ^c   Mittermaier, M ^d   Nurmi, P ^e   Gill, P ^d   Gober M ^f   Joslyn, S ^g   Brown, B ^h   Fowler, T ^h   Watkins, A ⁱ  

^a Bureau of Meteorology Centre for Australian Weather and Climate Research   (Australia)

^b ENVIRONMENT CANADA   (Canada)

^c MeteoSwiss   (Switzerland)

^d MET OFFICE   (United Kingdom)

^e FINNISH METEOROLOGICAL INSTITUTE   (Finland)

^f DEUTSCHER WETTERDIENST   (Germany)

^g UNIVERSITY OF WASHINGTON   (United States)

^h NATIONAL CENTER FOR ATMOSPHERIC RESEARCH   (United States)

ⁱ BUREAU OF METEOROLOGY   (Australia)

Author keywords

Challenges; Evaluation; Forecast verification

Indexed keywords

CHALLENGES; EVALUATION; FORECAST VERIFICATIONS; IMPROVEMENT OF METHODS; MELBOURNE , AUSTRALIA; METEOROLOGICAL INFORMATION; SPATIAL AND TEMPORAL SCALE; VERIFICATION METHOD;

FORECASTING;

DATA ASSIMILATION; DETECTION METHOD; ENSEMBLE FORECASTING; METEOROLOGY; OBSERVATIONAL METHOD; PREDICTION; SPATIOTEMPORAL ANALYSIS;

EID: 84879023139     PISSN: 13504827     EISSN: 14698080     Source Type: Journal    
DOI: 10.1002/met.1392     Document Type: Article

References (96)

1. Ambühl J. 2010. Customer oriented warning systems. Veröffentlichung MeteoSchweiz 84: 93 pp.
2. Ben Bouallègue Z. 2011. Upscaled and Fuzzy Probabilistic Forecasts: Verification Results. COSMO Newsletter, No. 11. http://cosmo-model.cscs.ch/content/model/documentation/newsLetters/newsLetter11/4_bouallengue.pdf (accessed 1 April 2013).
3. Bowler NE. 2006. Explicitly accounting for observation error in categorical verification of forecasts. Mon. Weather Rev. 134: 1600-1606.
4. Bowler NE. 2008. Accounting for the effect of observation errors on verification of MOGREPS. Meteorol. Appl. 15: 199-205.
5. Bullock R. 2011. Development and implementation of MODE time domain object-based verification. In 24th Conference Weather and Forecasting, 24-27 January 2011, Seattle, Washington. American Meteorological Society: Boston, MA.
6. Candille G, Talagrand O. 2008. Impact of observational error on the validation of ensemble prediction systems. Q. J. R. Meteorol. Soc. 134: 959-971.
7. Casati B, Wilson LJ, Stephenson DB, Ghelli A, Pocernich M, Damrath U, Ebert EE, Brown BG, Mason S. 2008. Forecast verification: current status and future directions. Meteorol. Appl. 15: 3-18.
8. Clark WL, Yuan H, Jensen TL, Wick G, Tollerud EI, Bullock RG, Sukovich E. 2011. Evaluation of GFS water vapor forecast errors during the 2009-2010 West Coast cool season using the MET/MODE object analyses package. In 25th Conference Hydrology, 24-27 January 2011, Seattle, Washington. American Meteorological Society: Boston, MA.
9. Clayton AM, Lorenc AC, Barker DM. 2012. Operational implementation of a hybrid ensemble/4D-Var global data assimilation system at the Met Office. Q. J. R. Meteorol. Soc. DOI: 10.1002/qj.2054.
10. Davis C, Brown B, Bullock R. 2006. Object-based verification of precipitation forecasts. Part I: methods and application to mesoscale rain areas. Mon. Weather Rev. 134: 1772-1784.
11. Demargne J, Mullusky M, Werner K, Adams T, Lindsey S, Schwein N, Marosi W, Welles E. 2009. Application of forecast verification science to operational river forecasting in the U.S. National Weather Service. Bull. Am. Meteorol. Soc. 90: 779-784.
12. Duc L, Saito K, Seko H. 2011. Application of spatial-temporal fractions skill score to high-resolution ensemble forecast verification. In5th International Verification Methods Workshop, 1-7 December 2011, Melbourne, Australia. http://cawcr.gov.au/events/verif2011/ posters/48_Duc_L.pdf (accessed 1 April 2013).
13. Ebert E. 2011. Radius of reliability: a distance metric for interpreting and verifying spatial probabilistic warnings. CAWCR Res. Lett. 6: 4-10.
14. Ebert EE, McBride JL. 2000. Verification of precipitation in weather systems: determination of systematic errors. J. Hydrol. 239: 179-202.
15. Ferro CAT, Stephenson DB. 2011. Extremal dependence indices: improved verification measures for deterministic forecasts of rare binary events. Weather Forecast. 26: 699-713.
16. Foley AM, Leahy PG, Marvuglia A, McKeogh EJ. 2012. Current methods and advances in forecasting wind power generation. Renew. Energy 37: 1-8.
17. Fowler TL, Jensen T, Tollerud EI, Halley Gotway J, Oldenburg P, Bullock R. 2010. New Model Evaluation Tools (MET) software capabilities for QPF verification. Preprints, 3rd International Conference on QPE, QPF, 18-22 October 2010, Nanjing, China.
18. Friederichs P, Göber M, Bentzien S, Lenz A, Krampitz R. 2009. A probabilistic analysis of wind gusts using extreme value statistics. Meteorol. Z. 18: 615-629.
19. Gallus WA Jr. 2010. Application of object-based verification techniques to ensemble precipitation forecasts. Weather Forecast. 25: 144-158.
20. Gangsto R, Weigel AP, Liniger MA, Appenzeller C. 2013. Meteorological aspects of the evaluation of decadal predictions. Clim. Res. 55: 181-200.
21. Ghelli A, Primo C. 2009. On the use of the extreme dependency score to investigate the performance of an NWP model for rare events. Meteorol. Appl. 16: 537-544.
22. Gill PG. 2011. Aviation verification - Recent developments at the UK Met Office. 5th International Verification Methods Workshop, 1-7 December 2011, Melbourne, Australia. http://cawcr.gov.au/events/verif2011/ppt/P_Gill.pdf (accessed 1 April 2013).
23. Gill PG. 2013. Objective verification of World Area Forecast Centre clear air turbulence forecasts. Meteorol. Appl. DOI: 10.1002/met.1288.
24. Gill PG, Buchanan P. 2013. An ensemble based turbulence forecasting system. Meteorol. Appl. DOI: 10.1002/met.1373.
25. Gill PG, Stirling AJ. 2013. Including convection in global turbulence forecasts. Meteorol. Appl., 20: 107-114. DOI: 10.1002/met.1315.
26. Gilleland E, Ahijevych D, Brown BG, Casati B, Ebert EE. 2009. Intercomparison of spatial forecast verification methods. Weather Forecast. 24: 1416-1430.
27. Gneiting T, Stanberry LI, Grimit EP, Held L, Johnson NA. 2008. Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds. Test 17: 211-235.
28. Gombos D, Hansen JA, Du J, McQueen J. 2007. Theory and applications of the minimum spanning tree rank histogram. Mon. Weather Rev. 135: 1490-1505.
29. Gorgas T, Dorninger M. 2012a. Quantifying verification uncertainty by reference data variation. Meteorol. Z. 21: 259-277.
30. Gorgas T, Dorninger M. 2012b. Concepts for a pattern-oriented analysis ensemble based on observational uncertainties. Q. J. R. Meteorol. Soc. 138: 769-784.
31. Haiden T, Rodwell MJ, Richardson DS, Okagaki A, Robinson T, Hewson T. 2012. Intercomparison of global model precipitation forecast skill in 2010/11 using the SEEPS score. Mon. Weather Rev. 140: 2720-2733.
32. Hamill T. 2011. Examining characteristics of analysis and short-range forecast errors using TIGGE. Presentation to NCEP/EMC/GMB, December, 2011, http://www.esrl.noaa.gov/psd/people/tom.hamill/analysis-errors-wgne-hamill.pdf (accessed 1 April 2013).
33. Hogan R, O'Connor EJ, Illingworth AJ. 2009. Verification of cloud-fraction forecasts. Q. J. R. Meteorol. Soc. 135: 1494-1511.
34. Huuskonen A, Delobbe L, Urban B. 2012. EUMETNET OPERA: achievements of OPERA-3 and challenges ahead. Proceedings of ERAD 2012, 25-29 June 2012, Toulouse, France.
35. Jolliffe IT. 2007. Uncertainty and inference for verification measures. Weather Forecast. 22: 637-650.
36. Jolliffe IT, Stephenson DB. 2012. Forecast Verification: A Practitioner's Guide in Atmospheric Science, 2nd edn. Wiley and Sons Ltd.: Chichester, UK; 274pp.
37. Joslyn S, Nemec L, Savelli S. 2013. The benefits and challenges of predictive interval forecasts and verification graphics for end-users. Weather, Clim. Soc. (in press).
38. Joslyn S, Savelli S. 2010. Communicating forecast uncertainty: public perception of weather forecast uncertainty. Meteorol. Appl. 17: 180-195.
39. Judd K, Smith LA, Weisheimer A. 2007. How good is an ensemble at capturing truth? Using bounding boxes for forecast evaluation. Q. J. R. Meteorol. Soc. 133: 1309-1325.
40. Jung T. 2012. The WWRP Polar Prediction Project. Planning Workshop of the WCRP Polar Prediction Initiative, 12-14 April 2012, Toronto, Canada.
41. JWGFVR. 2013. Forecast verification: issues, methods and FAQ. http://www.cawcr.gov.au/projects/verification (accessed 28 January 2013).
42. Koh T-Y, Wang S, Bhatt BC. 2012. A diagnostic suite to assess NWP performance. J. Geophys. Res. 117: D13109, DOI: 10.1029/2011JD017103.
43. Kucera PA, Weeks C, Brown B, Bullock R. 2011. Development of new diagnostic tools to evaluate NWP cloud and precipitation products using A-Train satellite observations. 5th International Verification Methods Workshop, 1-7 December 2011, Melbourne, Australia. http://cawcr.gov.au/events/verif2011/ppt/P_Kucera.pdf (accessed 1 April 2013).
44. Layne GJ, Lack SA. 2010. Methods for estimating air traffic capacity reductions due to convective weather for verification. 14th Conference on Aviation, Range, and Aerospace Meteorology (ARAM), Atlanta, Georgia.
45. Lorenz EN. 1993. The Essence of Chaos. University of Washington Press: Seattle, WA; 226pp.
46. Louey J. 2011. Verification presentation in management reporting. 5th International Verification Methods Workshop, 1-7 December 2011, Melbourne, Australia. http://cawcr.gov.au/events/verif2011/ppt/J_Louey.pdf (accessed 1 April 2013).
47. Madsen H, Pinson P, Kariniotakis G, Nielsen HA, Nielsen TS. 2005. Standardizing the performance evaluation of short term wind power prediction models. Wind Eng. 29: 475-489.
48. Mason SJ. 2011. Seasonal and longer-range forecasts. In Forecast Verification: A Practitioner's Guide in Atmospheric Science, 2nd edn, Jolliffe IT, Stephenson DB (eds). John Wiley & Sons Ltd.: Chichester, UK; 203-220.
49. Mason SJ, Galpin JS, Goddard L, Graham NE, Rajartnam B. 2007. Conditional exceedance probabilities. Mon. Weather Rev. 135: 363-372.
50. Mason SJ, Graham NE. 2002. Areas beneath the relative operating characteristics (ROC) and levels (ROL) curves: statistical significance and interpretation. Q. J. R. Meteorol. Soc. 128: 2145-2166.
51. Mason SJ, Weigel AP. 2009. A generic forecast verification framework for administrative purposes. Mon. Weather Rev. 137: 331-349.
52. Mass C. 2012. Nowcasting: the promise of new technologies of communication, modelling, and observation. Bull. Am. Meteorol. Soc. 93: 797-809.
53. Met Office. 2010. Global NWP index documentation. http://www.wmo.int/pages/prog/www/DPFS/Meetings/CG-FV_Montreal2011/Doc4-6-Annex_Global_index_documentation_2010-2.pdf (accessed 1 April 2013).
54. Mittermaier MP, Bullock R. 2013. Using MODE to explore the spatial and temporal characteristics of cloud cover forecasts from high-resolution NWP models. Meteorol. Appl. 20: 187-196 DOI: 10.1002/met.1393 (this issue).
55. Mittermaier M, Roberts N. 2010. Intercomparison of spatial forecast verification methods: identifying skillful spatial scales using the fractions skill score. Weather Forecast. 25: 343-354.
56. Mittermaier MP, Roberts N, Thompson SA. 2011. A long-term assessment of precipitation forecast skill using the Fractions Skill Score. Meteorol. Appl. 20: 176-186 DOI: 10.1002/met.296 (this issue).
57. Moise AF, Delage FP. 2011. New climate model metrics based on object-orientated pattern matching of rainfall. J. Geophys. Res. 116: D12108, DOI: 10.1029/2010JD015318.
58. North R, Trueman M, Mittermaier M, Rodwell M. 2013. An assessment of the SEEPS and SEDI metrics for the verification of 6h forecast precipitation accumulations. Meteorol. Appl. 20: 164-175 DOI: 10.1002/met.1405 (this issue).
59. Nurmi P. 2010. Experimentation with new verification measures for categorized QPFs in the verification of high impact precipitation events. Proceedings 3rd WMO International Conference Quantitative Precipitation Estimation and Quantitative Precipitation Forecasting and Hydrology, 18-22 October 2010, Nanjing, China; 222-226.
60. Nurmi P. 2011. Operational verification systems at FMI. 5th International Verification Methods Workshop. http://cawcr.gov.au/events/verif2011/ppt1/Nurmi_P.pdf (accessed 1 April 2013).
61. Nurmi P, Perrels A, Nurmi V. 2013. Expected impacts and value of improvements in weather forecasting on the road transport sector. Meteorol. Appl. 20: 217-223 DOI: 10.1002/met.1399 (this issue).
62. Park YY, Buizza R, Leutbecher M. 2008. TIGGE: preliminary results on comparing and combining ensembles. Q. J. R. Meteorol. Soc. 134: 2029-2050.
63. Pierce CS. 1884. The numerical measure of success in predictions. Science 4: 453-454.
64. Pocernich M. 2010. Verification of wind forecasts of ramping events. http://ral.ucar.edu/projects/wind_energy_workshop/presentations/Ramp_Verification_Methods_Pocernich_28.pdf (accessed 1 April 2013).
65. Potts JM, Folland CK, Jolliffe IT, Sexton D. 1996. Revised "LEPS" scores for assessing climate model simulations and long-range forecasts. J. Climate 9: 34-53.
66. Primo C, Ghelli A. 2009. The effect of the base rate on the extreme dependency score. Meteorol. Appl. 16: 533-535.
67. R Development Core Team. 2011. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing: ViennaISBN 3-900051-07-0, http://www.R-project.org (accessed 1 April 2013).
68. Rickard GJ, Lunnon RW, Tenenbaum J. 2001. The Met Office upper air winds: prediction and verification in the context of commercial aviation data. Meteorol. Appl. 8: 351-360.
69. Rikus L, Elliott T, Dietachmayer G. 2011. Using image mapping techniques to generate model evaluation metrics. 5th International Verification Methods Workshop, 1-7 December 2011, Melbourne, Australia. http://cawcr.gov.au/events/verif2011/posters/46_Rikus_L.pdf (accessed 1 April 2013).
70. Roberts RD, Anderson ARS, Nelson E, Brown BG, Wilson JW, Pocernich M, Saxen T. 2012. Impacts of forecaster involvement on convective storm initiation and evolution nowcasting. Weather Forecast. 27: 1061-1089.
71. Roberts NM, Lean HW. 2008. Scale-selective verification of rainfall accumulations from high-resolution forecasts of convective events. Mon. Weather Rev. 136: 78-97.
72. Rodwell M, Haiden T, Richardson D. 2011. Developments in precipitation verification. ECMWF Newsl. 128: 12-16.
73. Rodwell MJ, Richardson DS, Hewson TD, Haiden T. 2010. A new equitable score suitable for verifying precipitation in numerical weather prediction. Q. J. R. Meteorol. Soc. 136: 1344-1363.
74. Roebber PJ. 2009. Visualizing multiple measures of forecast quality. Weather Forecast. 24: 601-608.
75. Saetra O, Hersbach H, Bidlot J-R, Richardson D. 2004. Effects of observation errors on the statistics for ensemble spread and reliability. Mon. Weather Rev. 132: 1487-1501.
76. Santos C, Ghelli A. 2012. Observational probability method to assess ensemble precipitation forecasts. Q. J. R. Meteorol. Soc. 138: 209-221.
77. Schwartz CS, Kain JS, Weiss SJ, Xue M, Bright DR, Kong F, Thomas KW, Levit JJ, Coniglio MC, Wandishin MS. 2010. Toward improved convection-allowing ensembles: model physics sensitivities and optimizing probabilistic guidance with small ensemble membership. Weather Forecast. 25: 263-280.
78. Sheskin DJ. 2007. Handbook of Parametric and Nonparametric Statistical Procedures. Chapman and Hall/CRC: Boca Raton, FL; 1776pp.
79. Smith LA. 2001. Disentangling uncertainty and error: on the predictability of nonlinear systems. In Nonlinear Dynamics and Statistics, Mees AI (ed). Birkhäuser Press: Boston, MA; 31-64.
80. Smith LA, Hansen JA. 2004. Extending the limits of ensemble forecast verification with the minimum spanning tree. Mon. Weather Rev. 132: 1522-1528.
81. Stephenson DB, Casati B, Ferro CAT, Wilson CA. 2008. The extreme dependency score: a non-vanishing measure for forecasts of rare events. Meteorol. Appl. 15: 41-50.
82. Stockdale TN, Anderson DLT, Alves JOS, Balmaseda MA. 1998. Global seasonal rainfall forecasts using a coupled ocean-atmosphere model. Nature 392: 370-373.
83. Weigel AP. 2011. Verification of ensemble forecasts. In Forecast Verification: A Practitioner's Guide in Atmospheric Science, 2nd edn, Jolliffe IT, Stephenson DB (eds). John Wiley & Sons Ltd.: Chichester, UK; 141-166.
84. Weigel AP, Mason SJ. 2011. The generalized discrimination score for ensemble forecasts. Mon. Weather Rev. 139: 3069-3074.
85. Weisheimer A, Smith LA, Judd K. 2005. A new view of seasonal forecast skill: bounding boxes from the DEMETER ensemble forecasts. Tellus 57A: 265-279.
86. Welles E, Sorooshian S. 2009. Scientific verification of deterministic river stage forecasts. J. Hydrometeorol. 10: 507-520.
87. Weusthoff T. 2011. Neighbourhood verification in space and time. 5th International Verification Methods Workshop, 1-7 December 2011, Melbourne, Australia. http://cawcr.gov.au/events/verif2011/posters/49_Weusthoff_T.pdf (accessed 1 April 2013).
88. Weusthoff T, Ament F, Arpagaus A, Rotach MW. 2010. Assessing the benefits of convection-permitting models by neighborhood verification: examples from MAP D-PHASE. Mon. Weather Rev. 138: 3418-3433.
89. Wilks DS. 2004. The minimum spanning tree histogram as a verification tool for multidimensional ensemble forecasts. Mon. Weather Rev. 132: 1329-1340.
90. Wilks DS. 2005. Statistical Methods in the Atmospheric Sciences, 2nd edn. Elsevier: Amsterdam; 627pp.
91. Williamson D, Nakagawa M, Klein S, Earnshaw P, Nunes A, Roads J. 2009. Transpose AMIP: a process oriented climate model evaluation and intercomparison using model weather forecasts and field campaign observations. EGU General Assembly, 19-24 April 2009, Vienna, Austria.
92. Wilson LJ, Giles A. 2013. A new index for the verification of accuracy and timeliness of weather warnings. Meteorol. Appl. 20: 206-216 DOI: 10.1002/met.1404 (this issue).
93. WMO. 2008. Severe Weather Forecasting Demonstration Project (SWFDP) Guidebook on Planning Regional Subprojects. World Meteorological Organization Commission for Basic Systems. World Meteorological Organization: Geneva, Switzerland; 60pp.
94. Zacharov P, Rezacova D. 2009. Using the fractions skill score to assess the relationship between an ensemble QPF spread and skill. Atmos. Res. 94: 684-693.
95. Zhu X, Genton MG. 2012. Short-term wind speed forecasting for power system operations. Int. Stat. Rev. 80: 2-23.
96. Zimmer M, Wernli H. 2011. Verification of quantitative precipitation forecasts on short time-scales: a fuzzy approach to handle timing errors with SAL. Meteorol. Z. 20: 95-105.