Testing competing precipitation forecasts accurately and efficiently: The Spatial Prediction Comparison Test

Monthly Weather Review

Volumn 141, Issue 1, 2013, Pages 340-355

  Gilleland, Eric ^a  

^a NATIONAL CENTER FOR ATMOSPHERIC RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ASSESSMENT TECHNIQUE; HIGH SPATIAL RESOLUTION; PRECIPITATION FORECAST; PRECIPITATION MODEL; SPATIAL ASSESSMENT; SPATIAL CORRELATIONS; SPATIAL PREDICTION; VERIFICATION RESULTS;

ERRORS; LASERS; NORMAL DISTRIBUTION; STATISTICAL TESTS;

WEATHER FORECASTING;

ASSESSMENT METHOD; CLIMATE MODELING; CLIMATE PREDICTION; CORRELATION; ERROR ANALYSIS; GEOSTATISTICS; PRECIPITATION (CLIMATOLOGY); SPATIAL ANALYSIS; TESTING METHOD; WEATHER FORECASTING;

EID: 84874925379     PISSN: 00270644     EISSN: 15200493     Source Type: Journal    
DOI: 10.1175/MWR-D-12-00155.1     Document Type: Article

References (107)

1. Aberg, S., F. Lindgren, A. Malmberg, J. Holst, and U. Holst, 2005: An image warping approach to spatio-temporal modelling. Environmetrics, 16, 833-848.
2. AghaKouchak, A., A. Behrangi, S. Sorooshian, K. Hsu, and E. Amitai, 2011: Evaluation of satellite-retrieved extreme precipitation rates across the central United States. J. Geophys. Res., 116, D02115, doi:10.1029/2010JD014741.
3. Ahijevych, D., E. Gilleland, B. G. Brown, and E. E. Ebert, 2009: Application of spatial verification methods to idealized and NWP-gridded precipitation forecasts. Wea. Forecasting, 24, 1485-1497.
4. Alexander, G. D., J. A. Weinman, and J. L. Schols, 1998: The use of digital warping of microwave integrated water vapor imagery to improve forecasts of marine extratropical cyclones. Mon. Wea. Rev., 126, 1469-1496.
5. Alexander, G. D., J. A. Weinman, and J. L. Schols, V. M. Karyampudi, W. S. Olson, and A. C. L. Lee, 1999: The effect of assimilating rain rates derived from satellites and lightning on forecasts of the 1993 superstorm. Mon. Wea. Rev., 127, 1433-1457.
6. Atger, F., 2001: Verification of intense precipitation forecasts from single models and ensemble prediction systems. Nonlinear Processes Geophys., 8, 401-417.
7. Baddeley, A. J., 1992a: Errors in binary images and an Lp version of the Hausdorff metric. Nieuw Arch. Wiskunde, 10, 157-183.
8. Baddeley, A. J., 1992b: An error metric for binary images. Robust Computer Vision: Quality of Vision Algorithms, W. Forstner and S. Ruwiedel, Eds., Wichmann, 59-78.
9. Baddeley, A. J., and R. Turner, 2005: Spatstat: An R package for analyzing spatial point patterns. J. Stat. Software, 12, 1-42.
10. Baldwin, M. E., and K. L. Elmore, 2005: Objective verification of high-resolution WRF forecasts during 2005 NSSL/SPC Spring Program. Preprints, 21st Conf. on Weather Analysis and Forecasting/17th Conf. on Numerical Weather Prediction, Washington, DC, Amer. Meteor. Soc., 11B4. [Available online at https://ams.confex.com/ams/pdfpapers/95172.pdf.].
11. Benjamini, Y., and Y. Hochberg, 1995: Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. Roy. Stat. Soc., 57B, 289-300.
12. Benjamini, Y., and Y. Hochberg, and R. Heller, 2007: False discovery rates for spatial signals. J. Amer. Stat. Assoc., 102, 1272-1281.
13. Borgefors, G., 1986: Distance transformations in digital images. Comput. Vis. Graph. Image Process., 34, 344-371.
14. Briggs, W. M., and R. A. Levine, 1997: Wavelets and field forecast verification. Mon. Wea. Rev., 125, 1329-1341.
15. Brill, K. F., 2009: A general analytic method for assessing sensitivity to bias of performance measures for dichotomous forecasts. Wea. Forecasting, 24, 307-318.
16. Brill, K. F., and F. Mesinger, 2009: Applying a general analytic method for assessing bias sensitivity to bias-adjusted threat and equitable threat scores. Wea. Forecasting, 24, 1748-1754.
17. Brooks, H. E., M. Kay, and J. A. Hart, 1998: Objective limits on forecasting skill of rare events. Preprints, 19th Conf. on Severe Local Storms, Minneapolis, MN, Amer. Meteor. Soc., 552-555.
18. Brown, B. G., E. Gilleland, and E. E. Ebert, 2011: Forecasts of spatial fields. Forecast Verification: A Practitioner's Guide in Atmospheric Science, 2nd ed. I. T. Jolliffe and D. B. Stephenson, Eds., John Wiley & Sons, Ltd., 95-117.
19. Casati, B., 2010: New developments of the intensity-scale technique within the Spatial Verification Methods Intercomparison Project. Wea. Forecasting, 25, 113-143.
20. Casati, B., G. Ross, and D. B. Stephenson, 2004: A new intensity-scale approach for the verification of spatial precipitation forecasts. Meteor. Appl., 11, 141-154.
21. Casati, B., G. Ross, and D. B. Stephenson, and Coauthors, 2008: Forecast verification: Current status and future directions. Meteor. Appl., 15, 3-18.
22. Cressie, N. A. C., 1993: Statistics for Spatial Data. rev. ed. Wiley, 900 pp.
23. Davis, C., B. Brown, and R. Bullock, 2006a: Object-based verification of precipitation forecasts. Part I: Methodology and application to mesoscale rain areas. Mon. Wea. Rev., 134, 1772-1784.
24. Davis, C., B. Brown, and R. Bullock, 2006b: Object-based verification of precipitation forecasts. Part II: Application to convective rain systems. Mon. Wea. Rev., 134, 1785-1795.
25. Davis, C., B. Brown, and R. Bullock, and J. Halley-Gotway, 2009: The Method for Object-Based Diagnostic Evaluation (MODE) applied to numerical forecasts from the 2005 NSSL/SPC Spring Program. Wea. Forecasting, 24, 1252-1267.
26. Demaria, E. M. C., D. A. Rodriguez, E. E. Ebert, P. Salio, F. Su, and J. B. Valdes, 2011: Evaluation of mesoscale convective systems in South America using multiple satellite products and an object-based approach. J. Geophys. Res., 116, D08103, doi:10.1029/2010JD015157.
27. De Sales, F., and Y. Xue, 2011: Assessing the dynamic-downscaling ability over South America using the intensity-scale verification technique. Int. J. Climatol., 31, 1205-1221.
28. Diebold, F. X., and R. S. Mariano, 1995: Comparing predictive accuracy. J. Bus. Econ. Stat., 13, 253-263.
29. Ebert, E. E., 2008: Fuzzy verification of high-resolution gridded forecasts: A review and proposed framework. Meteor. Appl., 15, 51-64.
30. Ebert, E. E., 2009: Neighborhood verification: A strategy for rewarding close forecasts. Wea. Forecasting, 24, 1498-1510.
31. Ebert, E. E., and J. L. McBride, 2000: Verification of precipitation in weather systems: Determination of systematic errors. J. Hydrol., 239, 179-202.
32. Ebert, E. E., and J. L. McBride, and W. A. Gallus, 2009: Toward better understanding of the Contiguous Rain Area (CRA) method for spatial forecast verification. Wea. Forecasting, 24, 1401-1415.
33. Elmore, K. L., M. E. Baldwin, and D. M. Schultz, 2006: Field significance revisited: Spatial bias errors in forecasts as applied to the eta model. Mon. Wea. Rev., 134, 519-531.
34. Furrer, R., D. W. Nychka, and S. Sain, cited 2012: Fields: Tools for spatial data. R package version 6.6.3. [Available online at http://CRAN.R-project.org/package=fields.].
35. Gallus, W. A., 2010: Application of object-based verification techniques to ensemble precipitation forecasts. Wea. Forecasting, 25, 144-158.
36. Gelfand, A., A. Schmidt, S. Banerjee, and C. Sirmans, 2004: Nonstationary multivariate process modeling through spatially varying coregionalization. TEST, 13, 263-312.
37. Germann, U., and I. Zawadzki, 2004: Scale dependence of the predictability of precipitation from continental radar images. Part II: Probability forecasts. J. Appl. Meteor., 43, 7489.
38. Gilleland, E., 2010: Confidence intervals for forecast verification. NCAR Tech. Rep. NCAR/TN-4791STR, 71 pp.
39. Gilleland, E., 2011: Spatial forecast verification: Baddeley's delta metric applied to the ICP test cases. Wea. Forecasting, 26, 409-415.
40. Gilleland, E., cited 2012: SpatialVx: Spatial forecast verification. R package version 0.1-2. [Available online at http://CRAN.R-project.org/package=SpatialVx.].
41. Gilleland, E., T. C. M. Lee, J. Halley Gotway, R. G. Bullock, and B. G. Brown, 2008: Computationally efficient spatial forecast verification using Baddeley's delta image metric. Mon. Wea. Rev., 136, 1747-1757.
42. Gilleland, E., T. C. M. Lee, J. Halley Gotway, R. G. Bullock, and B. G. Brown, D. Ahijevych, B. G. Brown, B. Casati, and E. E. Ebert, 2009: Intercomparison of spatial forecast verification methods. Wea. Forecasting, 24, 1416-1430.
43. Gilleland, E., T. C. M. Lee, J. Halley Gotway, R. G. Bullock, and B. G. Brown, D. Ahijevych, B. G. Brown, B. Casati, and E. E. Ebert, and E. E. Ebert, 2010a: Verifying forecasts spatially. Bull. Amer. Meteor. Soc., 91, 1365-1373.
44. Gilleland, E., T. C. M. Lee, J. Halley Gotway, R. G. Bullock, and B. G. Brown, D. Ahijevych, B. G. Brown, B. Casati, and E. E. Ebert, and E. E. Ebert, J. Lindström, and F. Lindgren, 2010b: Analyzing the image warp forecast verification method on precipitation fields from the ICP. Wea. Forecasting, 25, 1249-1262.
45. Gilleland, E., T. C. M. Lee, J. Halley Gotway, R. G. Bullock, and B. G. Brown, D. Ahijevych, B. G. Brown, B. Casati, and E. E. Ebert, and E. E. Ebert, J. Lindström, and F. Lindgren, and Coauthors, 2010c: Spatial forecast verification: Image warping. NCAR Tech. Rep. NCAR/TN-4821STR, 23 pp.
46. Gorgas, T., and M. Dorninger, 2012: Quantifying verification uncertainty by reference data variation. Meteor. Z., 21, 259-277, doi:10.1127/0941-2948/2012/0325.
47. Guan, Y., M. Sherman, and J. A. Calvin, 2006: Assessing isotropy for spatial point processes. Biometrics, 62, 119-125.
48. Hamill, T. M., 1999: Hypothesis tests for evaluating numerical precipitation forecasts. Wea. Forecasting, 14, 155-167.
49. Harris, D., E. Foufoula-Georgiou, K. K. Droegemeier, and J. J. Levit, 2001: Multiscale statistical properties of a high-resolution precipitation forecast. J. Hydrometeor., 2, 406-418.
50. Hartung, D. C., J. A. Otkin, R. A. Petersen, D. D. Turner, and W. F. Feltz, 2011: Assimilation of surface-based boundary layer profiler observations during a cool-season weather event using an observing system simulation experiment. Part II: Forecast assessment. Mon. Wea. Rev., 139, 2327-2346.
51. Hering, A. S., and M. G. Genton, 2011: Comparing spatial predictions. Technometrics, 53, 414-425.
52. Hoffman, R. N., and C. Grassotti, 1996: A technique for assimilating SSM/I observations of marine atmospheric storms: Tests with ECMWF analyses. J. Appl. Meteor., 35, 1177-1188.
53. Hoffman, R. N., and C. Grassotti, Z. Liu, J.-F. Louis, and C. Grassoti, 1995: Distortion representation of forecast errors. Mon. Wea. Rev., 123, 2758-2770.
54. Johnson, A., X. Wang, F. Kong, and M. Xue, 2011: Hierarchical cluster analysis of a convection-allowing ensemble during the Hazardous Weather Testbed 2009 Spring Experiment. Part I: Development of the object-oriented cluster analysis method for precipitation fields. Mon. Wea. Rev., 139, 3673-3693.
55. Jolliffe, I. T., 2007: Uncertainty and inference for verification measures. Wea. Forecasting, 22, 637-650.
56. Kain, J., and Coauthors, 2008: Some practical considerations regarding horizontal resolution in the first generation of operational convection-allowing NWP. Wea. Forecasting, 23, 931-952.
57. Keil, C., and G. C. Craig, 2007: A displacement-based error measure applied in a regional ensemble forecasting system. Mon. Wea. Rev., 135, 3248-3259.
58. Keil, C., and G. C. Craig, 2009: A displacement and amplitude score employing an optical flow technique. Wea. Forecasting, 24, 12971308.
59. Lack, S. A., G. L. Limpert, and N. I. Fox, 2010: An object-oriented multiscale verification scheme. Wea. Forecasting, 25, 79-92.
60. Lakshmanan, V., and J. S. Kain, 2010: A Gaussian mixture model approach to forecast verification. Wea. Forecasting, 25, 908-920.
61. Liu, Y., J. Brown, J. Demargne, and D.-J. Seo, 2011: A waveletbased approach to assessing timing errors in hydrologic predictions. J. Hydrol., 397, 210-224.
62. Livezey, R. E., and W. Y. Chen, 1983: Statistical field significance and its determination by Monte Carlo techniques. Mon. Wea. Rev., 111, 46-59.
63. Marsigli, C., F. Boccanera, A. Montani, and T. Paccagnella, 2005: The COSMO-LEPS mesoscale ensemble system: Validation of the methodology and verification. Nonlinear Processes Geophys., 12, 527-536.
64. Marzban, C., and S. Sandgathe, 2006: Cluster analysis for verification of precipitation fields. Wea. Forecasting, 21, 824-838.
65. Marzban, C., and S. Sandgathe, 2008: Cluster analysis for object-oriented verification of fields: A variation. Mon. Wea. Rev., 136, 1013-1025.
66. Marzban, C., and S. Sandgathe, 2009: Verification with variograms. Wea. Forecasting, 24, 1102-1120.
67. Marzban, C., and S. Sandgathe, 2010: Optical flow for verification. Wea. Forecasting, 25, 1479-1494.
68. Marzban, C., and S. Sandgathe, and H. Lyons, 2008: An object-oriented verification of three NWP model formulations via cluster analysis: An objective and a subjective analysis. Mon. Wea. Rev., 136, 33923407.
69. Marzban, C., and S. Sandgathe, and H. Lyons, and N. Lederer, 2009: Three spatial verification techniques: Cluster analysis, variogram, and optical flow. Wea. Forecasting, 24, 1457-1471.
70. Mass, C. F., D. Ovens, K. Westrick, and B. A. Colle, 2002: Does increasing horizontal resolution produce more skillful forecasts? Bull. Amer. Meteor. Soc., 83, 407-430.
71. Mesinger, F., 2008: Bias adjusted precipitation threat scores. Adv. Geosci., 16, 137-142.
72. Micheas, A. C., N. I. Fox, S. A. Lack, and C. K. Wikle, 2007: Cell identification and verification of QPF ensembles using shape analysis techniques. J. Hydrol., 343, 105-116.
73. Mittermaier, M. P., 2006: Using an intensity-scale technique to assess the added benefit of high-resolution model precipitation forecasts. Atmos. Sci. Lett., 7, 36-42.
74. Mittermaier, M. P., and N. Roberts, 2010: Intercomparison of spatial forecast verification methods: Identifying skillful spatial scales using the Fractions Skill Score. Wea. Forecasting, 25, 343-354.
75. Mittermaier, M. P., and N. Roberts, and S. A. Thompson, 2012: A long-term assessment of precipitation forecast skill using the Fractions Skill Score. Meteor. Appl., doi:10.1002/met.296, in press.
76. Nachamkin, J. E., 2004: Mesoscale verification using meteorological composites. Mon. Wea. Rev., 132, 941-955.
77. Nachamkin, J. E., 2009: Application of the composite method to the spatial forecast verification methods intercomparison dataset. Wea. Forecasting, 24, 1390-1400.
78. Nachamkin, J. E., S. Chen, and J. Schmidt, 2005: Evaluation of heavy precipitation forecasts using composite-based methods: A distributionsoriented approach. Mon. Wea. Rev., 133, 2163-2177.
79. Nan, Z., S. Wang, X. Liang, T. Adams, W. Teng, and Y. Liang, 2010: Analysis of spatial similarities between NEXRAD and NLDAS precipitation data products. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., 3, 371-385.
80. Nehrkorn, T., R. N. Hoffman, C. Grassotti, and J.-F. Louis, 2003: Feature calibration and alignment to represent model forecast errors: Empirical regularization. Quart. J. Roy. Meteor. Soc., 129, 195-218.
81. R Development Core Team, 2012: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, 409 pp.
82. Rezacova, D., Z. Sokol, and P. Pesice, 2007: A radar-based verification of precipitation forecast for local convective storms. Atmos. Res., 83, 211-224.
83. Roberts, N. M., and H. W. Lean, 2008: Scale-selective verification of rainfall accumulations from high-resolution forecasts of convective events. Mon. Wea. Rev., 136, 78-97.
84. Rosenfeld, A., and J. L. Pfaltz, 1966: Sequential operations in digital picture processing. J. Assoc. Comput. Mach., 13, 471494.
85. Rossa, A. M., P. Nurmi, and E. E. Ebert, 2008: Overview of methods for the verification of quantitative precipitation forecasts. Precipitation: Advances in Measurement, Estimation and Prediction, S. C. Michaelides, Ed., Springer, 418-450.
86. Schaffer, C. J., W. A. Gallus, and M. Segal, 2011: Improving probabilistic ensemble forecasts of convection through the application of QPF-POP relationships. Wea. Forecasting, 26, 319-336.
87. Schwedler, B. R. J., and M. E. Baldwin, 2011: Diagnosing the sensitivity of binary image measures to bias, location, and event frequency within a forecast verification framework. Wea. Forecasting, 26, 1032-1044.
88. Shen, X., H.-C. Huang, and N. Cressie, 2002: Nonparametric hypothesis testing for a spatial signal. J. Amer. Stat. Assoc., 97, 1122-1140.
89. Skamarock, W. C., J. B. Klemp, J. Dudhia, D. O. Gill, M. Barker, W. Wang, and J. G. Powers, 2005: A description of the Advanced Research WRF version 2. NCAR Tech. Rep. NCAR/TN-4681STR, 88 pp.
90. Sobash, R. A., J. S. Kain, D. R. Bright, A. R. Dean, M. C. Coniglio, and S. J. Weiss, 2011: Probabilistic forecast guidance for severe thunderstorms based on the identification of extreme phenomena in convection-allowing model forecasts. Wea. Forecasting, 26, 714-728.
91. Tafferner, A., C. Forster, M. Hagen, C. Keil, T. Zinner, and H. Volkert, 2008: Development and propagation of severe thunderstorms in the Upper Danube catchment area: Towards an integrated nowcasting and forecasting system using realtime data and high-resolution simulations. Meteor. Atmos. Phys., 101, 211-227.
92. Theis, S. E., A. Hense, and U. Damrath, 2005: Probabilistic precipitation forecasts from a deterministic model: A pragmatic approach. Meteor. Appl., 12, 257-268.
93. Ventura, V., C. J. Paciorek, and J. S. Risbey, 2004: Controlling the proportion of falsely rejected hypotheses when conducting multiple tests with climatological data. J. Climate, 17, 4343-4356.
94. Venugopal, V., S. Basu, and E. Foufoula-Georgiou, 2005: A new metric for comparing precipitation patterns with an applica- tion to ensemble forecasts. J. Geophys. Res., 110, D08111, doi:10.1029/2004JD005395.
95. Wapler, K., M. Goeber, and S. Trepte, 2012: Comparative verifi- cation of different nowcasting systems to support optimisa- tion of thunderstorm warnings. Adv. Sci. Res., 8, 121-127, doi:10.5194/asr-8-121-2012.
96. Wernli, H., M. Paulat, M. Hagen, and C. Frei, 2008: SAL-A novel quality measure for the verification of quantitative pre- cipitation forecasts. Mon. Wea. Rev., 136, 4470-4487.
97. Wernli, H., M. Paulat, M. Hagen, and C. Frei, C. Hofmann, and M. Zimmer, 2009: Spatial Forecast Verifi- cation Methods Intercomparison Project: Application of the SAL technique. Wea. Forecasting, 24, 1472-1484.
98. Weusthoff, T., F. Ament, M. Arpagaus, and M. W. Rotach, 2010: Assessing the benefits of convection-permitting models by neighborhood verification: Examples from MAP D-PHASE. Mon. Wea. Rev., 138, 3418-3433.
99. Weygandt, S. S., A. F. Loughe, S. G. Benjamin, and J. L. Mahoney, 2004: Scale sensitivities in model precipitation skill scores during IHOP. Preprints, 22nd Conf. on Severe Local Storms, Hyannis, MA, Amer. Meteor. Soc., 16A.8. [Available online at https://ams.confex.com/ams/pdfpapers/81986.pdf.].
100. Wilks, D. S., 1997: Resampling hypothesis tests for autocorrelated fields. J. Climate, 10, 65-82.
101. Willmott, C. J., and K. Matsuura, 2005: Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance. Climate Res., 30, 79-82.
102. Willmott, C. J., and K. Matsuura, 2006: On the use of dimensioned measures of error to evaluate the performance of spatial interpolators. Int. J. Geogr. Inf. Sci., 20, 89-102.
103. Willmott, C. J., and K. Matsuura, and S. M. Robeson, 2009: Ambiguities inherent in sums- of-squares-based error statistics. Atmos. Environ., 43, 749-752. Yates, E., S. Anquetin, V. Ducrocq, J.-D. Creutin, D. Ricard, and K. Chancibault, 2006: Point and areal validation of forecast precipitation fields. Meteor. Appl., 13, 1-20.
104. Yates, E., S. Anquetin, V. Ducrocq, J.-D. Creutin, D. Ricard, and K. Chancibault, 2006: Point and areal validation of forecast precipitation fields. Meteor. Appl., 13, 1-20.
105. Zepeda-Arce, J., E. Foufoula-Georgiou, and K. K. Droegemeier, 2000: Space-time rainfall organization and its role in validating quantitative precipitation forecasts. J. Geophys. Res., 105 (D8), 10 129-10 146.
106. Zhu, M., V. Lakshmanan, P. Zhang, Y. Hong, K. Cheng, and S. Chen, 2011: Spatial verification using a true metric. Atmos. Res., 102, 408-419.
107. Zimmer, M., and H. Wernli, 2011: Verification of quantitative precipitation forecasts on short time-scales: A fuzzy approach to handle timing errors with SAL. Meteor. Z., 20, 95-105.