Recommendations on statistics and benchmarks to assess photochemical model performance

Journal of the Air and Waste Management Association

Volumn 67, Issue 5, 2017, Pages 582-598

  Emery, Christopher ^a   Liu, Zhen ^a   Russell, Armistead G ^b   Odman, M Talat ^b   Yarwood, Greg ^a   Kumar, Naresh ^c  

^a ENVIRON INTERNATIONAL CORPORATION   (United States)

^b Georgia Institute of Technology   (United States)

^c ELECTRIC POWER RESEARCH INSTITUTE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AIR QUALITY; DIGITAL STORAGE; OZONE; QUALITY CONTROL; STATISTICS; UNCERTAINTY ANALYSIS;

GRAPHICAL METHODS; HISTORICAL MODELS; MODEL PERFORMANCE; MODEL UNCERTAINTIES; PARTICULATE MATTER; PHOTOCHEMICAL MODELING; PHOTOCHEMICAL MODELS; POLICY ASSESSMENT;

BENCHMARKING;

OZONE; AIR POLLUTANT; PARTICULATE MATTER;

AIR POLLUTION; AIR QUALITY; ARTICLE; BENCHMARKING; CONCENTRATION (PARAMETERS); ENVIRONMENTAL POLICY; EVALUATION STUDY; MODEL; NORTH AMERICAN; PARTICULATE MATTER; PERFORMANCE; PHOTOCHEMISTRY; PRIORITY JOURNAL; QUANTITATIVE ANALYSIS; SEASONAL VARIATION; STATISTICAL ANALYSIS; UNCERTAINTY; WASTE MANAGEMENT; AIR POLLUTANT; ANALYSIS; CHEMICAL MODEL; ENVIRONMENTAL MONITORING; PROCEDURES; STANDARDS; STATISTICS AND NUMERICAL DATA; UNITED STATES;

AIR POLLUTANTS; AIR POLLUTION; BENCHMARKING; DATA INTERPRETATION, STATISTICAL; ENVIRONMENTAL MONITORING; MODELS, CHEMICAL; OZONE; PARTICULATE MATTER; PHOTOCHEMICAL PROCESSES; UNITED STATES;

EID: 85014542937     PISSN: 10962247     EISSN: 21622906     Source Type: Journal    
DOI: 10.1080/10962247.2016.1265027     Document Type: Article

References (72)

1. Baker, K., and P., Scheff. 2007. Photochemical model performance for PM2.5 sulfate, nitrate, ammonium, and precursor species SO2, HNO3, and NH3 at background monitor locations in the central and eastern United States. Atmos. Environ. 41:6185–95. doi:10.1016/j.atmosenv.2007.04.006
2. Bessagnet, B., A., Hodzic, R., Vautard, M., Beekmann, S., Cheinet, C., Honoré, C., Liousse, and L., Rouil. 2004. Aerosol modeling with CHIMERE:Preliminary evaluation at the continental scale. Atmos. Environ. 38:2803–17. doi:10.1016/j.atmosenv.2004.02.034
3. Boylan, J.W., and A.G., Russell. 2006. PM and light extinction model performance metrics, goals, and criteria for three-dimensional air quality models. Atmos. Environ. 40:4946–59. doi:10.1016/j.atmosenv.2005.09.087
4. Brandt, J., J.H., Christensen, L.M., Frohn, C., Geels, K.M., Hansen, G.B., Hedegaard, M., Hvidberg, and C.A., Skjøth. 2007. THOR—An operational and integrated model system for air pollution forecasting and management from regional to local scale. Proceedings of the 2nd ACCENT Symposium, Urbino (Italy), July 23–27, 2007.
5. Byun, D.W., S.T., Kim, and S.B., Kim. 2007. Evaluation of air quality models for the simulation of a high ozone episode in the Houston metropolitan area. Atmos. Environ. 41:837–53. doi:10.1016/j.atmosenv.2006.08.038
6. Dennis, R., T., Fox, M., Fuentes, A., Gilliland, S., Hanna, C., Hogrefe, J., Irwin, S.T., Rao, R, Scheffe, K., Schere, D.A., Steyn, and A., Venkatram. 2010. A framework for evaluating regional-scale numerical photochemical modeling systems. J. Environ. Fluid Mech. 10:471–89. doi:10.1007/s10652-009-9163-2.
7. Downey, N., C., Emery, J., Jung, T., Sakulyanontvittaya, L., Hebert, D., Blewitt, and G., Yarwood. 2015. Emission reductions and urban ozone responses under more stringent US standards. Atmos. Environ. 101:209–16. doi:10.1016/j.atmosenv.2014.11.018
8. Dunker, A.M., 1981. Efficient calculation of sensitivity coefficients for complex atmospheric models. Atmos. Environ. (Part A) 15:1155–61. doi:10.1016/0004-6981(81)90305-X
9. Dunker, A.M., G., Yarwood, J.P., Ortmann, and G.M., Wilson. 2002. Comparison of source apportionment and source sensitivity of ozone in a three-dimensional air quality model. Environ. Sci. Technol. 36:2953–64. doi:10.1021/es011418f
10. Eder, B., D., Kang, R., Mathur, J., Pleim, S., Yu, T., Otte, and G., Pouliot. 2009. A performance evaluation of the National Air Quality Forecast Capability for the summer of 2007. Atmos. Environ. 43:2312–20. doi:10.1016/j.atmosenv.2009.01.033
11. Emery, C., E., Tai, and G., Yarwood. 2001. Enhanced Meteorological Modeling and Performance Evaluation for Two Texas Ozone Episodes. Report to the Texas Natural Resources Conservation Commission, prepared by ENVIRON International Corp, Novato, CA. http://www.tceq.state.tx.us/assets/public/implementation/air/am/contracts/reports/mm/EnhancedMetModelingAndPerformanceEvaluation.pdf.
12. Emery, C., J., Jung, N., Downey, J., Johnson, M., Jimenez, G., Yarwood, and R., Morris. 2012. Regional and global modeling estimates of policy relevant background ozone over the United States. Atmos. Environ. 47:206–217. doi:10.1016/j.atmosenv.2011.11.012
13. Foley, K.M., S.J., Roselle, K.W., Appel, P.V., Bhave, J.E., Pleim, T.L., Otte, R., Mathur, G., Sarwar, J.O., Young, R.C., Gilliam, et al. 2010. Incremental testing of the Community Multiscale Air Quality (CMAQ) modeling system version 4.7. Geoscientific Model Dev. 3:205–26. doi:10.5194/gmd-3-205-2010
14. Foley, K.M., C., Hogrefe, G., Pouliot, N., Possiel, S.J., Roselle, H., Simon, and B., Timin. 2015. Dynamic evaluation of CMAQ part I:Separating the effects of changing emissions and changing meteorology on ozone levels between 2002 and 2005 in the eastern US. Atmos. Environ. 103:247–55. doi:10.1016/j.atmosenv.2014.12.038
15. Gaydos, T.M., R., Pinder, B., Koo, K.M., Fahey, G., Yarwood, and S.N., Pandis. 2007. Development and application of a three-dimensional aerosol chemical transport model, PMCAMx. Atmos. Environ. 41:2594–611. doi:10.1016/j.atmosenv.2006.11.034
16. Grell, G.A., S., Emeis, W.R., Stockwell, T., Schoenemeyer, R., Forkel, J., Michalakes, R., Knoche, and W., Seidl. 2000. Application of a multiscale, coupled MM5/chemistry model to the complex terrain of the VOTALP valley campaign. Atmos. Environ. 34:1435–53. doi:10.1016/S1352-2310(99)00402-1
17. Grell, G.A., S.E., Peckham, R., Schmitz, S.A., McKeen, G., Frost, W.C., Skamarock, and B., Eder. 2005. Fully coupled “online” chemistry within the WRF model. Atmos. Environ. 39:6957–75. doi:10.1016/j.atmosenv.2005.04.027
18. Hakami, A., M.T., Odman, and A.G., Russell. 2003. High-order, direct sensitivity analysis of multidimensional air quality models. Environ. Sci. Technol. 37:2442–52. doi:10.1021/es020677h
19. Harvard, University. 2015. GEOS-Chem Online User’s Guide. Atmospheric Chemistry Modeling Group, School of Engineering and Applied Sciences, Harvard University, Cambridge, MA (http://acmg.seas.harvard.edu/geos/doc/man/index.html).
20. Hogrefe, C., W., Hao, K., Civerolo, J.Y., Ku, G., Sistla, R.S., Gaza, L., Sedefian, K., Schere, A., Gilliland, A., and R., Mathur. 2007. Daily simulation of ozone and fine particulates over New York State:Findings and challenges. J. Appl. Meteorol. Climatol. 46:961–79. doi:10.1175/JAM2520.1
21. Hogrefe, C., K.L., Civerolo, W., Hao, J.Y., Ku, E.E., Zalewsky, and G., Sistla. 2008. Rethinking the assessment of photochemical modeling systems in air quality planning applications. J. Air Waste Manage. Assoc. 58:1086–99. doi:10.3155/1047-3289.58.8.1086
22. Hogrefe, C., W., Hao, E.E., Zalewsky, J.Y., Ku, B., Lynn, C., Rosenzweig, M.G., Schultz, S., Rast, M.J., Newchurch, L., Wang, P.L., Kinney, and G., Sistla. 2011. An analysis of long-term regional-scale ozone simulations over the northeastern United States:Variability and trends. Atmos. Chem. Phys. 11:567–82. doi:10.5194/acp-11-567-2011
23. Hou, X., M.J., Strickland, and K.J., Liao. 2015. Contributions of regional air pollutant emissions to ozone and fine particulate matter-related mortalities in eastern U.S. urban areas. Environ. Res. 137:475–84. doi:10.1016/j.envres.2014.10.038
24. Im, U., R., Bianconi, E., Solazzo, I., Kioutsioukis, A., Badia, A., Balzarini, R., Baro, R., Bellasio, D., Brunner, C., Chemel, G., Curci, et al. 2015. Evaluation of operational on-line-coupled regional air quality models over Europe and North America in the context of AQMEII Phase 2. Part I:Ozone. Atmos. Environ. 115:404–20. doi:10.1016/j.atmosenv.2014.08.072
25. Jin, L., N.J., Brown, R.A., Harley, J.W., Bao, S.A., Michelson, and J.M., Wilczak. 2010. Seasonal versus episodic performance evaluation for an Eulerian photochemical air quality model. J. Geophys. Res. Atmos. 115:D09302. doi:10.1029/2009JD012680
26. Kang, D., R., Mathur, and S.T., Rao. 2010. Real-time bias-adjusted O3 and PM2.5 air quality index forecasts and their performance evaluations over the continental United States. Atmos. Environ. 44:2203–12. doi:10.1016/j.atmosenv.2010.03.017
27. Karamchandani, P., K., Vijayaraghavan, S.Y., Chen, C., Seigneur, and E.S., Edgerton, 2006. Plume-in-grid modeling for particulate matter. Atmos. Environ. 40:7280–97. doi:10.1016/j.atmosenv.2006.06.033
28. Kim, Y., J.S., Fu, and T.L, Miller. 2010. Improving ozone modeling in complex terrain at a fine grid resolution—Part II:influence of schemes in MM5 on daily maximum 8-h ozone concentrations and RRFs (Relative Reduction Factors) for SIPs in the non-attainment areas. Atmos. Environ. 44:2116–24. doi:10.1016/j.atmosenv.2010.02.038
29. Koo, B., N., Kumar, E., Knipping, U., Nopmongcol, T., Sakulyanontvittaya, M.T., Odman, A.G., Russell, and G., Yarwood. 2015. Chemical transport model consistency in simulating regulatory outcomes and the relationship to model performance. Atmos. Environ. 116:159–71. doi:10.1016/j.atmosenv.2015.06.036
30. Lee, D., D.W., Byun, H., Kim, F., Ngan, S., Kim, C., Lee, and C., Cho. 2011. Improved CMAQ predictions of particulate matter utilizing the satellite-derived aerosol optical depth. Atmos. Environ. 45:3730–41. doi:10.1016/j.atmosenv.2011.04.018
31. Liao, K.-J., X., Hou, and D. R., Baker. 2014. Impacts of interstate transport of pollutants on high ozone events over the Mid-Atlantic United States. Atmos. Environ. 84:100–12. doi:10.1016/j.atmosenv.2013.10.062
32. Liu, X.-H., Y., Zhang, K.M., Olsen, W.-X, Wang, B.A., Do, and G.M., Bridgers. 2010. Responses of future air quality to emission controls over North Carolina, Part I:Model evaluation for current-year simulations. Atmos. Environ. 44:2443–56. doi:10.1016/j.atmosenv.2010.04.002
33. Makar, P.A., W., Gong, C., Mooney, J., Zhang, D., Davignon, M., Samaali, M.D., Moran, H., He, D.W., Tarasick, D., Sills, and J., Chen. 2010. Dynamic adjustment of climatological ozone boundary conditions for air-quality forecasts. Atmos. Chem. Phys. 10:8997–9015. doi:10.5194/acp-10-8997-2010
34. Makar, P.A., W., Gong, C., Hogrefe, Y., Zhang, G., Curci, R., Zabkar, J., Milbrandt, U., Im, S., Galmarini, A., Balzarini, et al. 2015. Feedbacks between air pollution and weather, Part 1:Effects on chemistry. Atmos. Environ. 115:499–526. doi:10.1016/j.atmosenv.2014.10.021
35. Marmur, A., W., Liu, Y., Wang, A.G., Russell, and E.S., Edgerton. 2009. Evaluation of model simulated atmospheric constituents with observations in the factor projected space:CMAQ simulations of SEARCH measurements. Atmos. Environ. 43:1839–49. doi:10.1016/j.atmosenv.2008.12.027
36. Menut, L., R., Vautard, M., Beekmann, and C., Honore. 2000. Sensitivity of photochemical pollution using the adjoint of a simplified chemistry-transport model. J. Geophys. Res. Atmos. 105:15379–402. doi:10.1029/1999JD900953
37. National Research, Council. 2007. Models in Environmental Regulatory Decision Making. Washington, DC:National Research Council of the National Academies. doi:10.17226/11972.
38. Odman, T., and Z., Adelman. 2014. Emissions and Air Quality Modeling for SEMAP. Final Report to Southeastern States Air Resource Managers, Inc., December 31, 2014, Georgia Tech, http://semap.ce.gatech.edu/sites/default/files/files/SEMAP-Revised-Final-Report_Final.pdf.
39. Pun, B.K., C., Seigneur, K., Vijayaraghavan, S.Y., Wu, S.Y., Chen, E.M., Knipping, and N., Kumar. 2006. Modeling regional haze in the BRAVO study using CMAQMADRID:1. Model evaluation. J. Geophys. Res. Atmos. 111:D06302. doi:10.1029/2004JD005608
40. Queen, A., and Y., Zhang. 2008. Examining the sensitivity of MM5-CMAQ predictions to explicit microphysics schemes and horizontal grid resolutions, Part II—PM concentrations and wet deposition predictions. Atmos. Environ. 42:3856–68. doi:10.1016/j.atmosenv.2007.12.066
41. Ramboll, Environ. 2016. User’s Guide:Comprehensive Air quality Model with extensions, Version 6.30. Ramboll Environ, Novato, CA (www.camx.com).
42. Rodriguez, M.A., M.G., Barna, K.A., Gebhart, J.L., Hand, Z.E., Adelman, B.A., Schichtel, J.L., Collett, Jr., and W.C., Malm. 2011. Modeling the fate of atmospheric reduced nitrogen during the Rocky Mountain Atmospheric Nitrogen and Sulfur Study (RoMANS):Performance evaluation and diagnosis using integrated processes rate analysis. Atmos. Environ. 45:223–34. doi:10.1016/j.atmosenv.2010.09.011
43. Sakulyanontvittaya, T., A., Guenther, D., Helmig, J., Milford, and C., Wiedinmyer. 2008. Secondary organic aerosol from sesquiterpene and monoterpene emissions in the United States. Environ. Sci. Technol. 42:8784–90. doi:10.1021/es800817r
44. Simon, H., K.R., Baker, and S., Phillips. 2012. Compilation and interpretation of photochemical model performance statistics published between 2006 and 2012. Atmos. Environ. 61:124–39. doi:10.1016/j.atmosenv.2012.07.012
45. Smyth, S.C., W.M., Jiang, D.Z., Yin, H., Roth, and T., Giroux. 2006. Evaluation of CMAQ O3 and PM2.5 performance using Pacific 2001 measurement data. Atmos. Environ. 40:2735–49. doi:10.1016/j.atmosenv.2005.10.068
46. Smyth, S.C., W., Jiang, H., Roth, M.D., Moran, P.A., Makar, F., Yang, V.S., Bouchet, and H., Landry. 2009. A comparative performance evaluation of the AURAMS and CMAQ air-quality modelling systems. Atmos. Environ. 43:1059–70. doi:10.1016/j.atmosenv.2008.11.027
47. Solazzo, E., R., Bianconi, G., Pirovano, V., Matthias, R., Vautard, K.W., Appel, B., Bessagnet, J., Brandt, J.H., Christensen, C., Chemel, et al. 2012. Operational model evaluation for particulate matter in Europe and North America in the context of AQMEII. Atmos. Environ. 53:75–92. doi:10.1016/j.atmosenv.2012.02.045
48. Solazzo, E., and S. Galmarini. 2016. Error apportionment for atmospheric chemistry-transport models–A new approach to model evaluation. Atmos. Chem. Phys. 16:6263–6283 doi:10.5194/acp-16-6263-2016.
49. Spak, S.N., and T., Holloway. 2009. Seasonality of speciated aerosol transport over the Great Lakes region. J. Geophys. Res. Atmos. 114:D08302. doi:10.1029/2008JD010598
50. Stroud, C.A., P.A., Makar, M.D., Moran, W., Gong, S., Gong, J., Zhang, K., Hayden, C., Mihele, J.R., Brook, J.P.D., Abbatt, and J.G., Slowik. 2011. Impact of model grid spacing on regional- and urban- scale air quality predictions of organic aerosol. Atmos. Chem. Phys. 11:3107–18. doi:10.5194/acp-11-3107-2011
51. Tang, W., D.S., Cohan, G.A., Morris, D.W., Byun, and W.T., Luke. 2011. Influence of vertical mixing uncertainties on ozone simulation in CMAQ. Atmos. Environ. 45:2898–909. doi:10.1016/j.atmosenv.2011.01.057
52. Taylor, K.E., 2001. Summarizing multiple aspects of model performance in a single diagram. J. Geophys. Res. 106:7183–92. doi:10.1029/2000JD900719
53. Tesche, T.W., 1988. Accuracy of ozone air-quality models. J. Environ. Eng. ASCE 114:739–52. doi:10.1061/(ASCE)0733-9372(1988)114:4(739)
54. Thunis, P., E., Georgieva, and S., Galmarini. 2011. A procedure for air quality models benchmarking. Prepared by the Joint Research Centre, Ispra, Italy (16 February 2011). http://fairmode.jrc.ec.europa.eu/document/fairmode/WG1/WG2_SG4_benchmarking_V2.pdf
55. U.S. EPA. 1990. User’s Guide for the Urban Airshed Model, Volume I:User’s Manual for UAM (CB-IV). U.S. Environmental Protection Agency, Research Triangle Park, NC (EPA-450/4-90-007A).
56. U.S. EPA. 1991. Guideline for Regulatory Applications of the Urban Airshed Model. U.S. Environmental Protection Agency, Research Triangle Park, NC (EPA-450/4-91-013).
57. 2.5, and Regional Haze. U.S. Environmental Protection Agency, Research Triangle Park, NC (EPA-454/B-07-002).
58. U.S. EPA. 2011a. Policy Assessment for the Review of the Secondary National Ambient Air Quality Standards for Oxides of Nitrogen and Oxides of Sulfur. U.S. Environmental Protection Agency, Research Triangle Park, NC (EPA-452/R-11-005a, February).
59. U.S. EPA. 2011b. The Benefits and Costs of the Clean Air Act from 1990 to 2020. U.S. Environmental Protection Agency, Washington, DC (March).
60. U.S. EPA. 2011c. Policy Assessment for the Review of the Particulate Matter National Ambient Air Quality Standards. U.S. Environmental Protection Agency, Research Triangle Park, NC (EPA 452/R-11-003, April).
61. U.S. EPA. 2011d. Air Quality Modeling, Final Rule Technical Support Document. US Environmental Protection Agency, Research Triangle Park, NC (June).
62. 2.5, and Regional Haze. U.S. Environmental Protection Agency, Research Triangle Park, NC (December).
63. Willmott, C.J., 1982. Some comments on the evaluation of model performance. Bull. Am. Meteorol. Soc. 63:1309–13. doi:10.1175/1520-0477(1982)063<1309:SCOTEO>2.0.CO;2
64. Wong, D.C., J., Pleim, R., Mathur, F., Binkowski, T., Otte, R., Gilliam, G., Pouliot, A., Xiu, J.O., Young, and D., Kang. 2012. WRF-CMAQ two-way coupled system with aerosol feedback:software development and preliminary results. Geoscientific Model Dev. 5:299–312. doi:10.5194/gmd-5-299-2012.
65. Wu, S.-Y., S., Krishnan, and Y., Zhang, V., Aneja. 2008. Modeling atmospheric transport and fate of ammonia in North Carolina—Part I:Evaluation of meteorological and chemical predictions. Atmos. Environ. 42:3419–36. doi:10.1016/j.atmosenv.2007.04.031
66. Yu, S., R., Mathur, D., Kang, K., Schere, B., Eder, and J., Pleim. 2006. Performance and diagnostic evaluation of ozone predictions by the Eta-community multiscale air quality forecast system during the 2002 New England Air Quality Study. J. Air Waste Manage. Assoc. 56:1459–71. doi:10.1080/10473289.2006.10464554
67. Yu, S., R., Mathur, K., Schere, D., Kang, J., Pleim, and T.L., Otte. 2007. A detailed evaluation of the Eta-CMAQ forecast model performance for O3, its related precursors, and meteorological parameters during the 2004 ICARTT study. J. Geophys. Res. Atmos. 112:D12s14. doi:10.1029/2006JD007715
68. Yu, S., R., Mathur, J., Pleim, D., Wong, R., Gilliam, K., Alapaty, C., Zhao, and X., Liu. 2014. Aerosol indirect effect on the grid-scale clouds in the two-way coupled WRF–CMAQ:Model description, development, evaluation and regional analysis. Atmos. Chem. Phys. 14:11247–85. doi:10.5194/acp-14-11247-2014. doi:10.5194/acp-14-11247-2014
69. Zhang, L.M., M.D., Moran, P.A., Makar, J.R., Brook, and S.L., Gong. 2002. Modelling gaseous dry deposition in AURAMS:A unified regional air-quality modelling system. Atmos. Environ. 36:537–60. doi:10.1016/S1352-2310(01)00447-2
70. Zhang, Y., P., Liu, A., Queen, C., Misenis, B., Pun, C., Seigneur, and S.-Y., Wu. 2006. A comprehensive performance evaluation of MM5-CMAQ for the Summer 1999 Southern Oxidants Study episode—Part II:Gas and aerosol predictions. Atmos. Environ. 40:4839–55. doi:10.1016/j.atmosenv.2005.12.048
71. Zhang, Y., J.-P., Huang, D.K., Henze, and J.H., Seinfeld. 2007. Role of isoprene in secondary organic aerosol formation on a regional scale. J. Geophys. Res. Atmos. 112:D20207. doi:10.1029/2007JD008675
72. Zhang, Y., Y., Pan, K., Wang, J.D., Fast, and G.A., Grell. 2010. WRF/Chem-MADRID:Incorporation of an aerosol module into WRF/Chem and its initial application to the TexAQS2000 episode. J. Geophys. Res. Atmos. 115:D18202. doi:10.1029/2009JD013443