A synthesis of literature on evaluation of models for policy applications, with implications for forest carbon accounting

Forest Ecology and Management

Volumn 198, Issue 1-3, 2004, Pages 89-103

  Prisley, Stephen P ^a   Mortimer, Michael J ^a  

^a VIRGINIA POLYTECHNIC INSTITUTE AND STATE UNIVERSITY   (United States)

Author keywords

Forest models; Sensitivity analysis; Validation; Verification

Indexed keywords

CHEMICAL ANALYSIS; DATA ACQUISITION; SYNTHESIS (CHEMICAL); TRANSPARENCY;

FOREST CARBON ACCOUNTING; FOREST MODELING; POLICY APPLICATIONS;

FORESTRY;

CARBON; ECOLOGICAL MODELING; FOREST ECOSYSTEM; MODEL VALIDATION; SENSITIVITY ANALYSIS;

DATA PROCESSING; FORESTRY; QUANTITATIVE ANALYSIS; SYNTHESIS; TRANSPARENCE;

ARENA;

EID: 3543127755     PISSN: 03781127     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.foreco.2004.03.038     Document Type: Article

References (89)

1. Administrative Conference of the United States, 1985. Agency procedures for performing regulatory analysis or rules (Recommendation No. 85-2). 1 Code of Federal Regulations § 305.85-2, Washington, DC.
2. American Society for Testing and Materials (ASTM), 1992. Standard Practice for Evaluating Mathematical Models for the Environmental Fate of Chemicals. Standard E978-92. ASTM, West Conshohocken, Pennsylvania.
3. American Society for Testing and Materials (ASTM), 2000. Standard Guide for Statistical Evaluation of Atmospheric Dispersion Model Performance. Standard D6589-00. ASTM, West Conshohocken, Pennsylvania.
4. American Society for Testing and Materials (ASTM), 2002. Standard Guide for Comparing Ground-Water Flow Model Simulations to Site-Specific Information. Standard D5490-93. ASTM, West Conshohocken, Pennsylvania.
5. 2 exchange and evapotranspiration predicted by nine ecosystem process models: inter-model comparisons and relationships to field measurements. J. Geophys. Res. 106 (D24), 33, 623-633 and 648.
6. Banfield G.E, Bhatti J.S, Jiang H, Apps M.J. Variability in regional scale estimates of carbon stocks in boreal forest ecosystems: results from West-Central Alberta. For. Ecol. Manage. 169:2002;15-27
7. Barton-Aschman Associates, Inc., 1997. Model Validation and Reasonableness Checking Manual. Travel model improvement program. Federal Highway Administration. http://tmip.fhwa.dot.gov/clearinghouse/docs/mvrcm/ (accessed 13 May 2003).
8. Battaglia M, Sands P.J. Process-based forest productivity models and their application in forest management. For. Ecol. Manage. 102:1998;13-32
9. Beck, M.B., Mulkey, L.A., Barnwell, T.O., 1994. Model Validation for Predictive Exposure Assessments. Report to EPA, 52 pp. http://cfpub.epa.gov/ crem/library/whitepaper_1994.pdf.
10. Beck M.B, Ravetz J.R, Mulkey L.A, Barnwell T.O. On the problem of model validation for predictive exposure assessments. Stoch. Hydrol. Hydraul. 11:1997;229-254
11. Beck, M.B., Chen, J., 2000. Assuring the quality of models designed for predictive tasks. Sensitivity Analysis. Wiley, pp. 401-420.
12. Bergh J, Freeman M, Sigurdsson B, Kellomäki S, Laitinen K, Niinistö S, Peltola H, Linder S. Modelling the short-term effects of climate change on the productivity of selected tree species in Nordic countries. For. Ecol. Manage. 183:2003;327-340
13. Birdsey, R.A., Heath, L.S., 2001. Forest inventory data, models, and assumptions for monitoring carbon flux. In: Soil Carbon Sequestration and the Greenhouse Effect. Soil Science Society of America, Madison, WI. SSSA Special Publication no. 57, Chapter 10, pp. 125-135.
14. Brown T.N, Kulasiri D. Validating models of complex, stochastic, biological systems. Ecol. Model. 86:1996;129-134
15. Butler, B.P., Cox, M.G., Forbes, A.B., Harris, P.M., Lord, G.J., 1999. Model validation in the context of metrology: a survey. NPL Report CISE 19/99. National Physical Laboratory, Teddington, Middlesex, UK, 69 pp.
16. Case C.D. Problems in judicial review arising from the use of computer models and other quantitative methodologies in environmental decisionmaking. Boston College Environ. Affairs Law Rev. 10:(4):1982;251-363
17. Caswell, H., 1976. The validation problem. In: Patten, B.C. (Ed.), Systems Analysis and Simulation in Ecology, vol. IV. Academic Press, New York, pp. 313-325.
18. Chemical Manufacturers Association v. EPA, 1994. 28 F.3d 289 (D.C. Cir.).
19. Chen, J., Beck, M.B., 1999. Quality Assurance of Multi-media Model for Predictive Screening Tasks (EPA/600/R-98/106). US EPA, Washington, DC, 63 pp.
20. Chen Y.D, McCutcheon S.C, Norton D.J, Nutter W.L. Stream temperature simulation of forested riparian areas: model application. J. Environ. Eng. 124:(4):1998;316-328
21. Crawford-Brown D. Scientific models of human health risk analysis in legal and policy decisions. Law Contemp. Probl. 64:(4):2001;63-81
22. Díaz-Balteiro L, Romero C. Forest management optimisation models when carbon captured is considered: a goal programming approach. For. Ecol. Manage. 174:2003;447-457
23. Economist, 2002. The Chicago Climate Exchange: Trading Hot Air.
24. Fienberg S.E, Krislov S.H, Straf M.I. Understanding and evaluating statistical evidence in litigation. Jurimetrics J. 36:1995;1-28
25. Gentil S, Blake G. Validation of complex ecosystem models. Ecol. Model. 14:1981;21-38
26. Grossman S.J. Science and scientific evidence. IDEA: The J. Law Technol. 32:1992;179-202
27. Grossman S.J, Gagne C.K. Science and scientific evidence II. Connecticut Law Rev. 25:1993;1053-1065
28. Håkanson L. Optimal size of predictive models. Ecol. Modell. 78:1995;195-204
29. Helms, J.A. (Ed.), 1998. The Dictionary of Forestry. Society of American Foresters, Bethesda, MD, 210 pp.
30. Homann P.S, McKane R.B, Sollins P. Belowground processes in forest-ecosystem biogeochemical simulation models. For. Ecol. Manage. 138:2000;3-18
31. House, P.W., 1982. The Art of Public Policy Analysis. Sage, Beverly Hills, CA.
32. Jenkins J.C, Chojnacky D.C, Heath L.S, Birdsey R.A. National-scale biomass estimators for United States tree species. For. Sci. 49:(1):2003;12-34
33. Jiang H, Apps M.J, Peng C, Zhang Y, Liu J. Modelling the influence of harvesting on Chinese boreal forest carbon dynamics. For. Ecol. Manage. 169:2002;65-82
34. Johnson D.H, Conroy M.J, Nichols J.D. The need for accuracy in modeling: an example. Ecol. Model. 30:1985;157-161
35. Johnson W.B. Application of air quality modeling to decisionmaking under the Clean Air Act (42 USCA §§7401-7626). Am. Law Rep. 84:1987;710
36. Jones E.M. A component approach to minimal rationality review for agency rulemaking. Admin. Law Rev. 39:1987;275-313
37. Joosten R, Schumacher J, Wirth C, Schulte A. Evaluating tree carbon predictions for beech (Fagus sylvatica L.) in western Germany. For. Ecol. Manage. 189:2003;87-96
38. Kagan, R.A., 2001. Adversarial Legalism: The American Way of Life. Harvard University Press, Cambridge, MA.
39. Karjalainen T. Dynamics and potentials of carbon sequestration in managed stands and wood products in Finland under changing climatic conditions. For. Ecol. Manage. 80:1996;112-132
40. Konikow L.F, Bredehoft J.D. Ground-water models cannot be validated. Ad. Water Resour. 15:1992;75-83
41. Korzukhin M.D, Ter-Mikaelian M.T, Wagner R.G. Process versus empirical models: which approach for forest ecosystem management? Can. J. For. Res. 26:1996;879-887
42. Kurz, W.A., Apps, M.J., Webb, T.M., McNamee, P.J., 1992. The Carbon Budget of the Canadian Forest Sector: Phase I. Forestry Canada, Edmonton, 93 pp.
43. Kurz W.A, Beukema S.J, Apps M.J. Estimation of root biomass and dynamics for the carbon budget model of the Canadian forest sector. Can. J. For. Res. 26:(11):1996;1973-1979
44. Laclau P. Biomass and carbon sequestration of ponderosa pine plantations and native cypress forests in northwest Patagonia. For. Ecol. Manage. 180:2003;317-333
45. Landsberg J. Modelling forest ecosystems: state-of-the-art, challenges, and future directions. Can. J. For. Res. 33:(3):2001;385-397
46. Loehle C. A hypothesis testing framework for evaluating ecosystem model performance. Ecol. Model. 97:1997;153-165
47. Losi C.J, Siccama T.G, Condit R, Morales J.E. Analysis of alternative methods for estimating carbon stock in young tropical plantations. For. Ecol. Manage. 184:2003;355-368
48. Luis S.J, McLaughlin D. A stochastic approach to model validation. Adv. Water Resour. 15:1992;15-32
49. Mäkelä A, Landsberg J.J, Ek A.E, Burk T.E, Ter-Mikaelian M, Ågren G, Oliver C.D, Puttonen P. Process-based models for forest ecosystem management: current state-of-the-art and challenges for practical implementation. Tree Physiol. 20:2000;289-298
50. Mäkipää R, Karjalainen T, Pussinen A, Kukkola M, Kellomäki S, Mälkönen E. Applicability of a forest simulation model for estimating effects of nitrogen deposition on a forest ecosystem: test of the validity of a gap-type model. For. Ecol. Manage. 108:1998;239-250
51. Monserud, R.A., 2003. Evaluating forest models in a sustainable forest management context. For. Biometry Model. Inform. Sci. 1, 35-47. http://www.fbmis.info.
52. Murray B.C, Prisley S.P, Birdsey R.A, Sampson R.N. Carbon sinks in the Kyoto Protocol: potential relevance for US Forests. J. For. 98:(9):2000;6-11
53. Newberry, J.D., Stage, A.R., 1988. Validating forest growth models: procedures defined by resource decisions. In: Ek, A.R., Shifley, S.R., Burk, T.E. (Eds.), Forest Growth Modelling and Prediction: Proceedings of the IUFRO Conference, 23-27 August 1987, Minneapolis, MN. USFS General Technical Report NC-120, pp. 786-793.
54. Office of Technology Assessment, 1982. Use of Models for Water Resources Management, Planning, and Policy. US OTA, Washington, DC, 242 pp.
55. Olesen, H.R., 2001. Ten years of harmonization activities: past, present, and future. In: Proceedings of the Seventh International Conference on Harmonisation within Atmospheric Dispersion Modelling for Regulatory Purposes. http://www.harmo.org/Docs/TenYears.pdf.
56. Olesen, H.R., 2001. A platform for model validation. In: Proceedings of the Seventh International Conference on Harmonisation within Atmospheric Dispersion Modelling for Regulatory Purposes. http://www.harmo.org/Docs/ ME_platform.pdf.
57. Oreskes N, Shrader-Frechette K, Belitz K. Verification, validation, and confirmation of numerical models in the earth sciences. Science. 263:1994;641-646
58. Oreskes N. Evaluation (not validation) of quantitative models. Environ. Health Perspect. 106:(Suppl. 6):1998;1453-1460
59. Otero-Phillips C.V. What's in the forecast? A look at the EPA's use of computer models in emissions trading. Rutgers Comput. Technol. Law J. 24:1998;187-222
60. Parker P, Letcher R, Jakeman A, Beck M.B, Harris G, Argent R.M, Hare M, Pahl-Wostl C, Voinov A, Janssen M, Sullivan P, Scoccimarro M, Friend A, Sonnenshein M, Barker D, Matejicek L, Odulaja D, Deadman P, Lim K, Larocque G, Tarikhi P, Fletcher C, Put A, Maxwell T, Charles A, Breeze H, Nakatani N, Mudgal S, Naito W, Osidele O, Eriksson I, Kautsky U, Kautsky E, Naeslund B, Kumblad L, Park R, Maltagliati S, Girardin P, Rizzoli A, Mauriello D, Hoch R, Pelletier D, Reilly J, Olafsdottir R, Bin S. Progress in integrated assessment and modeling. Environ. Model. Softw. 17:2002;209-217
61. Paul K.I, Polglase P.J, Richards G.P. Predicted change in soil carbon following afforestation or reforestation, and analysis of controlling factors by linking a C accounting model (CAMFor) to models of forest growth (3PG), litter decomposition (GENDEC) and soil C turnover (RothC). For. Ecol. Manage. 177:2003;485-501
62. Potter C.S, Wang S, Nikolov N.T, McGuire A.D, Liu J, King A.W, Kimball J.S, Grant R.F, Frolking S.E, Clein J.S, Chen J.M, Amthor J.S. Comparison of boreal ecosystem model sensitivity to variability in climate and forest site parameters. J. Geophys. Res. 106:(33):2001;671-688
63. Powlson, D.S., 1996. Why evaluate soil organic matter models? In: Powlson, D.S., Smith, P., Smith, Jo.U. (Eds.), Evaluation of Soil Organic Matter Models Using Existing Long-term Datasets. Spinger-Verlag, Berlin. NATO ASI Series, Series I, Global Environmental Change, vol. 38, pp. 3-11.
64. Proctor, P., Heath, L.S., Van Deusen, P.C., Gove, J.H., Smith, J.E., 2004. COLE: a web-based tool for interfacing with forest inventory data. In: McRoberts, R., Reams, G., Van Deusen, P., McWilliams, W. (Eds.), Proceedings of the Fourth Annual Forest Inventory Analysis Symposium, November 19-21, 2002, New Orleans, LA. General Technical Report NC. US Department of Agriculture, Forest Service, North Central Research Station, St. Paul, MN.
65. Ritson P, Sochacki S. Measurement and prediction of biomass and carbon content of Pinus pinaster trees in farm forestry plantations, south-western Australia. For. Ecol. Manage. 175:2003;103-117
66. Rykiel E.J Jr. Testing ecological models: the meaning of validation. Ecol. Model. 90:1996;229-244
67. Schlesinger S, Crosbie R.E, Gagné R.E, Innis G.S, Lalwani C.S, Loch J, Sylvester R.J, Wright R.D, Kheir N, Bartos D. Terminology for model credibility. SCS Technical Committee on Model Credibility. Simulation. 32:(3):1979;103-104
68. Seely B, Welham C, Kimmins H. Carbon sequestration in a boreal forest ecosystem: results from the ecosystem simulation model, FORECAST. For. Ecol. Manage. 169:2002;123-135
69. Sharpe, P.J.H., 1990. Forest modeling approaches: compromises between generality and precision. In: Dixon, R.K., Meldahl, R.S., Ruark, G.A., Warren, W.G. (Eds.), Process Modeling of Forest Growth Responses to Environmental Stress. Timber Press, Portland, OR, pp. 180-190.
70. Schimel, D.S., VEMAP Participants, Braswell, B.H., 1997. Continental scale variability in ecosystem processes: models, data, and the role of disturbance. Ecol. Monogr. 67 (2), 251-271.
71. 2 and climate to carbon storage by ecosystems of the United States. Science. 287:2000;2004-2006
72. Shook L.R, Tartal J.A. Unreliable speculation or sound science under Daubert? - use of expert testimony to model contaminant release and transport. Am. Law Inst.-Am. Bar Assoc. 55:2000;525-545
73. Sklash M, Schroeder M, Dragun J. Groundwater models: Can you believe what they are saying? Nat. Resour. Environ. 13:(4):1999;542-546
74. Smith J.E, Heath L.S. Identifying influences on model uncertainty: an application using a forest carbon budget model. Environ. Manage. 27:(2):2001;253-267
75. Smith, J.E., Heath, L.S., 2002. A model of forest floor carbon mass for United States forest types. Research Paper NE-722. US Department of Agriculture, Forest Service, Northeastern Research Station, Newtown Square, PA, 37 pp.
76. Smith, J.E., Heath, L.S., Jenkins, J.C., 2003. Forest volume-to-biomass models and estimates of mass for live and standing dead trees of US forests. General Technical Report NE-298. US Department of Agriculture, Forest Service, Northeastern Research Station, Newtown Square, PA, 57 pp.
77. Stage A. How forest models are connected to reality: evaluation criteria for their use in decision support? Can. J. For. Res. 33:(3):2003;410-421
78. Soares P, Tome M, Skovsgaard J.P, Vanclay J.K. Evaluating a growth model for forest management using continuous forest inventory data. For. Ecol. Manage. 71:1995;251-265
79. United Nations Framework Convention on Climate Change (UNFCCC), 1997. Kyoto Protocol to the United Nations Framework Convention on Climate Change. UNFCCC Document FCCC/CP/1997/7/Add. 1.
80. US Department of Agriculture, 2003a. Supplementary Guidelines for the Quality of Regulatory Information Disseminated by USDA Agencies and Offices. Office of the Chief Information Officer, Washington, DC. http://www.ocio.usda. gov/irm/qi_guide/regualtory.htm.
81. US Department of Agriculture, 2003b. Public Disclosure of Quality of Information Requests. Forest Service, Washington, DC. http://www.fs.fed.us/qoi/ disclosure.shtml (accessed 12 December 2003).
82. US Environmental Protection Agency, 1994a. Guidance for Conducting External Peer Review of Environmental Regulatory Models (EPA 100-B-94-001). Office of the Administrator, Washington, DC. http://www.epa.gov/osp/spc/modelpr. htm.
83. US Environmental Protection Agency, 1994b. Validation strategy for the Integrated Exposure Uptake Biokinetic Model for lead in children. EPA 540/R-94-039.
84. US Environmental Protection Agency, 1999. White paper on the nature and scope of issues on adoption of model use acceptability guidelines. Science Policy Council, Model Acceptance Criteria and Peer Review White Paper Working Group. http://cfpub.epa.gov/crem/library/whitepaper_1999.pdf.
85. US Environmental Protection Agency, 2002. Guidance for Quality Assurance Project Plans for Modeling. EPA QA/G-5M. US Environmental Protection Agency, Washington, DC. http://cfpub.epa.gov/crem/library/QA_Guidance.pdf.
86. Vanclay J.K, Skovsgaard J.P. Evaluating forest growth models. Ecol. Model. 98:1997;1-12
87. Versar Inc., 1988. Current and suggested practices in the validation of exposure assessment models. Report. Office of Health and Environmental Assessment, US EPA, Washington, DC.
88. Westerbrook J.A. Air dispersion models: tools to assess impacts from pollution sources. Nat. Resour. Environ. 13:(4):1999;546-551
89. Woodbury P.B, Smith J.E, Weinstein D.A, Laurence J.A. Assessing potential climate change effects on loblolly pine growth: a probabilistic regional modeling approach. For. Ecol. Manage. 107:1998;99-116