Technical assessment and evaluation of environmental models and software: Letter to the Editor

Environmental Modelling and Software

Volumn 26, Issue 3, 2011, Pages 328-336

  Alexandrov, G A ^a   Ames, D ^b   Bellocchi, G ^c   Bruen, M ^d   Crout, N ^e   Erechtchoukova, M ^f   Hildebrandt, A ^g   Hoffman, F ^h   Jackisch, C ⁱ   Khaiter, P ^f   Mannina, G ^j   Matsunaga, T ^a   Purucker, S T ^k   Rivington, M ^l   Samaniego, L ^g  

^a NATIONAL INSTITUTE FOR ENVIRONMENTAL STUDIES   (Japan)

^b IDAHO STATE UNIVERSITY   (United States)

^c French National Institute of Agricultural Research   (France)

^d UNIVERSITY COLLEGE DUBLIN   (Ireland)

^e UNIVERSITY OF NOTTINGHAM   (United Kingdom)

^f YORK UNIVERSITY   (Canada)

^g HELMHOLTZ CENTRE FOR ENVIRONMENTAL RESEARCH UFZ   (Germany)

^h OAK RIDGE NATIONAL LABORATORY   (United States)

ⁱ TECHNICAL UNIVERSITY OF MUNICH   (Germany)

^j UNIVERSITY OF PALERMO   (Italy)

^k U S ENVIRONMENTAL PROTECTION AGENCY   (United States)

^l SCOTTISH CROP RESEARCH INSTITUTE   (United Kingdom)

more..

Author keywords

Environmental assessment; Model credibility; Model evaluation; Software verification

Indexed keywords

BEST-PRACTICES; ENVIRONMENTAL ASSESSMENT; ENVIRONMENTAL MODEL; ENVIRONMENTAL MODELLERS; ENVIRONMENTAL MODELLING; EVALUATION CRITERIA; EVALUATION TOOL; FUNDING AGENCIES; KEY ISSUES; MODEL DEVELOPMENT; MODEL EVALUATION; MODEL TESTING; MODEL-BASED; PEER-REVIEW PROCESS; REVIEW PROCESS; SOFTWARE VERIFICATION; TECHNICAL ASSESSMENT;

COMPUTER SOFTWARE SELECTION AND EVALUATION; RATING; RESEARCH; VERIFICATION;

QUALITY CONTROL;

ENVIRONMENTAL ASSESSMENT; ENVIRONMENTAL MODELING; LITERATURE REVIEW; RISK FACTOR; SOFTWARE; STAKEHOLDER; STANDARDIZATION;

EID: 78649856857     PISSN: 13648152     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.envsoft.2010.08.004     Document Type: Article

References (71)

1. Alexandrov G.A. The purpose of peer review in the case of an open-access publication. Carbon Balance and Management 2006, 1:10. http://www.cbmjournal.com/content/1/1/10.
2. Alexandrov G.A., Matsunaga T. Evaluating consistency of biosphere models: software tools for a web-based service. Proceedings of the IEMSs Fourth Biennial Meeting: International Congress on Environmental Modelling and Software (IEMSs 2008) 2008, International Environmental Modelling and Software Society, Barcelona, Catalonia. http://www.iemss.org/iemss2008/uploads/Main/S12-02-Alexandrov_et_al-IEMSS2008.pdf, M. Sànchez-Marrè, J.J. BéjarComas, A.E. Rizzoli, E. Guariso (Eds.).
3. Aumann C. A methodology for building credible models for policy evaluation. Proceedings of the IEMSs Fourth Biennial Meeting: International Congress on Environmental Modelling and Software (IEMSs 2008) 2008, International Environmental Modelling and Software Society, Barcelona, Catalonia. http://www.iemss.org/iemss2008/uploads/Main/S12-01-Aumann_et_al-IEMSS2008.pdf, M. Sànchez-Marrè, J.J. BéjarComas, A.E. Rizzoli, E. Guariso (Eds.).
4. Bellocchi G., Acutis M., Fila G., Donatelli M. An indicator of solar radiation model performance based on a fuzzy expert system. Agronomy Journal 2002, 94:1222-1233. http://agron.scijournals.org/cgi/reprint/94/6/1222.pdf.
5. Bellocchi G., Confalonieri R., Donatelli M. Crop modelling and validation: integration of IRENE_DLL in the WARM environment. Italian Journal of Agrometeorology 2006, 3:35-39.
6. Bellocchi G., Habyarimana E., Donatelli M., Acutis M. A software component for model output evaluation. Proceedings of the IEMSs Fourth Biennial Meeting: International Congress on Environmental Modelling and Software (IEMSs 2008) 2008, International Environmental Modelling and Software Society, Barcelona, Catalonia. http://www.iemss.org/iemss2008/uploads/Main/S12-06-Donatelli_et_al-IEMSS2008.pdf, M. Sànchez-Marrè, J.J. BéjarComas, A.E. Rizzoli, E. Guariso (Eds.).
7. Bellocchi G., Rivington M., Donatelli M., Matthews K.B. Validation of biophysical models: issues and methodologies. A review. Agronomy for Sustainable Development 2010, 30:109-130.
8. Benz J., Knorrenschild M. Call for a common model documentation etiquette. Ecological Modelling 1997, 97:141-143.
9. Beven K. A manifesto for the equifinality thesis. Journal of Hydrology 2006, 320:18-36.
10. Beven K.J., Freer J. Equifinality, data assimilation, and uncertainty estimation inmechanistic modelling of complex environmental systems using the GLUE methodology. Journal of Hydrology 2001, 249:11-29.
11. Brun R., Reichert P., Künsch H.R. Practical identifiability analysis of large environmental simulation models. Water Resources Research 2001, 37(4):1015-1030.
12. Burnham K.P., Anderson D.R. Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach 2002, Springer-Verlag, New York (NY). second ed.
13. Carberry P.S., Hochman Z., McCown R.L., Dalgliesh N.P., et al. The FARMSCAPE approach to decision support: farmers', advisers, researchers' monitoring, simulation, communication and performance evaluation. Agricultural Systems 2002, 74:141-177.
14. Cheng R.T., Burau J.R., Gartner J.W. Interfacing data analysis and numerical modelling for tidal hydrodynamic phenomena. Tidal Hydrodynamics 1991, 201-219. John Wiley & Sons, New York. B.B. Parker (Ed.).
15. Confalonieri R., Acutis M., Bellocchi G., Donatelli M. Multi-metric evaluation of the models WARM, CropSyst, and WOFOST for rice. Ecological Modelling 2009, 220:1395-1410.
16. Confalonieri R., Bellocchi G., Boschetti M., Acutis M. Evaluation of parameterization strategies for rice modelling. Spanish Journal of Agricultural Research 2009, 7:680-686.
17. Confalonieri R., Bellocchi G., Tarantola S., Acutis M., Donatelli M., Genovese G. Sensitivity analysis of the rice model WARM in Europe: Exploring the effects of different locations, climates and methods of analysis on model sensitivity to crop parameters. Environmental Modelling and Software 2010, 25:479-488.
18. Council of Science Editors CSE's White Paper on Promoting Integrity 2006, Scientific Journal Publications, CSE, Reston, Va. http://www.councilscienceeditors.org/editorial_policies/whitepaper/entire_whitepaper.pdf.
19. Cox G.M., Gibbons J.M., Wood A.T.A., Craigon J., Crout N.M.J. Towards the systematic simplification of mechanistic models. Ecological Modelling 2006, 198:240-246.
20. Crout N., Kokkonen T., Jakeman A.J., Norton J.P., Newham L.T.H., Anderson R., Assaf H., Croke B.F.W., Gaber N., Gibbons J., Holzworth D., Mysiak J., Reichl J., Seppelt R., Wagener T., Whitfield P. Good modelling practice. Environmental Modelling, Software and Decision Support 2009, 15-31. Elsevier. A.J. Jakeman, A.A. Voinov, A.E. Rizzoli, S.H. Chen (Eds.).
21. EEA Modelling Environmental Change in Europe: Towards a Model Inventory (SEIS/Forward). EEA Technical Report 11/2008 2008, European Environment Agency, Copenhagen.
22. French S., Geldermann J. The varied contexts of environmental decision problems and their implications for decision support. Environmental Science and Policy 2005, 8:378-391.
23. Freni G., Mannina G., Viviani G. Urban runoff modelling uncertainty: Comparison among Bayesian and pseudo-Bayesian methods. Environmental Modelling and Software 2009, 24(9):1100-1111.
24. Freni G., Mannina G., Viviani G. Identifiability analysis for receiving water body quality modelling. Environmental Modelling and Software 2009, 24(1):54-62.
25. Freni G., Mannina G., Viviani G. Assessment of the integrated urban water quality model complexity through identifiability analysis. Water Research 2010, 10.1016/j.watres.2010.08.004.
26. Funtowicz S.O., Ravetz J.R. Uncertainty and Quality in Science for Policy 1990, Kluwer Academic Press, Dordrecht (The Netherlands).
27. Gaber N., Pascual P., Stiber N., Sunderland E., Cope B., Nold A. Guidance on the Development, Evaluation and Application of Environmental Models 2008, Council for Regulatory Environmental Modeling, U.S. Environmental Protection Agency, Washington.
28. Gardner R.H., Urban D.L. Model validation and testing: past lessons, present concerns, future prospects. Models in Ecosystem Science 2003, 184-203. PrincetonUniversity Press, Princeton (NJ). C.D. Canham, J.J. Cole, W.K. Lauenroth (Eds.).
29. Glasow, P.A., Pace, D.K., 1999. SIMVAL '99: Making VV&A Effective and Affordable Workshop, The Simulation Validation Workshop 1999, January 26-29, Laurel, MD, USA.
30. Haag D., Kaupenjohann M. Parameters, prediction, post-normal science and the precautionary principle - a roadmap for modelling for decision-making. Ecological Modelling 2001, 144:45-60.
31. Hames I. Peer-review and Manuscript Management in Scientific Journals: Guidelines for Good Practice 2007, Blackwell Publishing, Oxford.
32. Harremoës, P., 2003. The role of uncertainty in application of integrated urban water modeling. In: Proceedings of the International IMUG Conference, Tilburg, 23-25 April 2003.
33. Hoffman F., Bonan G., Covey C., Fung I., Lee Y.-H., Randerson J., Running S., Thornton P. The Carbon-Land Model Intercomparison Project (C-LAMP): a protocol and evaluation metrics for global terrestrial biogeochemistry models. Proceedings of the IEMSs Fourth Biennial Meeting: International Congress on Environmental Modelling and Software (IEMSs 2008) 2008, International Environmental Modelling and Software Society, Barcelona, Catalonia. http://www.iemss.org/iemss2008/uploads/Main/S12-03-Hoffmann_et_al-IEMSS2008.pdf, M. Sànchez-Marrè, J.J. BéjarComas, A.E. Rizzoli, E. Guariso (Eds.).
34. Hsu M.H., Kuo A.Y., Kuo J.T., Liu W.C. Procedure to calibrate and verify numerical models of estuarine hydrodynamics. Journal of Hydraulic Engineering 1999, 125:166-182.
35. Huth N., Holzworth D. Common sense in model testing. Proceedings of MODSIM 2005 International Congress on Modelling and Simulation: Advances and Applications for Management and Decision Making, 12-15 December, Melbourne, Australia 2005, 2804-2809. A. Zerger, R.M. Argent (Eds.).
36. Jakeman A.J., Letcher R.A., Norton J.P. Ten iterative steps in development and evaluation of environmental models. Environmental Modelling and Software 2006, 21(5):602-614.
37. Jakeman A., Chen S., Rizzoli A., Voinov A.A. Modelling and software as instruments for advancing sustainability. Environmental Modelling, Software and Decision Support 2009, 345-366. Elsevier. A.J. Jakeman, A.A. Voinov, A.E. Rizzoli, S.H. Chen (Eds.).
38. Janssen P.H.M., Heuberger P.S.C. Calibration of process-oriented models. Ecological Modelling 1995, 83:55-66.
39. Jørgensen S.E., Fath B.D., Grant W., Nielsen S.N. The editorial policy of ecological modelling. Ecological Modelling 2006, 199:1-3.
40. Keating B.A., Gaydon D., Huth N.I., Probert M.E., Verburg K., Smith C.J., Bond W. Use of modelling to explore the water balance of dryland farming systems in the Murray-Darling Basin, Australia. European Journal of Agronomy 2002, 18:159-169.
41. Leffelaar P.A., Meike H., Smith P., Wallach D. Modelling cropping systems - highlights of the symposium and preface to the special issues. 3. Session B. Model parameterisation and testing. European Journal of Agronomy 2003, 18:189-191.
42. Li W., Arena V.C., Sussman N.B., Mazumdar S. Model validation software for classification models using repeated partitioning: MVREP. Computer Methods and Programs in Biomedicine 2003, 72:81-87.
43. Mankin J.B., O'Neill R.V., Shugart H.H., Rust B.W. The Importance of Validation in Ecosystem Analysis. In New Directions in the Analysis of Ecological Systems, Part 1 1975, vol. 5. Society for Computer Simulation, La Jolla, CA, pp. 63-71.
44. Mannina G., Viviani G. An urban drainage stormwater quality model: Model development and uncertainty quantification. Journal of hydrology 2010, 381(3-4):248-265.
45. Marcus A.H., Elias R.W. Some useful statistical methods for model validation. Environmental Health Perspectives 1998, 106:1541-1550.
46. Matthews, K.B., Rivington, M., Blackstock, K., Buchan, K., Miller, D.G. Raising the bar - Is evaluating the outcomes of decision and information support tools a bridge too far? Environmental Modelling and Software, this volume.
47. Min S.-K., Hense A. A Bayesian assessment of climate change using multimodel ensembles. Part I: global mean surface temperature. Journal of Climate 2006, 19:3237-3256.
48. Müller, U.T., 2009. Peer-Review-Verfahren zur Qualitätssicherung von Open-Access-Zeitschriften - systematische Klassifikation und empirische Untersuchung, PhD thesis, Humboldt-Universität zu Berlin, 2009. Available at: http://edoc.hu-berlin.de/docviews/abstract.php%3Fid%3D29636.
49. Murray-Darling Basin Commission Murray-Darling Basin Commission, Groundwater Flow Modelling Guideline 2000, Murray-Darling Basin Commission, Canberra, Project no. 125.
50. Pachepsky Y.A., Guber A.K., Van Genuchten M.T., Nicholson T.J., Cady R.E., Simunek J., Schaap M.G. Model Abstraction Techniques for Soil Water Flow and Transport 2006, 175 pp., Nuclear Regulatory Commission, Washington, DC, NUREG/CR-6884.
51. Parker V.T. Conceptual problems and scale limitations of defining ecological communities: a critique of the CI concept (Community of Individuals). Perspectives in Plant Ecology, Evolution and Systematics 2001, 4:80-96.
52. Raupach M.R., Finnigan J.J. Single layer models of evaporation from plant canopies are incorrect, but useful, whereas multilayer models are correct, but useless: discussion. Australian Journal of Plant Physiology 1988, 15:705-716.
53. Reichert P., Borsuk M.E. Does high forecast uncertainty preclude effective decision support?. Environmental Modelling and Software 2005, 20:991-1001.
54. Reynolds J.F., Ford E.D. Multi-criteria assessment of ecological process models. Ecology 1999, 80:538-553.
55. Robson B., Hamilton D., Webster I., Chan T. Ten steps applied to development and evaluation of process-based biogeochemical models of estuaries. Environmental Modelling and Software 2008, 23(4):369-384.
56. Rykiel E.J. Testing ecological models: the meaning of validation. Ecological Modelling 1996, 90:229-244.
57. Sinclair T.R., Seligman N. Criteria for publishing papers on crop modelling. Field Crops Research 2000, 68:165-172.
58. STOWA/RIZA STOWA/RIZA, Smooth Modelling in Water Management, Good Modelling Practice Handbook STOWA Report 99-05 1999, Dutch Department of Public Works, Institute for Inland Water Management and Waste Water Treatment, Report 99.036.
59. van der Slujis J.P. Uncertainty and precaution in environmental management: insights from the UPEM conference. Environmental Modelling and Software 2007, 22(5):590-598.
60. van Nes E.H., Scheffer M. Alternative attractors may boost uncertainty and sensitivity in ecological models. Ecological Modelling 2003, 159:117-124.
61. van Oijen M. On the use of specific publication criteria for papers on process-based modelling in plant science. Field Crops Research 2002, 74:197-205.
62. Vanclay J.K. Modelling Forest Growth and Yield 1994, CAB International, Wallingford, United Kingdom.
63. Vanclay J.K., Skovsgaard J.P. Evaluating forest growth models. Ecological Modelling 1997, 98:1-12.
64. Vischel T., Pegram G., Sinclair S., Wagner W., Bartsch A. Comparison of soil moisture fields estimated by catchment modelling and remote sensing: a case study in South Africa. Hydrology and Earth System Sciences Discussion 2007, 4:2273-2306.
65. Voinov A., Hood R.R., Daues J.D., Assaf H., Stewart R. Building a community modelling and information sharing culture. Environmental Modelling, Software and Decision Support 2009, 345-366. Elsevier. A.J. Jakeman, A.A. Voinov, A.E. Rizzoli, S.H. Chen (Eds.).
66. Wager E. Ethics: what is it for?. Nature 2006, Web Debate - Peer-review. http://www.nature.com/nature/peerreview/debate/nature04990.html.
67. Walker T.J. Free internet access to traditional journals. American Scientist 1998, 86:463.
68. Wang Y.-P., Trudinger C.M., Enting I.G. A review of applications of model-data fusion to studies of terrestrial carbon fluxes at different scales. Agricultural and Forest Meteorology 2009, 149:1829-1842.
69. Welsh W. Water balance modelling in Bowen, Queensland, and the ten iterative steps in model development and evaluation. Environmental Modelling and Software 2008, 23(2):195-205.
70. Willems P. Quantification and relative comparison of different types of uncertainties in sewer water quality modelling. Water Research 2008, 42(13):3539-3551.
71. Ye M., Meyer P.D., Neuman S.P. On model selection criteria in multimodel analysis. Water Resources Research 2008, 44:W03428.