Perspectives to performance of environment and health assessments and models-from outputs to outcomes?

International Journal of Environmental Research and Public Health

Volumn 10, Issue 7, 2013, Pages 2621-2642

  Pohjola, Mikko V ^a,b   Pohjola, Pasi ^a   Tainio, Marko ^c   Tuomisto, Jouni T ^a  

^a NATIONAL PUBLIC HEALTH INSTITUTE   (Finland)

^b Nordem Oy   (Finland)

^c SYSTEMS RESEARCH INSTITUTE   (Poland)

Author keywords

Assessment; Effectiveness; Environment; Evaluation; Health; Management; Model; Performance; Quality; Uncertainty

Indexed keywords

EFFICIENCY MEASUREMENT; ENVIRONMENTAL ASSESSMENT; HEALTH RISK; INPUT-OUTPUT ANALYSIS; MODEL TEST; PERFORMANCE ASSESSMENT; PLANNING METHOD; POLICY APPROACH; UNCERTAINTY ANALYSIS;

DECISION MAKING; ENVIRONMENTAL HEALTH; ENVIRONMENTAL MANAGEMENT; POLICY; QUALITY CONTROL; REVIEW; SCIENTIFIC LITERATURE; SOCIAL ASPECT; STATISTICAL MODEL;

EID: 84879572160     PISSN: 16617827     EISSN: 16604601     Source Type: Journal    
DOI: 10.3390/ijerph10072621     Document Type: Review

References (119)

1. Pohjola, M. Assessments are to Change the World-Prerequisites for Effective Environmental Health Assessment; University of Eastern Finland: Kuopio, Finland, May 2013.
2. McIntosh, B.S.; Alexandrov, G.; Matthews, K.; Mysiak, J.; van Ittersum, M. Thematic issue on the assessment and evaluation of environmental models and software. Environ. Model. Software 2011, 26, 245-336.
3. Matthews, K.B.; Rivington, M.; Blackstock, K.L.; McCrum, G.; Buchan, K.; Miller, D.G. Raising the bar?-The challenges of evaluating the outcomes of environmental modeling and software. Environ. Model. Software 2011, 26, 247-257.
4. Van Der Sluijs, J.P. Integrated Assessment. Responding to Global Environmental Change. In Encyclopedia of Global Environmental Change; Munn, T., Ed.; John Wiley & Sons Ltd: Chichester, UK, 2002; pp. 250-253.
5. Health Impact Assessment (HIA), main concepts and suggested approach. Gothenburg consensus paper; World Health Organization: Brussels, Belgium, 1999.
6. NRC. Risk Assessment in the Federal Government: Managing the Progress; The National Research Council National Academy Press: Washington, DC, USA, 1983.
7. NRC. Understanding Risk: Informing Decisions in a Democratic Society; The National Research Council National Academy Press: Washington, DC, USA, 1996.
8. NRC. Science and Decisions: Advancing Risk Assessment; The National Research Council National Academy Press: Washington, DC, USA, 2009.
9. ECHA. Guidance on Information Requirements and Chemical Safety Assessment. Guidance for the implementation of REACH. Available online: www.echa.europa.eu/guidance-documents/guidance-on-information-requirements-and-chemical-safety-assessment (accessed on 18 June 2013)
10. Wood, C. Environmental Impact Assessment: A Comparative Review; Longman Scientific & Technical: New York, NY, USA, 1995.
11. Briggs, D.J. A framework for integrated environmental health impact assessment of systemic risks. Environ. Health 2008, 7, 61, doi:10.1186/1476-069X-7-61.
12. Pohjola, M.V.; Leino, O.; Kollanus, V.; Tuomisto, J.T.; Gunnlaugsdóttir, H.; Holm, F.; Kalogeras, N.; Luteijn, J.M.; Magnusson, S.H.; Odekerken, G, et al. State of the art in benefit-Risk analysis: Environmental health. Food Chem. Toxicol. 2012, 50, 40-55.
13. McIntosh, B.S.; Alexandrov, G.; Matthews, K.; Mysiak, J.; van Ittersum, M. Preface: Thematic issue on the assessment and evaluation of environmental models and software. Environ. Model. Software 2011, 26, 245-246.
14. Hummel, J.; Huitt, W. What you measure is what you get. GaASCD Newsletter: The Reporter, 1994. Available online: www.edpsycinteractive.org/papers/wymiwyg.html (accessed on 18 June 2013).
15. Norton, B.G. Integration or Reduction: Two Approaches to Environmental Values. In Searching for Sustainability, Interdisciplinary Essays in the Philosophy of Conservation Biology; Norton, B.G., Ed.; Cambridge University Press: Cambridge, UK, 2002.
16. Jakeman, A.J.; Letcher, R A. Integrated assessment and modeling: Features, principles and examples for catchment management. Environ. Model. Software 2003, 18, 491-501.
17. Rizzoli, A.E.; Leavesley, G.; Ascough II J.C.; Argent, R.M.; Athanasiadis, I.N.; Brilhante, V.; Claeys, F.H.A.; David, O.; Donatelli, M.; Gijsbergs, P. et al. Integrated Modeling Frameworks for Environmental Assessment Decision Support. In Environmental modeling, Software and Decision Support; Jakeman, A.J., Voinov, A.A., Rizzoli, A.E., Chen, S.H., Eds.; Elsevier: Amsterdam, The Netherlands, 2008; pp. 101-118.
18. Pope, J.; Annandale, D.; Morrison-Saunders, A. Conceptualising sustainability assessment. Environ. Impact Assess. Rev. 2004, 24, 595-616.
19. Blackstock, K.L.; Kelly, G.J.; Horsey, B.L. Developing and applying a framework to evaluate participatory research for sustainability. Ecol. Econ. 2007, 60, 726-742.
20. Bina, O. Context and systems: Thinking more broadly about effectiveness in strategic environmental assessments in China. Environ. Manage. 2008, 42, 717-733.
21. Leviton, L.C. Evaluation use: Advances, challenges and applications. Am. J. Eval. 2003, 24, 525-535.
22. Patton, M.Q. Utilization-Focused Evaluation; SAGE Publications Inc: Thousand Oakes, CA, USA, 2008.
23. Reeves, C.A.; Bednar, D.A. Defining quality: Alternatives and implications. Acad. Manage. Rev. 1994, 19, 419-445.
24. Harteloh, P.P.M. Quality systems in health care: A sociotechnical approach. Health Policy 2002, 64, 391-398.
25. Guyatt, G.H.; Sackett, D.L.; Sinclair, J.C.; Hayward, R.; Cook, D.J.; Cook, R.J. Users' guides to the medical literature. IX. A method for grading health care recommendations. JAMA 1995, 274, 1800-1804.
26. Cartwright, N. Are RCTs the gold standard? BioSocieties 2007, 2, 11-20.
27. Jakeman, A.J.; Letcher, R.A.; Norton, J.P. Ten iterative steps in development and evaluation of environmental models. Environ. Model. Software 2006, 21, 602-614.
28. Refsgaard, J.C.; Henriksen, H.J.; Harrar, W.G.; Scholten, H.; Kassahun, A. Quality assurance in model-based water management-Review of existing practice and outline of new approaches. Environ. Model. Software 2005, 20, 1201-1215.
29. van Delden, H.; Seppelt, R.; White, R.; Jakeman, A. J. A methodology for the design and development of integrated models for policy support. Environ. Model. Software 2011, 26, 266-279.
30. Hoekstra, J.; Hart, A.; Boobis, A.; Claupein, E.; Cockburn, A.; Hunt, A.; Knudsen, I.; Richardson, D.; Schilter, B.; Schütte, K, et al. BRAFO tiered approach for benefit-risk assessment of foods. Food Chem. Toxicol. 2012, 50, S684-S698.
31. Liu, Y.; Gupta, H.; Springer, E.; Wagener, T. Linking science with environmental decision making: Experiences from an integrated modeling approach to supporting sustainable water resources management. Environ. Model. Software 2008, 23, 846-858.
32. Mahmoud, M.; Liu, Y.; Hartmann, H.; Stewart, S.; Wagener, T.; Semmens, D.; Stewart, R.; Gupta, H.; Dominguez, D.; Dominguez, F, et al. A formal framework for scenario development in support of environmental decision-making. Environ. Model. Software 2009, 24, 798-808.
33. Granger Morgan, M.; Dowlatadabi, H. Learning from integrated assessment of climate change. Climatic change 1996, 34, 337-368.
34. Risbey, J.; Kandlikar, M.; Patwardhan, A. Assessing integrated assessment. Climatic Change 1996, 34, 369-395.
35. Forristal, P.M.; Wilke, D.L.; McCarthy, L.S. Improving the quality of risk assessments in Canada using a principle-based apporach. Regul. Toxicol. Pharmacol. 2008, 50, 336-344.
36. Risbey, J.; van der Sluijs, J.P.; Kloprogge, P.; Ravetz, J.; Funtowicz, S.; Corral Quintana, S. Application of a checklist for quality assistance in environmental modeling to an energy model. Environ. Model. Assess. 2005, 10, 63-79.
37. van der Sluijs, J.P.; Craye, M.; Funtowicz, S.; Kloprogge, P.; Ravetz, J.; Risbey, J. Combining quantitative and qualitative measures of uncertainty in model-based environmental assessment: The NUSAP system. Risk Anal. 2005, 25, 481-492.
38. Brown, J.D.; Heuvelink, G.B.; Refsgaard, J.C. An integrated methodology for recording uncertainties about environmental data. Water Sci. Technol. 2005, 52, 153-160.
39. Kloprogge, P.; van der Sluijs, J.P.; Petersen, A.C. A method for the analysis of assumptions in model-based environmental assessments. Environ. Model. Software 2011, 26, 289-301.
40. O'Hagan, A. Probabilistic uncertainty specification: Overview, elaboration techniques and their application to a mechanistic model of carbon flux. Environ. Model. Software 2012, 36, 35-48.
41. Colyvan, M. Is probability the only coherent approach to uncertainty? Risk Anal. 2008, 28, 645-652.
42. Chutia, R.; Mahanta, S.; Datta, D. Non-probabilistic sensitivity and uncertainty analysis of atmospheric dispersion. AFMI 2013, 5, 213-228.
43. Moens, D.; Vandepitte, D. A survey of non-probabilistic uncertainty treatment in finite element analysis. Comput. Method. Appl. Mech. Eng. 2005, 194, 1527-1555.
44. Walker, W.E.; Harremoës, P.; Rotmans, J.; van der Sluijs, J.P.; van Asselt, M.B.A.; Janssen, P.; Krayer von Krauss, M.P. Defining uncertainty: A conceptual basis for uncertainty management in model-based decision support. Integrated Assess. 2003, 4, 5-17.
45. Briggs, D.J.; Sable, C.E.; Lee, K. Uncertainty in epidemiology and health risk and impact assessment. Environ. Geochem. Health 2009, 31, 189-203.
46. van Asselt, M.B.A.; Rotmans, J. Uncertainty in integrated assessment modeling: From positivism to Pluralism. Climatic Change 2002, 54, 75-105.
47. van der Sluijs, J.P.; Petersen, A.C.; Janssen, P.H.M.; Risbey, J.S.; Ravetz, J.R. Exploring the quality of evidence for complex and contested policy decisions. Environ. Res. Lett. 2008, 3, doi:10.1088/1748-9326/3/2/024008.
48. Blind, M.W.; Refsgaard, J.C. Operationalising uncertainty in data and models for integrated water resource management. Water Sci. Technol. 2007, 56, 1-12.
49. Refsgaard, J.C.; van der Sluijs, J.P.; Lajer Højberg, A.; Vanrolleghem, P.A. Uncertainty in the environmental modeling process-A framework and guidance. Environ. Model. Software 2007, 22, 1543-1556.
50. Kann, A.; Weyant, J.P. Approaches for performing uncertainty analysis in large-scale energy/economic policy models. Environ. Model. Assess. 2000, 5, 29-46.
51. Brouwer, R.; De Blois, C. Integrated modeling of risk and uncertainty underlying the cost and effectiveness of water quality measures. Environ. Model. Software 2008, 23, 922-937.
52. Basson, L.; Petrie, J.G. An integrated approach for the consideration of uncertainty in decision making supported by Life Cycle Assessment. Environ. Model. Software 2007, 22, 167-176.
53. Borgonovo, E. Sensitivity analysis of model output with input constraints: A generalized rationale for local methods. Risk Anal. 2008, 28, 667-680.
54. Beck, B. Model Evaluation and Performance. In Encyclopedia of Environmetrics; El-Shaarawi, A.H., Piegorsch, W.W., Eds.; John Wiley & Sons Ltd: Chichester, UK, 2002; Volume 3, pp. 1275-1279.
55. Sargent, R. Verification and validation of simulation models. J. Simulat. 2013, 7, 12-24.
56. Thacker, B.H.; Doebling, S.W.; Hemez, F.M.; Anderson, M.C.; Pepin, J.E.; Rodriguez, E.A. Concepts of Model Verification and Validation; Los Alamos National Laboratory: Los Alamos, NM, USA 2004.
57. Oreskes, N. Evaluation (not validation) of quantitative models. Environ. Health Perspect. 1998, 106, 1453-1458.
58. Van Delden, H. Lessons Learnt in the Development, Implementation and Use of Integrated Spatial Decision Support Systems. In Proceedings of 18th World IMACS Congress and MODSIM09 International Congress on Modelling and Simulation, Cairns, Australia, 13-17 July 2009 Anderssen, R.S., Braddock, R.D., Newham, L.T.H., Eds.; Modelling and Simulation Society of Australia and New Zealand and International Association for Mathematics and Computers in Simulation: Cairns, Australia, 2009; pp. 2922-2928.
59. Alexandrov, G.A.; Ames, D.; Bellochi, G.; Bruen, M.; Crout, N.; Erechtchoukova, M.; Hildebrandt, A.; Hoffman, F.; Jackisch, C.; Khaiter, P, et al. Technical assessment and evaluation of environmental models and software: Letter to the editor. Environ. Model. Software 2011, 26, 328-336.
60. Refsgaard, J.C.; Henriksen, H.J. Modeling guidelines-Terminology and guiding principles. Adv. Water Resour. 2004, 27, 71-82.
61. Bai, Y.; Wagener, T.; Reed, P. A top-down framework for watershed model evaluation and selection under uncertainty. Environ. Model. Software 2009, 24, 901-916.
62. Wyat Appel, K.; Gilliam, R.C.; Davis, N.; Howard, S.C. Overview of the atmospheric model evaluation tool (AMET) v1.1 for evaluating meteorological and air quality models. Environ. Model. Software 2011, 26, 434-443.
63. Xu, Y.; Booij, M.J.; Mynett, A.E. An appropriateness framework for the Dutch Meuse decision support system. Environ. Model. Software 2007, 22, 1667-1678.
64. Sojda, R.S. Empirical evaluation of decision support systems: Needs, definitions, potential methods, and an example pertaining to waterfowl management. Environ. Model. Software 2007, 22, 269-277.
65. Wagener, T.; Kollat, J. Numerical and visual evaluation of hydrological and environmental models using the Monte Carlo analysis toolbox. Environ. Model. Software 2007, 22, 1021-1033.
66. Mo, X.; Liu, S.; Lin, Z. Evaluation of an ecosystem model for a wheat-maize double cropping system over the North China Plain. Environ. Model. Software 2012, 32, 61-73.
67. Pollino, C.A.; Woodberry, O.; Nicholson, A.; Korb, K.; Hart, B.T. Parameterisation and evaluation of a Bayesian network for use in an ecological risk assessment. Environ. Model. Software 2007, 22, 1140-1152.
68. Sonneveld, B.G.J.S.; Keyzer, M.A.; Stroosnijder, L. Evaluating quantitative and qualitative models: An application for nationwide water erosion assessment in Ethiopia. Environ. Model. Software 2011, 26, 1161-1170.
69. Aertsen, W.; Kint, V.; van Orshoven, J.; Muys, B. Evaluation of modeling techniques for forest site productivity prediction in contrasting ecoregions using stochastic multicriteria acceptability analysis (SMAA). Environ. Model. Software 2011, 26, 929-937.
70. Nilsen, T.; Aven, T. Models and model uncertainty in the context of risk analysis. Reliab. Eng. Syst. Safety 2003, 79, 309-317.
71. Moschandreas, D.J.; Karuchit, S. Scenario-model-parameter: A new method of cumulative risk uncertainty analysis. Environ. Int. 2002, 28, 247-261.
72. Refsgaard, J.C.; van der Sluijs, J.P.; Brown, J.; van der Keur, P. A framework for dealing with uncertainty due to model structure error. Adv. Water Resour. 2006, 29, 1586-1597.
73. Hokkanen, P.; Kojo, M. How Environmental Impact Assessment Influences Decision-Making (in Finnish); Ympäristöministeriö: Helsinki, Finland, 2003.
74. Kauppinen, T.; Nelimarkka, K.; Perttilä, K. The effectiveness of human impact assessment in the Finnish Health Cities Network. Public Health 2006, 120, 1033-1041.
75. Ekboir, J. Why impact analysis should not be used for research evaluation and what the alternatives are. Agr. Syst. 2003, 78, 166-184.
76. Quigley, R.J.; Taylor, L.C. Evaluating health impact assessment. Public Health 2004, 118, 544-552.
77. Clark, W.C.; Majone, G. The critical appraisal of scientific inquiries with policy implications. Sci. Technol. Hum. Val. 1985, 10, 6-19.
78. Hildén, M.; Furman, E.; Kaljonen, M. Views on planning and expectations of SEA: The case of transport planning. Environ. Impact Assess. Rev. 2004, 24, 519-536.
79. Baker, D.C.; McLelland, J.N. Evaluating the effectiveness of British Columbia's environmental assessment process for first nations' participation in mining development. Environ. Impact Assess. Rev. 2003, 23, 581-603.
80. Pohjola, M.V.; Tuomisto, J.T. Openness in participation, assessment, and policy making upon issues of environment and environmental health: a review of literature and recent project results. Environ. Health 2011, 10, 58, doi:10.1186/1476-069X-10-58.
81. Sandström, V.; Tuomisto, J.T.; Majaniemi, S.; Rintala, T.; Pohjola, M.V. Evaluating Effectiveness of Open Assessments on Alternative Biofuel Sources. Available online: www.julkari.fi/bitstream/handle/10024/104443/URN_ISBN_978-952-245-883-4.pdf?sequence=1#page=179 (accessed on 18 June 2013).
82. Wismar, M.; Blau, J.; Ernst, K.; Figueras, J. The Effectiveness of Health Impact Assessment: Scope and Limitations of Supporting Decision-Making in Europe; WHO: Copenhagen, Denmark, 2007.
83. Fischer, T.B.; Gazzola, P. SEA effectiveness criteria-Equally valid in all countries? The case of Italy. Environ. Impact Assess. Rev. 2006, 26, 396-409.
84. Leu, W.; Williams, W.P.; Bark, A.W. Development of an environmental impact assessment evaluation model and its application: Taiwan case study. Environ. Impact Assess. Rev. 1996, 16, 115-133.
85. Pölönen, I.; Hokkanen, P.; Jalava, K. The effectiveness of the Finnish EIA system-What works, what doesn't, and what could be improved? Environ. Impact Assess. Rev. 2011, 31, 120-128.
86. Van Delden, H.; Phyn, D.; Fenton, T.; Huser, B.; Rutledge, D.; Wedderburn, L. User interaction during the development of the Waikato Integrated Scenario Explorer. In Proceedings of the iEMSs Fifth Biennial Meeting: "Modelling for Environment's sake", Ottawa, Canada, 5-8 July 2008; Swayne, D.A., Yang, W., Rizzoli, A., Voinov, A., Filatova, T., Eds.; International Environmental Modelling and Software Society: Ottawa, Canada, 2008.
87. Diez, E.; McIntosh, B.S. A review of the factors which influence the use and usefulness of information systems. Environ. Model. Software 2009, 24, 588-602.
88. Vonk, G.; Geertman, S.; Schot, P. Bottlenecks blocking widespread usage of planning support systems. Environ. Plan. A 2005, 37, 909-924.
89. Larocque, G.R.; Bhatti, J.S.; Ascough, J.C., II; Liu, J.; Luckai, N.; Mailly, D.; Archambault, L.; Gordon, A.M. An analytical framework to assist decision makers in the use of forest ecosystem model predictions. Environ. Model. Software 2011, 26, 280-288.
90. Sterk, B.; van Ittersum, M.K.; Leeuwis, C. How, when, and for what reasons does land use modeling contribute to societal problem solving? Environ. Model. Software 2011, 26, 310-316.
91. Diez, E.; McIntosh, B.S. Organisational drivers for, constraints on and impacts of decision and information support tool use in desertification policy and management. Environ. Model. Software 2011, 26, 317-327.
92. McIntosh, B.S.; Giupponi, C.; Voinov, A.A.; Smith, C.; Matthews, K.B.; Monticino, M.; Kolkman, M.J.; Crossman, N.; van Ittersum, M.; Haase, D, et al. Bridging the Gaps between Design and Use: Developing Tools to Support Environmental Management and Policy. In Environmental Modeling, Software and Decision Support; Jakeman, A.J., Voinov, A.A., Rizzoli, A.E., Chen, S.H., Eds.; Elsevier: Amsterdam, The Netherlands, 2008; pp. 33-48.
93. Siebenhüner, B.; Barth, V. The role of computer modeling in participatory integrated assessments. Environ. Impact Assess. Rev. 2005, 25, 367-389.
94. Inman, D.; Blind, M.; Ribarova, I.; Krause, A.; Roosenschoon, O.; Kassahun, A.; Scholten, H.; Arampatzis, G.; Abrami, G.; McIntosh, B.; Jeffrey, P. Perceived effectiveness of environmental decision support systems in participatory planning: Evidence from small groups of end-users. Environ. Model. Software 2011, 26, 302-309.
95. Dewar, J.A.; Bankes, S.C.; Hodges, J.S.; Lucas, T.; Saunders-Newton, D.K.; Vye, P. Credible Uses of the Distributed Interactive Simulation (DIS) System; RAND: Santa Monica, CA, USA, 1996.
96. Funtowicz, S.O.; Ravetz, J.R. Uncertainty and Quality in Science for Policy; Kluwer Academic Publishers: Dordrecht, the Netherlands, 1990.
97. Food Safety Risk Analysis: A Guide for National Food Safety Authorities; World Health Organization, Food and Agriculture Organization of the United Nations: Rome, Italy, 2006.
98. Wang, R.Y.; Strong, D.M. Beyond accuracy: What data quality means to data consumers. J. Manage. Inform. Syst. 1996, 12, 5-34.
99. Moody, D.; Walsh, P. Measuring the Value of Information: An Asset Valuation Approach. In Proceedings of Seventh European Conference on Information System (ECIS'99), Copenhagen Business School, Frederiksberg, Denmark, 23-25 June 1999.
100. Skyrme, D.J. Ten ways to add value to your business. Manag. Inform. 1994, 1, 20-25.
101. Tongchuay, C.; Praneetpolgrang, P. Knowledge Quality and Quality Metrics in Knowledge Management Systems. In Proceedings of the Fifth International Conference on ELearning for Knowledge-Based Society, Bangkok Metro, Thailand, 11-12 December 2008.
102. Aumann, C.A. Constructing model credibility in the context of policy appraisal. Environ. Model. Software 2011, 26, 258-265.
103. Cash, D.W.; Clark, W.; Alcock, F.; Dickson, N.; Eckley, N.; Jäger, J. Salience, Credibility, Legitimacy and Boundaries: Linking Research, Assessment and Decision Making; Harvard University: Boston, MA, USA, 2002.
104. Wardekker, J.A.; van der Sluijs, J.P.; Janssen, P.H.M.; Kloprogge, P.; Petersen, A.C. Uncertainty communication in environmental assessments: Views from the Dutch science-policy interface. Environ. Sci. Policy 2008, 11, 627-641.
105. Janssen, P.H.M.; Petersen, A.C.; van der Sluijs, J.P.; Risbey, J.S.; Ravetz, J.R. A guidance for assessing and communicating uncertainties. Water Sci. Technol. 2005, 52, 125-131.
106. Covello, V.T.; Peters, R.G.; Wojtecki, J.G.; Hyde, R.C. Risk communication, the west nile virus epidemic, and bioterrorism: Responding to the communication challenges posed by the intentional or unintentional release of a pathogen in an urban setting. J. Urban Health: Bull. N. Y. Acad. Med. 2001, 78, 382-391.
107. Bischof, N.; Eppler, M.J. Caring for clarity in knowledge communication. J. Univers. Comput. Sci. 2011, 17, 1455-1473.
108. O'Faircheallaigh, C. Public participation and environmental impact assessment: Purposes, implications, and lessons for public policy making. Environ. Impact Assess. Rev. 2010, 30, 19-27.
109. OECD. Stakeholder Involvement Techniques-Short Guide and Annotated Bibliography. Available online: www.oecd-nea.org/rwm/reports/2004/nea5418-stakeholder.pdf (accessed on 18 June 2013).
110. MNP. Stakeholder Participation Guide for the Netherlands Environmental Assessment Agency: Main Document, Practice Guide & Checklist; Netherlands Environmental Assessment Agency and Radboud University: Nijmegen, The Netherlands, 2008.
111. Rosenhead, J.; Mingers, J. Rational Analysis for a Problematic World Revisited: Problem Structuring Methods For complexity, Uncertainty and Conflict; Wiley: Chichester, UK, 2001.
112. University of Cambridge. Institute for Manufacturing. Decision Support Tools (Website). Available online: www.ifm.eng.cam.ac.uk/research/dstools/ (accessed on 18 June 2013).
113. Narayan, S.M.; Corcoran-Perry, S.; Drew, D.; Hoyman, K.; Lewis, M. Decision analysis as a tool to support an analytical pattern-of-reasoning. Nurs. Health Sci. 2003, 5, 229-243.
114. Harris, G. Integrated assessment and modeling: An essential way of doing science. Environ. Model. Software 2002, 17, 201-207.
115. Pohjola, M.V.; Pohjola, P.; Paavola, S.; Bauters, M.; Tuomisto, J.T. Pragmatic Knowledge Services. J. Univers. Comput. Sci. 2011, 17, 472-497.
116. Koivisto, J.; Pohjola, P. Practices, modifications and generativity-REA: A practical tool for managing the innovation processes of practices. Syst. Signs Actions 2012, 5, 100-116.
117. Tijhuis, M.J.; Pohjola, M.V.; Gunnlaugsdóttir, H.; Kalogeras, N.; Leino, O.; Luteijn, J.M.; Magnússon, S.H.; Odekerken, G.; Poto, M.; Tuomisto, J.T, et al. Looking beyond Borders: Integrating best practices in benefit-risk analysis into the field of food and nutrition. Food Chem. Toxicol. 2012, 50, 77-93.
118. Miles, S.B. Towards Policy Relevant Environmental Modeling: Contextual Validity and Pragmatic Models. U.S. Department of the Interior, U.S. Geological Survey. Available online: www.geopubs.wr.usgs.gov/open-file/of00-401/of00-401.pdf (accessed on 18 June 2013).
119. Oreskes, N.; Schrader-Frechette, K.; Belitz, K. Verification, validation, and confirmation of numerical models in the earth sciences. Science 1994, 263, 641-646.