Uncertainty assessment in ecosystem services analyses: Seven challenges and practical responses

Ecosystem Services

Volumn 24, Issue , 2017, Pages 1-15

  Hamel, Perrine ^a   Bryant, Benjamin P ^a  

^a STANFORD UNIVERSITY   (United States)

Author keywords

Best practice; Ecosystem services; Guidance; Modeling; Uncertainty assessment

Indexed keywords

EID: 85013339565     PISSN: 22120416     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.ecoser.2016.12.008     Document Type: Article

References (98)

1. Andrew, M.E., Wulder, M.A., Nelson, T.A., Coops, N.C., Spatial data, analysis approaches, and information needs for spatial ecosystem service assessments: a review. GIScience Remote Sens. 52:3 (2015), 344–373, 10.1080/15481603.2015.1033809.
2. Arnold, J.G., Srinivasan, R., Muttiah, R.S., Williams, J.R., Large-area hydrologic modeling and assessment: part I. Model development. J. Am. Water Resour. Assoc. 34:1 (1998), 73–89.
3. Aumann, C.A., Constructing model credibility in the context of policy appraisal. Environ. Model. Softw. 26:3 (2011), 258–265, 10.1016/j.envsoft.2009.09.006.
4. Bennett, G, Carroll, N., 2014. Gaining Depth: State of Watershed Investment 2014 Available at 〈 www.ecosystemmarketplace.com/reports/sowi2014〉.
5. Birkhofer, K., Diehl, E., Andersson, J., Ekroos, J., Früh-Müller, A., Machnikowski, F., Mader, V.L., Nilsson, L., Sasaki, K., Rundl[oe]f, M., et al. Ecosystem services: current challenges and opportunities for ecological research. Front. Ecol. Evol., 2, 2015, 10.3389/fevo.2014.00087.
6. Braat, L.C., de Groot, R., The ecosystem services agenda: bridging the worlds of natural science and economics, conservation and development, and public and private policy. Ecosystem Services 1:1 (2012), 4–15, 10.1016/j.ecoser.2012.07.011.
7. Brewer, C.A., Hatchard, G.W., Harrower, M.A., ColorBrewer in Print: A Catalog of Color Schemes for Maps. Cartogr. Geogr. Inform. 30:1 (2003), 5–32, 10.1559/152304003100010929.
8. Brown, C., Ghile, Y., Laverty, M., Li, K., Decision scaling: linking bottom-up vulnerability analysis with climate projections in the water sector. Water Resour. Res., 48(9), 2012 〈http://doi.wiley.com/10.1029/2011WR011212〉 (accessed 10.08.15).
9. Brugnach, M., Pahl-Wostl, C., Lindenschmidt, K.E., JAEB, Janssen, Filatova, T., Mouton, A., Holtz, G., van der Keur, P., Gaber, N., Complexity and Uncertainty: Rethinking the Modelling Activity. 2008, Elsevier (DOI: S1574101X08006042).
10. Brugnach, M., Pahl-Wostl, C., Lindenschmidt, K.E., Janssen, JAEB, Filatova, T., Mouton, A., Holtz, G., van der Keur, P., Gaber, N., Environmental Modelling, Software and Decision Support. Elsevier, 2008, 10.1016/S1574-101X(08)00604-2.
11. Bryant, B.P., Lempert, R.J., Thinking inside the box: a participatory, computer-assisted approach to scenario discovery. Technol. Forecast. Soc. Change 77:1 (2010), 34–49, 10.1016/j.techfore.2009.08.002.
12. Bryant, B.P., Westerling, A.L., Scenarios for future wildfire risk in California: links between changing demography, land use, climate, and wildfire. Environmetrics 25:6 (2014), 454–471, 10.1002/env.2280.
13. Cash, D.W., Clark, W.C., Alcock, F., Dickson, N.M., Eckley, N., Guston, D.H., Ja, J., Mitchell, R.B., Knowledge systems for sustainable development. Proc. Natl. Acad. Sci. USA, 100, 2003, 14.
14. Christopher Frey, H., Patil, S.R., Identification and review of sensitivity analysis methods. Risk Anal. 22:3 (2002), 553–578, 10.1111/0272-4332.00039.
15. Clark, W.C., Tomich, T.P., van Noordwijk, M., Dickson, N.M., Catacutan, D., Guston, D., McNieClark, E., 2010. Toward a General Theory of Boundary Work: Insights from the CGIAR's Natural Resource Management Programs.
16. Costanza, R., de Groot, R., Sutton, P., van der Ploeg, S., Anderson, S.J., Kubiszewski, I., Farber, S., Turner, R.K., Changes in the global value of ecosystem services. Glob. Environ. Change 26 (2014), 152–158 (DOI: S0959378014000685).
17. Cowling, R.M., Egoh, B., Knight, A.T., O'Farrell, P.J., Reyers, B., Rouget, M., Roux, D.J., Welz, A., Wilhelm-Rechman, A., An operational model for mainstreaming ecosystem services for implementation. Proc. Natl. Acad. Sci. USA 105:28 (2008), 9483–9488, 10.1073/pnas.0706559105.
18. Easterly, W., How the millennium development goals are unfair to Africa. World Dev. 37:1 (2009), 26–35 Available at 〈http://ssrn.com/abstract=2066805〉.
19. Eigenbrod, F., Armsworth, P.R., Anderson, B.J., Heinemeyer, A., Gillings, S., Roy, D.B., Thomas, C.D., Gaston, K.J., The impact of proxy-based methods on mapping the distribution of ecosystem services. J. Appl. Ecol. 47:2 (2010), 377–385, 10.1111/j.1365-2664.2010.01777.x.
20. EPA, 2009. Guidance on the Development, Evaluation, and Application of Environmental Models. EPA/100/K-09/003. Washington, D.C. Available at: 〈www.epa.gov/modeling/environmental-modeling-guidance-documents〉.
21. Feizizadeh, B., Jankowski, P., Blaschke, T., A GIS based spatially-explicit sensitivity and uncertainty analysis approach for multi-criteria decision analysis. Comput. Geosci. 64 (2014), 81–95, 10.1016/j.cageo.2013.11.009.
22. Fisher, P., Comber, A., Wadsworth, R., 2006. Approaches to Uncertainty in Spatial Data. In Fundamentals of Spatial Data Quality, Devillers, R., Jeansoulin, R., (eds).London, UK; 43–59. 〈 http://dx.doi.org/10.1002/9780470612156.ch3〉.
23. Gao, L., Bryan, B.A., Nolan, M., Connor, J.D., Song, X., Zhao, G., Robust global sensitivity analysis under deep uncertainty via scenario analysis. Environ. Model. Softw. 76 (2016), 154–166, 10.1016/j.envsoft.2015.11.001.
24. Goldman-Benner, R.L., Benitez, S., Boucher, T., Calvache, A., Daily, G., Kareiva, P., Kroeger, T., Ramos, A., Water funds and payments for ecosystem services: practice learns from theory and theory can learn from practice. Oryx 46:1 (2012), 55–63.
25. Grêt-Regamey, A., Brunner, S., Altwegg, J., Christen, M., Bebi, P., Integrating expert knowledge into mapping ecosystem services trade- offs for sustainable forest management. Soc. Ecol., 18(3), 2013 105751/ES-05800-180334).
26. Groves, D.G., Lempert, R.J., A new analytic method for finding policy-relevant scenarios. Glob. Environ. Change-Hum. Policy Dimens. 17:1 (2007), 73–85, 10.1016/j.gloenvcha.2006.11.006.
27. Guillaume, J.H.A., Arshad, M., Jakeman, A.J., Jalava, M., Kummu, M., Robust discrimination between uncertain management alternatives by iterative reflection on crossover point scenarios: principles, design and implementations. Environ. Model. Softw. 83 (2016), 326–343, 10.1016/j.envsoft.2016.04.005.
28. Guillaume, J.H.A., Qureshi, M.E., Jakeman, A.J., A structured analysis of uncertainty surrounding modeled impacts of groundwater-extraction rules. Hydrogeol. J. 20:5 (2012), 915–932, 10.1007/s10040-012-0864-0.
29. Guswa, A., Brauman, K.A., Brown, C., Hamel, P., Keeler, B.L., Sayre, S.S., Ecosystem services: challenges and opportunities for hydrologic modeling to support decision making. Water Resour. Res. 50:5 (2014), 4535–4544, 10.1002/2014WR015497.
30. Haasnoot, M., Kwakkel, J.H., Walker, W.E., ter Maat, J., Dynamic adaptive policy pathways: a method for crafting robust decisions for a deeply uncertain world. Glob. Environ. Change 23:2 (2013), 485–498, 10.1016/j.gloenvcha.2012.12.006.
31. Hadka, D., Herman, J., Reed, P., Keller, K., An open source framework for many-objective robust decision making. Environ. Model. Softw. 74 (2015), 114–129, 10.1016/j.envsoft.2015.07.014.
32. Halpern, B.S., Longo, C., Hardy, D., McLeod, K.L., Samhouri, J.F., Katona, S.K., Kleisner, K., Lester, S.E., O/’Leary, J., Ranelletti, M., et al. An index to assess the health and benefits of the global ocean. Nature 488:7413 (2012), 615–620, 10.1038/nature11397.
33. Hamarat, C., Kwakkel, J.H., Pruyt, E., Loonen, E.T., An exploratory approach for adaptive policymaking by using multi-objective robust optimization. Simul. Model. Pract. Theory 46 (2014), 25–39, 10.1016/j.simpat.2014.02.008.
34. Herman, J.D., Reed, P.M., Zeff, H.B., Characklis, G.W., How should robustness be defined for water systems planning under change?. J. Water Resour. Plan. Manag., 2015, 4015012, 10.1061/(ASCE)WR0.1943-5452.0000509.
35. Hijmans, R.J., Cameron, S.E., Parra, J.L., Jones, P.G., Jarvis, A., Very high resolution interpolated climate surfaces for global land areas. Int. J. Climatol. 25:15 (2005), 1965–1978, 10.1002/joc.1276.
36. Hou, Y., Burkhard, B., Müller, F., Uncertainties in landscape analysis and ecosystem service assessment. J. Environ. Manag. 127 (2013), S117–S131, 10.1016/j.jenvman.2012.12.002.
37. Hsu, A., Emerson, J., Levy, M., Sherbinin, A., de, Johnson, L., Malik, O., Schwartz, J., Jaiteh, M., The 2014 Environmental Performance Index. 2014, Haven, CT 〈www.epi.yale.edu〉.
38. Jacobs, S., Burkhard, B., Van Daele, T., Staes, J., Schneiders, A., ‘The Matrix Reloaded’: a review of expert knowledge use for mapping ecosystem services. Ecol. Model. 295 (2015), 21–30 〈http://www.sciencedirect.com/science/article/pii/S0304380014004141〉 (accessed 22.09.14).
39. Jakeman, A.J., Chen, S.H., Rizzolib, A.E., Voinov, A.A., Modelling and Software as Instruments for Advancing Sustainability. Environmental Modelling, Software and Decision Support, 2008, Elsevier, 1–13, 10.1016/S1574-101X(08)00601-7.
40. Jakeman, A.J., Letcher, R.A., Norton, J.P., Ten iterative steps in development and evaluation of environmental models. Environ. Model. Softw. 21:5 (2006), 602–614, 10.1016/j.envsoft.2006.01.004.
41. Kahneman, D., Tversky, A., Prospect theory: an analysis of decision under risk. Économ.: J. Econom. Soc., 47(2), 1979 〈http://www.jstor.org/stable/10.2307/1914185〉 (accessed 26.11.12).
42. Kasprzyk, J.R., Nataraj, S., Reed, P.M., Lempert, R.J., Many objective robust decision making for complex environmental systems undergoing change. Environ. Model. Softw. 42 (2013), 55–71.
43. Kaul, S., Boyle, K.J., Kuminoff, N.V., Parmeter, C.F., Pope, J.C., What can we learn from benefit transfer errors? Evidence from 20 years of research on convergent validity. J. Environ. Econ. Manag. 66:1 (2013), 90–104, 10.1016/j.jeem.2013.03.001.
44. Landuyt, D., Broekx, S., D'hondt, R., Engelen, G., Aertsens, J., Goethals, P.L.M., A review of Bayesian belief networks in ecosystem service modelling. Environ. Model. Softw. 46 (2013), 1–11, 10.1016/j.envsoft.2013.03.011.
45. Laniak, G.F., Olchin, G., Goodall, J., Voinov, A., Hill, M., Glynn, P., Whelan, G., Geller, G., Quinn, N., Blind, M., et al. Integrated environmental modeling: a vision and roadmap for the future. Environ. Model. Softw. 39 (2013), 3–23, 10.1016/j.envsoft.2012.09.006.
46. Leamer, E., Sensitivity analyses would help. Am. Econ. Rev. 75:3 (1985), 308–313.
47. Lempert, R.J., Scenarios that illuminate vulnerabilities and robust responses. Clim. Change 117:4 (2012), 627–646, 10.1007/s10584-012-0574-6.
48. Lempert, R.J., Groves, D.G., Popper, S.W., Bankes, S.C., A general, analytic method for generating robust strategies and narrative scenarios. Manag. Sci. 52:4 (2006), 514–528, 10.1287/mnsc.1050.0472.
49. Ligmann-Zielinska, A., Spatially-explicit sensitivity analysis of an agent-based model of land use change. Int. J. Geogr. Inf. Sci. 27:9 (2013), 1764–1781, 10.1080/13658816.2013.782613.
50. Ligmann-Zielinska, A., Jankowski, P., A Framework for Sensitivity Analysis in Spatial Multiple Criteria Evaluation. Cova, T.J., Miller, H.J., Beard, K., Frank, A.U., Goodchild, M., (eds.) GIScience, 2008, Springer-Verlag, Berlin, Heidelberg, 217–233.
51. Lin, G., Engel, D., Esliner, P., Survey and evaluate [sic] Uncertainty Quantification Methodologies. 2012, Richland, Washington.
52. Mastrandrea, M.D, Field, C.B, Stocker, T.F, Edenhofer, O., Ebi, K.L., Frame, D.J., Held, H., Kriegler, E., Mach, K.J., Matschoss, P.R., et al. 2010. Guidance Note for Lead Authors of the IPCC Fifth Assessment Report on Consistent Treatment of Uncertainties. Intergovernmental Panel on Climate Change (IPCC). Available at: http://www.ipcc.ch.
53. Matott, L.S., Babendreier, J.E., Purucker, S.T., Evaluating uncertainty in integrated environmental models: a review of concepts and tools. Water Resour. Res., 45(6), 2009, 10.1029/2008WR007301.
54. McIntosh, B.S., Ascough, J.C., Twery, M., Chew, J., Elmahdi, A., Haase, D., Harou, J.J., Hepting, D., Cuddy, S., Jakeman, A.J., et al. Environmental decision support systems (EDSS) development – Challenges and best practices. Environ. Model. Softw. 26:12 (2011), 1389–1402, 10.1016/j.envsoft.2011.09.009.
55. McIntosh, B.S., Giupponi, C., Smith, C., Matthews, K.B., Monticino, M., Kolkman, M.J., Crossman, N., van Ittersum, M., Haase, D., Haase, A., et al. Bridging the Gaps Between Design and Use: Developing Tools to Support Environmental Management and Policy. Environmental Modelling, Software and Decision Support, 2008, Elsevier, 33–48 (DOI: S1574101X08006030).
56. McKenzie, E., Rosenthal, A., Bernhardt, J., Girvetz, E., Kovacs, K., Olwero, N., ToftJ., 2012. Developing Scenarios to Assess Ecosystem Service Tradeoffs: Guidance and Case Studies for InVEST Users. Washington, D.C. Available at: 〈http://www.naturalcapitalproject.org/pubs/ScenariosGuide.pdf〉.
57. Morgan, M.G., Use (and abuse) of expert elicitation in support of decision making for public policy. Proc. Natl. Acad. Sci. USA 111:20 (2014), 7176–7184, 10.1073/pnas.1319946111.
58. Nahuelhual, L., Laterra, P., Villarino, S., Mastrángelo, M., Carmona, A., Jaramillo, A., Barral, P., Burgos, N., Mapping of ecosystem services: missing links between purposes and procedures. Ecosyst. Serv. 13 (2015), 162–172, 10.1016/j.ecoser.2015.03.005.
59. Olea, R.A., A Practical Primer on Geostatistics. 2009, USGS Open-File Report 2009-1103.
60. Pannell, D.J., Sensitivity analysis of normative economic models: theoretical framework and practical strategies. Agr. Econ. 16:2 (1997), 139–152, 10.1016/S0169-5150(96)01217-0.
61. Pappenberger, F., 2006. Implementation plan for library of tools for uncertainty evaluation Available at: 〈www.floodrisk.org.uk〉.
62. Pappenberger F, Beven. 2006. Ignorance is bliss: Or seven reasons not to use uncertainty analysis. 42 (December 2005): 1–8 〈 http://dx.doi.org/10.1029/2005WR004820〉.
63. Petersen, A.C., Janssen, P.H.M., van der Sluijs, J.P., Risbey, J.S., Ravetz, J.R., Wardekker, J.A., Hughes Martinson, H. 2013. Guidance for Uncertainty Assessment and Communication, 2nd edition.
64. Pianosi, F., Beven, K., Freer, J., Hall, J.W., Rougier, J., Stephenson, D.B., Wagener, T., Sensitivity analysis of environmental models: a systematic review with practical workflow. Environ. Model. Softw. 79 (2016), 214–232, 10.1016/j.envsoft.2016.02.008.
65. Van der Ploeg, S., RS de, G., 2010. The TEEB Valuation Database – a searchable database of 1310 estimates of monetary values of ecosystem services. Wageningen, The Netherlands.
66. Polasky, S., Carpenter, S., Folke, C., Keeler, B., Decision-making under great uncertainty: environmental management in an era of global change. Trends Ecol. Evol., 2011 DOI: S0169534711001157.
67. Polasky, S., Tallis, H., Reyers, B., Setting the bar: standards for ecosystem services. Proc. Natl. Acad. Sci. USA 112:24 (2015), 7356–7361, 10.1073/pnas.1406490112.
68. Pontius, R.G., Santacruz, A., Quantity, exchange, and shift components of difference in a square contingency table. Int. J. Remote Sens. 35:21 (2014), 7543–7554, 10.1080/2150704X.2014.969814.
69. Pontius, R.G. Jr, Millones, M., Death to Kappa: birth of quantity disagreement and allocation disagreement for accuracy assessment. Int. J. Remote Sens. 32:15 (2011), 4407–4429, 10.1080/01431161.2011.552923.
70. Ranganathan, J., Raudsepp-Hearn, C., Lucas, N., Irwin, F., Zurek, M., Al, E., Ecosystem services: a guide for decision makers. World Resour. Inst., 2008 (Available at) 〈http://www.wri.org/publication/ecosystem-services〉.
71. Refsgaard, J.C., van der Sluijs, J.P., Højberg, A.L., Vanrolleghem, P.A., Uncertainty in the environmental modelling process – A framework and guidance. Environ. Model. Softw. 22:11 (2007), 1543–1556, 10.1016/j.envsoft.2007.02.004.
72. Retchless, D.P., Brewer, C.A., Guidance for representing uncertainty on global temperature change maps. Int. J. Climatol. 36:3 (2016), 1143–1159, 10.1002/joc.4408.
73. Richardson, L., Loomis, J., Kroeger, T., Casey, F., The role of benefit transfer in ecosystem service valuation. Ecol. Econ. 115 (2015), 51–58, 10.1016/j.ecolecon.2014.02.018.
74. Rosa, I.M.D., Ahmed, S.E., Ewers, R.M., The transparency, reliability and utility of tropical rainforest land-use and land-cover change models. Glob. Change Biol. 20:6 (2014), 1707–1722, 10.1111/gcb.12523.
75. Rosario Partidario, M., Sheate, W.R., Knowledge brokerage - potential for increased capacities and shared power in impact assessment. Environ. Impact Assess. Rev. 39 (2013), 26–36, 10.1016/j.eiar.2012.02.002.
76. Rosenberger, R.S., Stanley, T.D., Measurement, generalization, and publication: sources of error in benefit transfers and their management. Ecol. Econ. 60:2 (2006), 372–378, 10.1016/j.ecolecon.2006.03.018.
77. Ruckelshaus, M., McKenzie, E., Tallis, H., Guerry, A., Daily, G., Kareiva, P., Polasky, S., Ricketts, T., Bhagabati, N., Wood, S.A., et al. Notes from the field: lessons learned from using ecosystem service approaches to inform real-world decisions. Ecol. Econ. 115 (2015), 11–21, 10.1016/j.ecolecon.2013.07.009.
78. Saltelli, A., Annoni, P., How to avoid a perfunctory sensitivity analysis. Environ. Model. Softw. 25:12 (2010), 1508–1517, 10.1016/j.envsoft.2010.04.012.
79. Saltelli, A., Vidoni, D., Mascherini, M., Recommended practices in global sensitivity analysis. Baveye, P.C., et al. (eds.) Uncertainties in Environmental Modelling and Consequences for Policy Making, 2009, Springer Science + Business Media BV, 183–202.
80. Schägner, J., Brander, L., Maes, J., Hartje, V., 2013. Mapping ecosystem services' values: Current practice and future prospects. Ecosyst. Serv., 4: 33–46, Available at: 〈http://www.sciencedirect.com/science/article/pii/S2212041613000120〉(accessed 09.07.14).
81. Schulp, C.J.E., Burkhard, B., Maes, J., Van, Vliet J., Verburg, P.H., Uncertainties in ecosystem service maps: a comparison on the European scale. PLoS One, 9(10), 2014, e109643, 10.1371/journal.pone.0109643.
82. Sense About Science, 2013. Making Sense of Uncertainty: Why uncertainty is part of science: 28 Available at: 〈 http://www.senseaboutscience.org/resources.php/127/making-sense-of-uncertainty〉.
83. Seppelt, R., Dormann, C.F., Eppink, F.V., Lautenbach, S., Schmidt, S., A quantitative review of ecosystem service studies: approaches, shortcomings and the road ahead. J. Appl. Ecol. 48:3 (2011), 630–636, 10.1111/j.1365-2664.2010.01952.x.
84. Shi, W., Principles of Modeling Uncertainties in Spatial Data and Spatial Analysis. 2010, Taylor & Francis Group, Boca Raton, FL.
85. Simpson, R.D., The ‘Ecosystem Service Framework’: A Critical Assessment. 2011, Publishing Services Section, UNON, Nairobi, Kenya.
86. van der Sluijs, J.P., Janssen, P.H.M., Petersen, A.C., Kloprogge, P., Risbey, J.S., Tuinstra, W., Ravetz, J.R., 2004. Tool Catalogue for Uncertainty Assessment. RIVM/MNP Guidance for Uncertainty Assessment and Communication Series.
87. Song, X., Zhang, J., Zhan, C., Xuan, Y., Ye, M., Xu, C., Global sensitivity analysis in hydrological modeling: review of concepts, methods, theoretical framework, and applications. J. Hydrol. 523 (2015), 739–757 DOI: S0022169415001249.
88. Stirling, A., Keep it complex. Nature 468:7327 (2010), 1029–1031.
89. TNC, 2015. Upper Tana-NairobiWater Fund Business Case. Version2. Nairobi, Kenya.
90. Toman M, Griffin J, Lempert RJ. 2008. Impacts on U.S. Energy Expenditures and Greenhouse-Gas Emissions of Increasing Renewable Energy Use — Appendix A. Santa Monica, Calif.
91. Turnpenny, J., Jones, M., Lorenzoni, I., Where now for post-normal science?: A critical review of its development, definitions, and uses. Sci. Technol. Hum. Values 36:3 (2011), 287–306, 10.1177/0162243910385789.
92. Visser, H., Petersen, A.C., Beusen, A.H.W., Heuberger, P.S.C., Janssen, P.H.M., 2005. Checklist for uncertainty in spatial information and visualising spatial uncertainty. RIVM/MNP Guidance for Uncertainty Assessment and Communication Series.
93. Vogl, A., Tallis, H., Douglass, J., Sharp, R., Wolny, S., Viega, F., Benitez, S., Leon, J., Game, E., Petry, P., et al.2015. Resource Investment Optimization System: Introduction & Theoretical Documentation Available at: 〈www.naturalcapitalproject.org〉.
94. Walker, W., Haasnoot, M., Kwakkel, J., Adapt or perish: a review of planning approaches for adaptation under deep uncertainty. Sustainability 5:3 (2013), 955–979, 10.3390/su5030955.
95. Walker, W.E., Harremoës, P., Rotmans, J., van der Sluijs, J.P., van Asselt, M.B.A., Janssen, P., Krayer von Krauss, M.P., Defining uncertainty: a conceptual basis for uncertainty management in model-based decision support. Integr. Assess. 4:1 (2003), 5–17, 10.1076/iaij.4.1.5.16466.
96. Warmink, J.J., Janssen, J.A.E.B., Booij, M.J., Krol, M.S., Identification and classification of uncertainties in the application of environmental models. Environ. Model. Softw. 25:12 (2010), 1518–1527, 10.1016/j.envsoft.2010.04.011.
97. Wilkinson, Leland. 2005. The Grammar of Graphics. Springer-Verlag: New York.
98. Yoe, C., Skaggs, L., Risk and Uncertainty Procedures for the Evaluation of Environmental Outputs. 1997, US Army Corps of Engineers Available at: IWR Report 97-R-7.