Facilitating parameter estimation and sensitivity analysis of agent-based models: A cookbook using NetLogo and R

JASSS

Volumn 17, Issue 3, 2014, Pages 11-

  Thiele, Jan C ^a   Kurth, Winfried ^a   Grimm, Volker ^b  

^a Department of Ecological Modelling ^*  (Germany)

^b HELMHOLTZ CENTRE FOR ENVIRONMENTAL RESEARCH UFZ   (Germany)

Author keywords

Agent based model; Inverse modeling; Model calibration; NetLogo; Parameter fitting; Sensitivity analysis

Indexed keywords

EID: 84903731143     PISSN: 14607425     EISSN: None     Source Type: Journal    
DOI: 10.18564/jasss.2503     Document Type: Article

References (150)

1. AARTS, E., & Korst, J. (1989). Simulated Annealing and Boltzmann Machines. Chichester: Wiley.
2. ARIFIN, S., Madey, G., & Collins, F. (2013). Examining the Impact of Larval Source Management and Insecticide-Treated Nets Using a Spatial Agent-Based Model of Anopheles Gambiae and a Landscape Generator Tool. Malaria Journal, 12(1), 290.
3. ASCOUGH IL, J., Green, T., Ma, L., & Ahjua, L. (2005). Key Criteria and Selection of Sensitivity Analysis Methods Applied to Natural Resource Models (pp. 2463-2469). Presented at the Modsim 2005 International Congress on Modeling and Simulation, Modeling and Simulation Society of Australia and New Zealand, Melbourne. http://www.mssanz.org.au/modsim05/papers/ascough_2.pdf Archived at: http://www.webcitation.org/6MlOGzavB
4. BACK, T. (1996). Evolutionary Algorithms in Theory and Practice: Evolution Strategies, Evolutionary Programming, Genetic Algorithms. Oxford: Oxford University Press.
5. BAKSHY, E., & Wilensky, U. (2007). Turtle Histories and Alternate Universes; Exploratory Modeling with NetLogo and Mathematica. In M. North, C. Macal, & D. Sallach (Eds.), Proceedings of the Agent 2007 Conference on Complex Interaction and Social Emergence (pp. 147-158). L'Argonne National Laboratory and Northwestern University.
6. BANSAL, R. (2005). Optimization Methods for Electric Power Systems: An Overview. International Journal of Emerging Electric Power Systems, 2(1).
7. BAR MASSADA, A., & Carmel, Y. (2008). Incorporating Output Variance in Local Sensitivity Analysis for Stochastic Models. Ecological Modelling, 213(3-4), 463-467.
8. BEAUMONT, M. (2010). Approximate Bayesian Computation in Evolution and Ecology. Annual Review of Ecology, Evolution, and Systematics, 41(1), 379-406.
9. BEAUMONT, M., Zhang, W., & Balding, D. (2002). Approximate Bayesian Computation in Population Genetics. Genetics, 162(4), 2025-2035.
10. BIETHAHN, J., Lackner, A., & Range, M. (2004). Optimierung und Simulation. München: Oldenbourg.
11. BLOWER, S., & Dowlatabadi, H. (1994). Sensitivity and Uncertainty Analysis of Complex Models of Disease Transmission: An HIV Model, as an Example. International Statistical Review/Revue Internationale de Statistique, 62(2), 229-243.
12. BONNANS, J.-F., Gilbert, J., Lemarechal, C., & Sagastizábal, C. (2006). Numerical Optimization: Theoretical and Practical Aspects (2nd ed.). Berlin/Heidelberg: Springer.
13. BOX, G. E. P., Hunter, W. G., & Hunter, J. S. (1978). Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building. Wiley series in probability and mathematical statistics. New York: Wiley.
14. BYRD, R., Lu, P., Nocedal, J., & Zhu, C. (1995). A Limited Memory Algorithm for Bound Constrained Optimization. SIAM Journal on Scientific Computing, 16(5), 1190-1208.
15. CALVEZ, B., & Hutzler, G. (2006). Automatic Tuning of Agent-Based Models Using Genetic Algorithms. In J. Sichman & L. Antunes (Eds.), Multi-Agent Based Simulation VI, Lecture Notes in Computer Science (pp. 41-57). Berlin/Heidelberg: Springer.
16. CAMPOLONGO, F., Cariboni, J., & Saltelli, A. (2007). An Effective Screening Design for Sensitivity Analysis of Large Models. Environmental Modelling & Software, 22(10), 1509-1518.
17. CAMPOLONGO, F., Kleijnen, J., & Andres, T. (2000). Screening Methods. In A. Saltelli, K. Chan, & E. Scott (Eds.), Sensitivity Analysis (pp. 65-80). Chichester: Wiley.
18. CAMPOLONGO, F., & Saltelli, A. (2000). Design of Experiments. In A. Saltelli, K. Chan, & E. Scott (Eds.), Sensitivity Analysis (pp. 51-63). Chichester: Wiley.
19. CAMPOLONGO, F., Saltelli, A., Sorensen, T., & Tarantola, S. (2000). Hitchhiker's Guide to Sensitivity Analysis. In A. Saltelli, K. Chan, & E. Scott (Eds.), Sensitivity Analysis (pp. 17-47). Chichester: Wiley.
20. CAMPOLONGO, F., Tarantola, S., & Saltelli, A. (1999). Tackling Quantitatively Large Dimensionality Problems. Computer Physics Communications, 117(1/2), 75-85.
21. CARIBONI, J., Gatelli, D., Liska, R., & Saltelli, A. (2007). The Role of Sensitivity Analysis in Ecological Modelling. Ecological Modelling, 203(1/2), 167-182.
22. CARNELL, R. (2012). lhs: Latin Hypercube Samples. http://cran.r-project.org/web/packages/lhs/ Archived at: http://www.webcitation.org/6H9MePfSw
23. CERDEIRA, J., Silva, P., Cadima, J., & Minhoto, M. (2012). subselect: Selecting Variable Subsets. http://cran.r-project.org/web/packages/subselect Archived at: http://www.webcitation.org/6H9MsDvZK
24. CHAN, K., Tarantola, S., Saltelli, A., & Sobol', I. (2000). Variance-Based Methods. In A. Saltelli, K. Chan, & E. Scott (Eds.), Sensitivity Analysis (pp. 167-197). Chichester: Wiley.
25. CONFALONIERI, R., Bellocchi, G., Bregaglio, S., Donatelli, M., & Acutis, M. (2010). Comparison of Sensitivity Analysis Techniques: A Case Study With the Rice Model WARM. Ecological Modelling, 221(16), 1897-1906.
26. COURNÈDE, P.-H., Chen, Y., Wu, Q., Baey, C., & Bayol, B. (2013). Development and Evaluation of Plant Growth Models: Methodology and Implementation in the PYGMALION platform. Mathematical Modelling of Natural Phenomena, 8(4), 112-130.
27. CRAWLEY, M. (2005). Statistics: An Introduction Using R. Chichester: Wiley.
28. CSILLERY, K., Blum, M., & Francois, O. (2012). abc: Tools for Approximate Bayesian Computation (ABC). http://cran.r-project.org/web/packages/abc Archived at: http://www.webcitation.org/6H9Mx9bQ1
29. CSILLERY, K., Francois, O., & Blum, M. (2012). Approximate Bayesian Computation (ABC) in R: A Vignette. http://cran.r-project.org/web/packages/abc/vignettes/abcvignette.pdf Archived at: http://www.webcitation.org/6H9N0oSlv
30. CUKIER, R., Fortuin, C., Shuler, K., Petschek, A., & Schaibly, J. (1973). Study of the Sensitivity of Coupled Reaction Systems to Uncertainties in Rate Coefficients. I Theory. The Journal of Chemical Physics, 59, 3873-3878.
31. DALGAARD, P. (2008). Introductory Statistics with R (2nd ed.). New York: Springer.
32. DANCIK, G. (2009). mlegpFULL: Maximum Likelihood Estimates of Gaussian Processes. (available from author on request, see http://cran.r-project.org/web/packages/mlegp/index.html Archived at: http://www.webcitation.org/6MlT9MYsu)
33. DANCIK, G., Jones, D., & Dorman, K. (2010). Parameter Estimation and Sensitivity Analysis in an Agent-Based Model of Leishmania Major Infection. Journal of Theoretical Biology, 262(3), 398-412.
34. DUBOZ, R., Versmisse, D., Travers, M., Ramat, E., & Shin, Y.-J. (2010). Application of an Evolutionary Algorithm to the Inverse Parameter Estimation of an Individual-Based Model. Ecological Modelling, 221(5), 840-849.
35. DUPUY, D., & Helbert, C. (2011). DiceEval: Construction and Evaluation of Metamodels. http://CRAN.R-project.org/package=DiceEval Archived at: http://www.webcitation.org/6MlOvl3GE
36. FIELDING, M. (2011). MCMChybridGP: Hybrid Markov Chain Monte Carlo Using Gaussian Processes. http://cran.r-project.org/web/packages/MCMChybridGP/ Archived at: http://www.webcitation.org/6H9N6EAbv
37. FLETCHER, R. (1987). Practical Methods of Optimization (2nd ed.). Chichester: Wiley.
38. FROST, C., Hygnstrom, S., Tyre, A., Eskridge, K., Baasch, D., Boner, J., Clements, G., et al. (2009). Probabilistic Movement Model with Emigration Simulates Movements of Deer in Nebraska, 1990-2006. Ecological Modelling, 220(19), 2481-2490.
39. GAN, Y., Duan, Q., Gong, W., Tong, C., Sun, Y., Chu, W., Ye, A., et al. (2014). A Comprehensive Evaluation of Various Sensitivity Analysis Methods: A Case Study With a Hydrological Model. Environmental Modelling & Software, 51, 269-285.
40. GEYER, C., & Johnson, L. (2013). mcmc: Markov Chain Monte Carlo. http://cran.r-project.org/web/packages/mcmc/ Archived at: http://www.webcitation.org/6H9NC0Gow
41. GILBERT, N. (2007). Agent-Based Models. Los Angeles: Sage.
42. GINOT, V., Gaba, S., Beaudouin, R., Aries, F., & Monod, H. (2006). Combined Use of Local and ANOVA-Based Global Sensitivity Analyses for the Investigation of a Stochastic Dynamic Model: Application to the Case Study of an Individual-Based Model of a Fish Population. Ecological Modelling, 193(3/4), 479-491.
43. GRAMACY, R., & Taddy, M. (2013). Tgp: Bayesian Treed Gaussian Process Models. http://cran.r-project.org/web/packages/tgp/ Archived at: http://www.webcitation.org/6H9NH3MZD
44. GRIMM, V. (2008). Individual-Based Models. In S. Jørgensen (Ed.), Ecological Models (pp. 1959-1968). Oxford: Elsevier.
45. GRIMM, V., Berger, U., DeAngelis, D., Polhill, G., Giske, J., & Railsback, S. (2010). The ODD Protocol: A Review and First Update. Ecological Modelling, 221(23), 2760-2768.
46. GRIMM, V., & Railsback, S. (2005). Individual-Based Modeling and Ecology. Princeton: Princeton University Press.
47. GRIMM, V., & Railsback, S. (2012). Pattern-oriented Modelling: A Multi-scope for Predictive Systems Ecology. Philosophical Transactions of the Royal Society B: Biological Sciences, 367(1586), 298-310.
48. GRIMM, V., Revilla, E., Berger, U., Jeltsch, F., Mooij, W., Railsback, S., Thulke, H.-H., et al. (2005). Pattern-Oriented Modeling of Agent-Based Complex Systems: Lessons from Ecology. Science, 310(5750), 987-991.
49. GRIMM, V., Revilla, E., Groeneveld, J., Kramer-Schadt, S., Schwager, M., Tews, J., Wichmann, M., et al. (2005). Importance of Buffer Mechanisms for Population Viability Analysis. Conservation Biology, 19(2), 578-580.
50. GRIMMETT, G., & Stirzaker, D. (1993). Probability and Random Processes (2nd ed., reprint.). Oxford: Clarendon Press.
51. GROEMPING, U. (2011). FrF2: Fractional Factorial Designs with 2-Level Factors. http://cran.r-project.org/web/packages/FrF2/ Archived at: http://www.webcitation.org/6H9NO65Hn
52. GROEMPING, U. (2013a). CRAN Task View: Design of Experiments (DoE) & Analysis of Experimental Data. http://cran.r-project.org/web/views/ExperimentalDesign.html Archived at: http://www.webcitation.org/6H9NRcCYN
53. GROEMPING, U. (2013b). DoE.base: Full Factorials, Orthogonal Arrays and Base Utilities for DoE Packages. http://cran.r-project.org/web/packages/DoE.base/ Archived at: http://www.webcitation.org/6H9NV5KCl
54. GUICHARD, S., Kriticos, D. J., Kean, J. M., & Worner, S. P. (2010). Modelling Pheromone Anemotaxis for Biosecurity Surveillance: Moth Movement Patterns Reveal a Downwind Component of Anemotaxis. Ecological Modelling, 221(23), 2801-2807.
55. HAMBY, D. M. (1995). A Comparison of Sensitivity Analysis Techniques. Health Physics, 68(2), 195-204.
56. HAN, L., Da Silva, D., Boudon, F., Cokelaer, T., Pradal, C., Faivre, R., & Costes, E. (2012). Investigating the Influence of Geometrical Traits on Light Interception Efficiency of Apple Trees: A Modelling Study with MApple T. In M. Kang, Y. Dumont, & Y. Guo (Eds.), Fourth International Symposium on Plant Growth Modeling, Simulation, Visualization and Applications (pp. 152-159). IEEE Press. http://hal.inria.fr/docs/00/79/06/50/PDF/Han_etal_PMA2012_26-2.pdf Archived at: http://www.webcitation.org/6MlPQm4tv
57. HAPPE, K. (2005). Agent-Based Modelling and Sensitivity Analysis by Experimental Design and Metamodelling: An Application to Modelling Regional Structural Change. Presented at the European Association of Agricultural Economists 2005 International Congress, Copenhagen, Denmark. http://purl.umn.edu/24464 Archived at: http://www.webcitation.org/6MlPFiZqu
58. HARTIG, F., Calabrese, J., Reineking, B., Wiegand, T., & Huth, A. (2011). Statistical Inference for Stochastic Simulation Models - Theory and Application. Ecology Letters, 14(8), 816-827.
59. HARTIG, F., Dislich, C., Wiegand, T., & Huth, A. (2013). Technical Note: Approximate Bayesian Parameterization of a Complex Tropical Forest Model. Biogeosciences Discuss., 10(8), 13097-13128.
60. HASTIE, T. (2011). gam: Generalized Additive Models. http://CRAN.R-project.org/package=gam Archived at: http://www.webcitation.org/6MlPWODr8
61. HELTON, J., Hiermer, J., Sallaberry, C., & Storlie, C. (2006). Survey of Sampling-Based Methods for Uncertainty and Sensitivity Analysis. Reliability Engineering & System Safety, 91(10/11), 1175-1209.
62. HOLLAND, J. (2001). Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control and Artificial Intelligence (6th ed.). Cambridge: MIT Press.
63. HOTHORN, T. (2013). CRAN Task View: Machine Learning & Statistical Learning. http://cran.r-project.org/web/views/MachineLearning.html Archived at: http://www.webcitation.org/6H9NZJFa0
64. HYNDMAN, R. (2013). CRAN Task View: Time Series Analysis. http://cran.r-project.org/web/views/TimeSeries.html Archived at: http://www.webcitation.org/6H9NdY31r
65. IMRON, M., Gergs, A., & Berger, U. (2012). Structure and Sensitivity Analysis of Individual-Based Predator-Prey Models. Reliability Engineering & System Safety, 107, 71-81.
66. IVÁNYI, M., Beckman, R., & Conover, W. (1979). A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output from a Computer Code. Technometrics, 21(2), 239-245.
67. IVÁNYI, M., Gulyás, L., Bocsi, R., Szemes, G., & Mészáros, R. (2007). Model Exploration Module (pp. 207-215). Presented at the Agent 2007, Evanston, IL. http://www.dis.anl.gov/pubs/60568.pdf Archived at: http://www.webcitation.org/6H9Nqgeqd
68. JABOT, F., Faure, T., & Dumoulin, N. (2013). EasyABC: Performing Efficient Approximate Bayesian Computation Sampling Schemes Using R. Methods in Ecology and Evolution.
69. JAKOBY, O., Grimm, V., & Frank, K. (2014). Pattern-oriented Parameterization of General Models for Ecological Application: Towards realistic Evaluations of Management Approaches. Ecological Modelling, 275, 78-88.
70. KELSO, J., & Milne, G. (2011). Stochastic Indiviual-Based Modelling of Influenza Spread for the Assessment of Public Health Interventions. Presented at the 19th International Congress on Modelling and Simulation, Perth, Austrialia. http://www.mssanz.org.au/modsim2011/B2/kelso.pdf Archived at: http://www.webcitation.org/6MlPgqAad
71. KERR, B., Riley, M. A., Feldman, M. W., & Bohannan, B. J. M. (2002). Local Dispersal Promotes Biodiversity in a Real-Life Game of Rock-Paper-Scissors. Nature, 418(6894), 171-174.
72. KING, A., Ionides, E., Breto, C., Ellner, S., Ferrari, M., Kendall, B., Lavine, M., et al. (2013). Pomp: Statistical inference for partially observed Markov processes (R package). http://cran.r-project.org/web/packages/pomp/index.html Archived at: http://www.webcitation.org/6KdUJrXfv
73. KLEIJNEN, J. (1995). Sensitivity Analysis and Related Analysis: A Survey of Statistical Techniques. October 6, 2011, http://econpapers.repec.org/paper/dgrkubrem/1995706.htm Archived at: http://www.webcitation.org/6MlPnSnDV
74. KLEIJNEN, J. (2008). Design and Analysis of Simulation Experiments. New York, NY: Springer.
75. LENTH, R. (2009). Response-Surface Methods in R, Using rsm. Journal of Statistical Software, 32(7), 1-17.
76. LORSCHEID, I., Heine, B.-O., & Meyer, M. (2012). Opening the Black Box of Simulations: Increased Transparency and Effective Communication Through the Systematic Design of Experiments. Computational and Mathematical Organization Theory, 18(1), 22-62.
77. LUNN, D., Spiegelhalter, D., Thomas, A., & Best, N. (2009). The BUGS Project: Evolution, Critique and Future Directions. Statistics in Medicine, 28(25), 3049-3067.
78. MARINO, S., Hogue, I., Ray, C., & Kirschner, D. (2008). A Methodology for Performing Global Uncertainty and Sensitivity Analysis in Systems Biology. Journal of Theoretical Biology, 254(1), 178-196.
79. MARTIN, B. T., Jager, T., Nisbet, R. M., Preuss, T. G., & Grimm, V. (2013). Predicting Population Dynamics From the Properties of Individuals: A Cross-Level Test of Dynamic Energy Budget Theory. The American Naturalist, 181(4), 506-519.
80. MARTÍNEZ, I., Wiegand, T., Camarero, J., Batllori, E., & Gutiérrez, E. (2011). Disentangling the Formation of Contrasting Tree-Line Physiognomies Combining Model Selection and Bayesian Parameterization for Simulation Models. The American Naturalist, 177(5), 136-152.
81. MAY, F., Giladi, I., Ristow, M., Ziv, Y., & Jeltsch, F. (2013). Metacommunity, Mainland-island System or Island Communities? Assessing the Regional Dynamics of Plant Communities in a Fragmented Landscape. Ecography, 36(7), 842-853.
82. MEYER, K., Vos, M., Mooij, W., Gera Hol, W., Termorshuizen, A., Vet, L., & Van Der Putten, W. (2009). Quantifying the Impact of Above- and Belowground Higher Trophic Levels on Plant and Herbivore Performance by Modeling. Oikos, 118(7), 981-990.
83. MICHALEWICZ, Z., & Fogel, D. (2004). How to Solve It (2nd rev. and extended ed.). Berlin: Springer.
84. MIETTINEN, K. (1999). Nonlinear Multiobjective Optimization. Springer.
85. MITCHELL, M. (1998). An Introduction to Genetic Algorithms. Cambridge: MIT Press.
86. MOLINA, J., Clapp, C., Linden, D., Allmaras, R., Layese, M., Dowdy, R., & Cheng, H. (2001). Modeling the Incorporation of Corn (Zea Mays L.) Carbon from Roots and Rhizodeposition into Soil Organic Matter. Soil Biology and Biochemistry, 33(1), 83-92.
87. MORRIS, M. (1991). Factorial Sampling Plans for Preliminary Computational Experiments. Technometrics, 33, 161-174.
88. NASH, J. (2013). Rcgmin: Conjugate Gradient Minimization of Nonlinear Functions with Box Constraints. http://cran.r-project.org/web/packages/Rcgmin/ Archived at: http://www.webcitation.org/6H9O3Cx8P
89. NASH, J., & Varadhan, R. (2011). Unifying Optimization Algorithms to Aid Software System Users: optimx for R. Journal of Statistical Software, 43(9), 1-14.
90. NIKOLIC, I., Van Dam, K., & Kasmire, J. (2013). Practice. In K. Van Dam, I. Nikolic, & Z. Lukszo (Eds.), Agent-Based Modelling of Socio-Technical Systems, Agent-Based Social Systems (pp. 73-140). Dordrecht; New York: Springer.
91. NORTH, M., & Macal, C. (2007). Managing Business Complexity: Discovering Strategic Solutions with Agent-Based Modeling and Simulation. Oxford: Oxford University Press.
92. OLMEDO, O. (2011). Kriging: Ordinary Kriging. http://CRAN.R-project.org/package=kriging Archived at: http://www.webcitation.org/6MlPtXaUU
93. OSTROMSKY, T., Dimov, I., Georgieva, R., & Zlatev, Z. (2010). Sensitivity Analysis of a Large-Scale Air Pollution Model: Numerical Aspects and a Highly Parallel Implementation. In I. Lirkov, S. Margenov, & J. Wasniewski (Eds.), Large-Scale Scientific Computing: 7th International Conference, LSSC 2009 (pp. 197-205). Berlin/Heidelberg: Springer.
94. PAN, Z., & Kang, L. (1997). An Adaptive Evolutionary Algorithm for Numerical Optimization. In X. Yao, J.-H. Kim, & T. Furuhashi (Eds.), Simulated Evolution and Learning (Vol. 1285, pp. 27-34). Berlin/Heidelberg: Springer.
95. PARK, J. (2012). CRAN Task View: Bayesian Inference. http://cran.r-project.org/web/views/Bayesian.html Archived at: http://www.webcitation.org/6H9O9MpTB
96. PIOU, C., Berger, U., & Grimm, V. (2009). Proposing an Information Criterion for Individual-Based Models Developed in a Pattern-Oriented Modelling Framework. Ecological Modelling, 220(17), 1957-1967.
97. PLUMMER, M., Best, N., Cowles, K., & Vines, K. (2006). CODA: Convergence Diagnosis and Output Analysis for MCMC. R News, 6(1), 1-11.
98. PUJOL, G., Iooss, B., & Janon, A. (2012). Sensitivity: Sensitivity Analysis. http://cran.r-project.org/web/packages/sensitivity/index.html Archived at: http://www.webcitation.org/6MlPzCPmU
99. R Core Team. (2013a). R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. http://www.R-project.org Archived at: http://www.webcitation.org/6H9OGtr7f
100. R Core Team. (2013b). Books Related to R. http://www.r-project.org/doc/bib/R-books.html Archived at: http://www.webcitation.org/6H9OEXcIc
101. RADCHUK, V., Johst, K., Groeneveld, J., Grimm, V., & Schtickzelle, N. (2013). Behind the Scenes of Population Viability Modeling: Predicting Butterfly Metapopulation Dynamics under Climate Change. Ecological Modelling, 259, 62-73.
102. RAILSBACK, S., Cunningham, P., & Lamberson, R. (2006). A Strategy for Parameter Sensitivity and Uncertainty Analysis of Individual-based Models. unpubl. manuscript. http://www.humboldt.edu/ecomodel/documents/SensAnalysis_DRAFT.pdf Archived at: http://www.webcitation.org/6MlQB7agx
103. RAILSBACK, S., & Grimm, V. (2012). Agent-Based and Individual-Based Modeling: A Practical Introduction. Princeton: Princeton University Press.
104. RASMUSSEN, C., & Williams, C. (2006). Gaussian Processes for Machine Learning. MIT Press.
105. RAVALICO, J., Maier, H., Dandy, G., Norton, J., & Croke, B. (2005). A Comparison of Sensitivity Analysis Techniques for Complex Models for Environment Management. Proceedings (pp. 2533-2539). Presented at the Modsim05, International Congress on Modelling and Simulation: Advances and Applications for Management and Decision Making, Melbourne. http://www.mssanz.org.au/modsim05/papers/ravalico.pdf Archived at: http://www.webcitation.org/6MlQIdX9q
106. REUILLON, R., Chuffart, F., Leclaire, M., Faure, T., Dumoulin, N., & Hill, D. (2010). Declarative Task Delegation in Open MOLE (pp. 55-62). Presented at the International Conference on High Performance Computing and Simulation (HPCS), Caen, France. http://www.openmole.org Archived at: http://www.webcitation.org/6H9ObnaA2
107. REUSSER, D. (2012). Fast: Implementation of the Fourier Amplitute Sensitivity Test (FAST). http://cran.r-project.org/web/packages/fast/index.html Archived at: http://www.webcitation.org/6MlQOyZq3
108. SALLE, I., & Yildizoglu, M. (2012). Efficient Sampling and Metamodeling for Computational Economic Models (Cahiers du GREThA No. 2012-18). Groupe de Recherche en Economie Théorique et Appliquée. http://ideas.repec.org/p/grt/wpegrt/2012-18.html Archived at: http://www.webcitation.org/6MlQemGS7
109. SALTELLI, A. (2000). What Is Sensitivity Analysis? In A. Saltelli, K. Chan, & E. Scott (Eds.), Sensitivity Analysis (pp. 3-13). Chichester Wiley.
110. SALTELLI, A., Anders, S., & Chan, K. (1999). A Quantitative Model-Independent Method for Global Sensitivity Analysis of Model Output. Technometrics, 41(1), 39-56.
111. SALTELLI, A., Chan, K., & Scott, E. (2000). Sensitivity Analysis (1st ed.). Wiley.
112. SALTELLI, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M., et al. (2008). Global Sensitivity Analysis: The Primer (1st ed.). Wiley-Interscience.
113. SALTELLI, A., Tarantola, S., Campolongo, F., & Ratto, M. (2004). Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models (1st ed.). Wiley.
114. SCHLATHER, M., Menck, P., Singleton, R., Pfaff, B., & R Core Team. (2013). RandomFields: Simulation and Analysis of Random Fields. http://CRAN.R-project.org/package=RandomFields Archived at: http://www.webcitation.org/6MlRIsNEb
115. SCHMOLKE, A., Thorbek, P., DeAngelis, D., & Grimm, V. (2010). Ecological Models Supporting Environmental Decision Making: A Strategy for the Future. Trends in Ecology & Evolution, 25(8), 479-486.
116. SIEBERTZ, K., Bebber, D. van, & Hochkirchen, T. (2010). Statistische Versuchsplanung: Design of Experiments (DoE) (1st Edition.). Berlin/Heidelberg: Springer.
117. SIMLAB. (2012). Simlab: Simulation Environment for Uncertainty and Sensitivity Analysis. Joint Research Centre of the European Commission. http://simlab.jrc.ec.europa.eu/ Archived at: http://www.webcitation.org/6MlRhBPfj
118. SNELSON, E. (2007). Flexible and Efficient Gaussian Process Models for Machine Learning. University College London. http://www.gatsby.ucl.ac.uk/snelson/thesis.pdf Archived at: http://www.webcitation.org/6MlRqg0R6
119. SOBOL', I. (1990). On Sensitivity Estimation for Nonlinear Mathematical Models. Matem. Mod., 2(1), 112-118.
120. SOETAERT, K., & Herman, P. (2009). A Practical Guide to Ecological Modelling: Using R as a Simulation Platform. Dordrecht: Springer.
121. SOETAERT, K., & Petzoldt, T. (2010). Inverse Modelling, Sensitivity and Monte Carlo Analysis in R Using Package FME. Journal of Statistical Software, 33(3), 1-28.
122. SOTTORIVA, A., & Tavaré, S. (2010). Integrating Approximate Bayesian Computation with Complex Agent-Based Models for Cancer Research. COMPSTAT 2010 - Proceedings in Computational Statistics (pp. 57-66). Berlin/Heidelberg: Springer.
123. SPIEGELHALTER, D. J., Best, N. G., Carlin, B. P., & Van Der Linde, A. (2002). Bayesian Measures of Model Complexity and Fit. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 64(4), 583-639.
124. SQUAZZONI, F. (2012). Agent-Based Computational Sociology. Chichester, West Sussex: Wiley & Sons.
125. STONEDAHL, F., & Wilensky, U. (2010). Evolutionary Robustness Checking in the Artificial Anasazi Model. Proceedings of the 2010 AAAI Fall Symposium on Complex Adaptive Systems. Arlington, VA. http://www.aaai.org/ocs/index.php/FSS/FSS10/paper/download/2181/2622 Archived at: http://www.webcitation.org/6H9Rdec9p
126. STONEDAHL, F., & Wilensky, U. (2013). Behavior Search. http://behaviorsearch.org Archived at: http://www.webcitation.org/6H9OgKler
127. SUN, W., & Yuan, Y.-X. (2006). Optimization Theory and Methods: Nonlinear Programming. New York: Springer.
128. TESFATSION, L., & Judd, K. L. (2006). Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics (1st ed.). North Holland.
129. THE MATHWORKS INC. (2010). MATLAB. Natick, Massachusetts.
130. THEUSSL, S. (2013). CRAN Task View: Optimization and Mathematical Programming. http://cran.r-project.org/web/views/Optimization.html Archived at: http://www.webcitation.org/6H9OkkqDN
131. THIELE, J., Kurth, W., & Grimm, V. (2011). Agent- and Indivudal-Based Modelling with NetLogo: Introduction and New NetLogo Extensions. 22. Tagung Deutscher Verband Forstlicher Forschungsanstalten Sektion Forstliche Biometrie, Die Grü ne Reihe (pp. 68-101).
132. THIELE, J., Kurth, W., & Grimm, V. (2012). RNetLogo: An R Package for Running and Exploring Individual-Based Models Implemented in NetLogo. Methods in Ecology and Evolution, 3(3), 480-483.
133. THOMAS, A., O'Hara, B., Ligges, U., & Sturtz, S. (2006). Making BUGS Open. R News, 1(6), 12-17.
134. TRAUTMANN, H., Steuer, D., & Mersmann, O. (2013). Mco: Multi Criteria Optimization Algorithms and Related Functions. http://CRAN.R-project.org/package=mco Archived at: http://www.webcitation.org/6MlRyxndr
135. VAN OIJEN, M. (2008). Bayesian Calibration (BC) and Bayesian Model Comparison (BMC) of Process-Based Models: Theory, Implementation and Guidelines (Report). http://nora.nerc.ac.uk/6087/1/BC%26BMC_Guidance_2008-12-18_Final.pdf Archived at: http://www.webcitation.org/6H9Oq2pRo
136. VANDERWAL, J., & Januchowski, S. (2010). ConsPlan: Conservation Planning Tools. http://www.rforge.net/ConsPlan/index.html Archived at: http://www.webcitation.org/6H9Ova06M
137. VARADHAN, R., & Gilbert, P. (2009). BB: An R Package for Solving a Large System of Nonlinear Equations and for Optimizing a High-Dimensional Nonlinear Objective Function. Journal of Statistical Software, 32(4), 1-26.
138. VENABLES, W., & Ripley, B. (2002). Modern Applied Statistics with S. (4th ed.). New York, NY: Springer.
139. VINATIER, F., Gosme, M., & Valantin-Morison, M. (2013). Explaining Host-Parasitoid Interactions at the Landscape Scale: A New Approach for Calibration and Sensitivity Analysis of Complex Spatio-Temporal Models. Landscape Ecology, 28(2), 217-231.
140. VINATIER, F., Tixier, P., Le Page, C., Duyck, P.-F., & Lescourret, F. (2009). COSMOS, a Spatially Explicit Model to Simulate the Epidemiology of Cosmopolites Sordidus in Banana Fields. Ecological Modelling, 220(18), 2244-2254.
141. WECK, O. L. de. (2004). Multiobjective Optimization: History and Promise. The Third China-Japan-Korea Joint Symposium on Optimization of Structural and Mechanical Systems. http://strategic.mit.edu/docs/3_46_CJK-OSM3-Keynote.pdf Archived at: http://www.webcitation.org/6MlTLHV7d
142. WIEGAND, T., Naves, J., Stephan, T., & Fernandez, A. (1998). Assessing the Risk of Extinction for the Brown Bear (Ursus arctos) in the Cordillera Cantabrica, Spain. Ecological Monographs, 68(4), 539-570.
143. WIENER, N. (1950). Extrapolation, Interpolation, and Smoothing of Stationary Time Series: With Engineering Applications. Technology press books (2. print.). New York: Wiley.
144. WILENSKY, U. (1999). Net Logo. Evanston, IL: Center for Connected Learning and Computer-Based Modeling, Northwestern University. http://ccl.northwestern.edu/netlogo/ Archived at: http://www.webcitation.org/6H9SLE169
145. WILENSKY, U., & Rand, W. (in press). An Introduction to Agent-Based Modeling: Modeling Natural, Social and Engineered Complex Systems with NetLogo. Cambridge: MIT Press.
146. WILENSKY, U., & Shargel, B. (2002). Behavior Space. Evanston, IL: Center for Connected Learning and Computer Based Modeling, Northwestern University. http://ccl.northwestern.edu/netlogo/behaviorspace.html Archived at: http://www.webcitation.org/6H9P2JQby
147. WILLIGHAGEN, E. (2005). Genalg: R Based Genetic Algorithm v 0.1.1. http://cran.r-project.org/web/packages/genalg/index.html Archived at: http://www.webcitation.org/6H9P6le23
148. WOLFRAM RESEARCH. (2010). Mathematica. Champaign, IL.
149. XIANG, Y., Gubian, S., Suomela, B., & Hoeng, J. (forthcoming). Generalized Simulated Annealing for Efficient Global Optimization: the GenSA Package for R. http://cran.r-project.org/web/packages/GenSA/ Archived at: http://www.webcitation.org/6H9S6Tk9T
150. ZUUR, A., Ieno, E., & Meesters, E. (2009). A Beginner's Guide to R. New York: Springer