Good Practices in Model-Informed Drug Discovery and Development: Practice, Application, and Documentation

CPT: Pharmacometrics and Systems Pharmacology

Volumn 5, Issue 3, 2016, Pages 93-122

  Marshall, S F ^a   Burghaus, R ^b   Cosson, V ^c   Cheung, Sya ^d   Chenel, M ^e   DellaPasqua, O ^f   Frey, N ^c   Hamren B ^g   Harnisch, L ^a   Ivanow, F ^h   Kerbusch, T ⁱ   Lippert, J ^b   Milligan, P A ^a   Rohou, S ^d   Staab, A ^j   Steimer, J L ^k   Tornoe C ^l   Visser, S A G ^m  

^a PFIZER GLOBAL RESEARCH AND DEVELOPMENT   (United Kingdom)

^b BAYER PHARMA AG   (Germany)

^c F HOFFMANN LA ROCHE LTD   (Switzerland)

^d ASTRAZENECA   (United Kingdom)

^e INSTITUT DE RECHERCHES INTERNATIONALES SERVIER   (France)

^f GLAXOSMITHKLINE   (United Kingdom)

^g ASTRAZENECA R AND D   (Sweden)

^h Janssen R and D   (United Kingdom)

ⁱ MSD   (Netherlands)

^j BOEHRINGER INGELHEIM PHARMA GMBH AND CO KG   (Germany)

^k NOVARTIS PHARMA AG   (Switzerland)

^l NOVO NORDISK A S   (Denmark)

^m MERCK RESEARCH LABORATORIES   (United States)

more..

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; AUTHORITY; CLINICAL PRACTICE; COMMERCIAL PHENOMENA; COST EFFECTIVENESS ANALYSIS; DECISION MAKING; DOCUMENTATION; DRUG INDUSTRY; HEALTH CARE QUALITY; HUMAN; INTEGRATION; MODEL INFORMED DRUG DISCOVERY AND DEVELOPMENT; PLANNING; PRIORITY JOURNAL; PRODUCTIVITY; QUALITY CONTROL; RISK BENEFIT ANALYSIS; DRUG DESIGN; PHARMACEUTICS; PRACTICE GUIDELINE; PROCEDURES; STANDARDS;

DOCUMENTATION; DRUG DESIGN; GUIDELINES AS TOPIC; TECHNOLOGY, PHARMACEUTICAL;

EID: 84960856818     PISSN: None     EISSN: 21638306     Source Type: Journal    
DOI: 10.1002/psp4.12049     Document Type: Article

References (104)

1. EMA/EFPIA MandS workshop 2011. http://www.ema.europa.eu/ema/index.jsp? curl5pages/news-and-events/events/2011/07/event-detail-000440.jsp&mid5WC0-b01ac058004d5c3 (2011).
2. Manolis, E., Rohou, S., Hemmings, R., Salmonson, T., Karlsson, M. & Milligan, P.A. The role of modeling and simulation in development and registration of medicinal products: Output from the EFPIA/EMA Modeling and Simulation Workshop. CPT Pharmacometrics Syst. Pharmacol. 2, e31 (2013).
3. Visser, S.A., Manolis, E., Danhof, M., Kerbusch, T. Modeling and simulation at the interface of nonclinical and early clinical drug development. CPT Pharmacometrics Syst Pharmacol. 2013;2:e30
4. Marshall, S.F., Hemmings, R., Josephson, F., Karlsson, M.O., Posch, M. & Steimer, J.L. Modeling and simulation to optimize the design and analysis of confirmatory trials, characterize risk-benefit, and support label claims. CPT Pharmacometrics Syst. Pharmacol. 2:e27 (2013).
5. Harnisch, L., Shepard, T., Pons, G., Della Pasqua, O. Modeling and Simulation as a Tool to Bridge Efficacy and Safety Data in Special Populations CPT: Pharmacometrics & Systems Pharmacology 2, e28 (2013).
6. Staab, A., Rook, E., Maliepaard, M., Aarons, L. & Benson, C. Modeling and simulation in clinical pharmacology and dose finding. CPT Pharmacometrics Syst. Pharmacol. 2, e29 (2013).
7. EMA/EFPIA European Medicines Agency/European Federation of Pharmaceutical Industries and Associations workshop on the importance of dose finding and dose selection for the successful development, licensing and lifecycle management of medicinal products. http://www.ema.europa.eu/ema/index.jsp?curl5pages/news-and- events/events/2014/06/event-detail-000993.jsp&mid5WC0b01ac058004d5c3 (2014).
8. Oswalt, I. et al. Calculating Return on Investment for U.S. Department of Defense Modeling and Simulation A Publication of the Defense Acquisition University http://www.dau.mil/pubscats/PubsCats/AR%20Journal/arj58/Oswalt%20ARJ-58.pdf (2011).
9. Lin, J. Measuring return on investment of model-based design mathworks publication https://www.mathworks.com/model-based-design/mbd-roi-video/Measuring-ROI- of-MBD.pdf (2014).
10. Gray, M. Public Weather Service Value for Money Review, Public Weather Service Customer Group Secretariat Met Office http://www.metoffice.gov.uk/media/pdf/9/4/PWS-Value-for-Money-Review--March-2015.pdf (2015).
11. Goldbeck, G. The economic impact of molecular modelling: Goldbeck consulting http://psi-k.net/docs/Economic-impact-of-modelling.pdf (2012).
12. Kruhlak, N.L., Benz, R.D., Zhou, H. & Colatsky, T.J. (Q)SAR modeling and safety assessment in regulatory review. Clin. Pharmacol. Ther. 91, 529-534 (2012).
13. Philips, Z., Bojke, L., Sculpher, M., Claxton, K. & Golder, S. Good practice guidelines for decision-analytic modelling in health technology assessment. Pharmacoeconomics 24, 355-371 (2006).
14. NICE. Developing NICE guidelines: The manual: Section 7 Incorporating economic evaluation. http://www.nice.org.uk/article/pmg20/chapter/7-Incorporating-economicevaluation (2014).
15. FDA Guidance for Industry Population Pharmacokinetics:Center for Drug Evaluation and Research (CDER) and Center for Biologics Evaluation and Research (CBER) http://www.fda.gov/downloads/Drugs/Guidances/UCM072137.pdf (1999).
16. FDA Guidance for Industry Exposure-Response Relationships-StudyDesign, Data Analysis, and Regulatory Applications:Center for Drug Evaluation and Research (CDER) & Center for Biologics Evaluation and Research (CBER) http://www.fda.gov/downloads/Drugs/Guidances/UCM072137.pdf (2003).
17. EMA Guideline on reporting the results of population pharmacokinetic analysis CHMP http://www.ema.europa.eu/docs/en-GB/document-library/Scientific-guideline/2009/09/WC500003067.pdf (2007).
18. Peck, C.C. et al. Opportunities for integration of pharmacokinetics, pharmacodynamics, and toxicokinetics in rational drug development. Pharm. Res. 9, 826-833 (1992).
19. Gieschke, R., Reigner, B, G, & Steimer, J.L. Exploring clinical study design by computer simulation based on pharmacokinetic/pharmacodynamic modelling. Int. J. Clin. Pharmacol. Ther. 10, 469-474 (1997).
20. Gieschke, R. & Steimer, J.L. Pharmacometrics: Modelling and simulation tools to improve decision making in clinical drug development. Eur. J. Drug Metab. Pharmacokinet. 25, 49-58 (2000).
21. Sheiner, L.B. & Steimer, J.L. Pharmacokinetic/pharmacodynamic modeling in drug development. Annu. Rev. Pharmacol. Toxicol.40, 67-95 (2000).
22. Sheiner, L.B. Learning versus confirming in clinical drug development. Clin. Pharmacol. Ther. 61, 275-291 (1997).
23. PWC Price Waterhouse Coopers Pharma 2005: Silicon Rally: The Race to e-RandD http://www.pwc.com/en-GX/gx/pharma-life-sciences/pdf/silicon-rally.pdf (1999).
24. IBM. Pharma 2010 Silicon reality http://www-935.ibm.com/services/us/imc/pdf/ge510-3600-03.pdf (2010).
25. FDA Innovation or stagnation: Challenges and opportunity on the critical path to new medical products http://www.fda.gov/ScienceResearch/SpecialTopics/CriticalPathInitiative/OpportunitiesReports/ucm077262.htm (2004).
26. FDA. Advancing Regulatory Science at FDA: A strategic Plan http://www.fda.gov/downloads/ScienceResearch/SpecialTopics/RegulatoryScience/UCM268225.pdf (2011).
27. Romero, K. et al. Modeling and simulation for medical product development and evaluation: Highlights from the FDA-C-Path-ISOP 2013 workshop. J. Pharmacokinet. Pharmacodyn. 41, 545-552 (2014).
28. EMA: Modelling and simulation working group plan 2015 http://www.ema.europa.eu/docs/en-GB/document-library/Work-programme/2015/04/WC500185865.pdf (2015).
29. Reigner, B.G., Williams, P.E., Patel, I.H., Steimer, J.L., Peck, C. & Van Brummelen, P. An evaluation of the integration of pharmacokinetic and pharmacodynamic principles in clinical drug development. Experience within Hoffmann La Roche. Clin. Pharmacokinet. 33, 142-152 (1997).
30. Chein, J.Y., Friedrich, S., Heathman, M.A., deAlwis, D.P. & Sinha, V. Pharmacokinetics/pharmacodynamics and the stages of drug development: Role of modeling and simulation AAPS J. 7, E544-E559 (2005).
31. Milligan, P.A., Brown, M.J., Marchant, B., Martin, S.W., Van Der Graaf, P.H. & Benson, N. Model-based drug development: A rational approach to efficiently accelerate drug development. Clin. Pharmacol. Ther. 93, 502-514 (2013).
32. Visser, S.A., De Alwis, D.P., Kerbusch, T., Stone, J.A. & Allerheiligen, S.R. Implementation of quantitative and systems pharmacology in large pharma. CPT Pharmacometrics Syst. Pharmacol. 22, e142 (2014).
33. Allerheiligen, S.R. Impact of modeling and simulation: Myth or fact? Clin. Pharmacol. Ther. 96, 413-415 (2014).
34. Lee, J.Y. et al. Impact of pharmacometric analyses on new drug approval and labelling decisions: A review of 198 submissions between 2000 and 2008. Clin. Pharmacokinet. 50, 627-635 (2011).
35. EMA: 2014 Activity report of the Modelling and simulation working group (MSWG) http://www.ema.europa.eu/docs/en-GB/document-library/Report/2015/04/WC500185895.pdf (2015b).
36. Vicini, P. & Van Der Graaf, P.H. Systems pharmacology for drug discovery and development: Paradigm shift or flash in the pan? Clin. Pharmacol. Ther. 93, 379-381 (2013).
37. NIH: Quantitative and systems pharmacology in the post-genomic era: New approaches to discovering drugs and understanding therapeutic mechanisms. http://www.nigms.nih.gov/training/documents/systemspharmawpsorger2011.pdf (2011).
38. Visser, S.A. et al. Model-based drug discovery: Implementation and impact. Drug Discov. Today 18, 764-775 (2013).
39. Ette, E.I & Williams, P.J. Pharmacometrics: The Science of Quantitative Pharmacology (John Wiley & Sons, Inc., Hoboken, NJ, 2007).
40. Barrett, J.S., Fossler, M.J., Cadieu, K.D. & Gastonguay, M.R. Pharmacometrics: A multidisciplinary field to facilitate critical thinking in drug development and translational research settings. J. Clin. Pharmacol. 48, 632-649 (2008).
41. Lalonde, R.L. et al. Model-based drug development. Clin. Pharmacol. Ther. 82, 21-32 (2007).
42. Shepard T. Role of Modelling and Simulation in Regulatory Decision Making in Europe. November 30, 2011 EMA, London http://www.ema.europa.eu/docs/en-GB/document-library/Presentation/2011/11/WC500118262.pdf (2011).
43. Sinha, V. Contemporary Issues in Clinical Pharmacology The FDA Office of Clinical Pharmacology Experience ASCPT Annual Meeting March 6, 2013 Open Forum http://www.fda.gov/downloads/AboutFDA/CentersOffices/OfficeofMedicalProductsand-Tobacco/CDER/UCM344415.pdf (2013).
44. Zineh, I. & Woodcock, J. Clinical pharmacology and the catalysis of regulatory science: Opportunities for the advancement of drug development and evaluation. Clin. Pharmacol. Ther. 93, 515-525 (2013).
45. EMA: Qualification opinion and comments on MCP-Mod as an efficient statistical methodology for model-based design and analysis of phase-II dose-finding studies under model uncertainty. http://www.ema.europa.eu/docs/en-GB/document-library/Regulatory-and-procedural-guideline/2014/02/WC500161027.pdf (2014).#
46. Bornkamp, B., Bretz, F., Dmitrienko, A., Enas, G., Gaydos, B. & Hsu, C.H. Innovative approaches for designing and analyzing adaptive dose-ranging trials. J. Biopharm. Stat. 17, 965-995 (2007).
47. Bretz, F., Hsu, J., Pinheiro, J. & Liu, Y. Dose finding-a challenge in statistics. Biom. J. 50, 480-504 (2008).
48. ICH Topic E 4: Dose-response information to support drug registration. http://www. ich.org/fileadmin/Public-Web-Site/ICH-Products/Guidelines/Efficacy/E4/Step4/E4- Guideline.pdf (1994).
49. Holford, N.H.G., Mould, D.R. & Peck, C.C. Disease progression models. In Principles of Clinical Pharmacology 2nd edn. (ed. Atkinson, A.) 369-379 (Academic Press, New York, 2006).
50. Mould, D.R. Models for disease progression: New approaches and uses. Clin. Pharmacol. Ther. 92, 125-31 (2012).
51. Overgaard, RV, Ingwersen, SH and Tornøe, CW. Establishing Good Practices for Exposure-Response Analysis of Clinical Endpoints in Drug Development CPT Pharmacometrics Syst. Pharmacol. doi:10.1002/psp4.12015 (2015).
52. Holford, N.H. & Sheiner, L.B. Pharmacokinetic and pharmacodynamic modeling in vivo. Crit. Rev. Bioeng. 5, 273-322 (1981).
53. Meibohm, B. & Derendorf, H. Basic concept of PK/PD modeling. Int. J. Clin. Pharmacol. Ther. 35, 401-413 (1997).
54. Holford, N.H. & Sheiner, L.B. Understanding the concentration-effect relationship: Clinical application of pharmacokinetic-pharmacodynamic models. Clin. Pharmacokinet. 6, 429-453 (1981).
55. Mould, D.R. & Upton, R.N. Basic concepts in population modeling, simulation, and model-based drug development. CPT Pharmacometrics Syst. Pharmacol. 1, e6 (2012).
56. Mould, D.R. & Upton, R.N. Basic concepts in population modeling, simulation, and model-based drug development-part 2: Introduction to pharmacokinetic modeling methods. CPT Pharmacometrics Syst. Pharmacol. 2, e38 (2013).
57. Upton, R.N. & Mould, D.R. Basic concepts in population modeling, simulation, and model-based drug development: Part 3-introduction to pharmacodynamic modeling methods. CPT Pharmacometrics Syst. Pharmacol. 3, e88 (2014).
58. Mandema, J.W., Gibbs, M., Boyd, R.A., Wada, D.R. & Pfister, M. Model-based meta-analysis for comparative efficacy and safety: Application in drug development and beyond. Clin. Pharmacol. Ther. 90, 766-769 (2011).
59. Mould, D.R. Models for disease progression: New approaches and uses. Clin. Pharmacol. Ther. 92, 125-131 (2012).
60. M Boucher, M Bennetts The many flavours of model based meta-analysis: Part I-Introduction and Landmark data CPT Pharmacometrics Syst Pharmacol. 2015 DOI: 10.1002/psp4.12041
61. Van Der Graaf, P.H. & Danhof, M. Analysis of drug-receptor interactions in vivo: A new approach in pharmacokinetic-pharmacodynamic modelling. Int. J. Clin. Pharmacol. Ther. 35, 442-446 (1997).
62. Krzyzanski W1, Jusko WJ. Mathematical formalism for the properties of four basic models of indirect pharmacodynamic responses. J Pharmacokinet Biopharm. 1997 Feb;25(1):107-23.
63. Bender BC, Schindler E, and Friberg LE. Population pharmacokinetic-pharmacodynamic modelling in oncology: A tool for predicting clinical response Br J Clin Pharmacol. 2015 Jan; 79(1): 56-71.
64. Iyengar, R., Zhao, S., Chung, S.W., Mager, D.E. & Gallo, J.M. Merging systems biology with pharmacodynamics. Sci. Transl. Med. 4, 126ps7 (2012).
65. Zhao, P., Rowland, M. & Huang, S.M. Best practice in the use of physiologically based pharmacokinetic modeling and simulation to address clinical pharmacology regulatory questions. Clin. Pharmacol. Ther. 92, 17-20 (2012).
66. Rostami-Hodjegan, A. Physiologically based pharmacokinetics joined with in vitro-in vivo extrapolation of ADME: A marriage under the arch of systems pharmacology. Clin. Pharmacol. Ther. 92, 50-61 (2012).
67. Kostewicz ES, Aarons L, Bergstrand M, Bolger MB, Galetin A, Hatley O, Jamei M, Lloyd R, Pepin X, Rostami-Hodjegan A, Sjogren E, Tannergren C, Turner DB, Wagner C, Weitschies W, Dressman PBPK models for the prediction of in vivo performance of oral dosage forms. J. Eur J Pharm Sci. 2014 Jun 16;57:300-21. doi: 10.1016/j.ejps.2013.09.008.
68. Sager JE, Yu J1, Raguenau-Majlessi I, Isoherranen N Physiologically Based Pharmacokinetic (PBPK) Modeling and Simulation Approaches: A systematic review of published models, applications and model verification. Drug Metab Dispos. 2015 Aug 21 pii: Dmd.115.065920.
69. Jones HM and Rowland-Yeo K Basic Concepts in Physiologically Based Pharmacokinetic Modeling in Drug Discovery and Development. CPT Pharmacometrics Syst Pharmacol. 2015 4: E00016.
70. Jones, H.M. et al. Physiologically based pharmacokinetic modeling in drug discovery and development: A pharmaceutical industry perspective. Clin. Pharmacol. Ther. 97, 247-262 (2015).
71. Ringel, M., Tollman, P., Hersch, G. & Schulze, U. Does size matter in RandD productivity? If not, what does? Nat.Rev. Drug Discov. 12, 901-902 (2013).
72. Dykstra, K. et al. Reporting guidelines for population pharmacokinetic analyses. J. Pharmacokinet. Pharmacodyn. 42, 301-314 (2015).
73. Burman, C.F. & Wiklund, S.J. Modelling and simulation in the pharmaceutical industry-some reflections. Pharm. Stat. 10, 508-516 (2011).
74. Bhattaram, V.A. et al. Impact of pharmacometrics on drug approval and labeling decisions: A survey of 42 new drug applications. AAPS J. 7, E503-E512 (2005).
75. Jonsson, S., Henningsson, A., Edholm, M. & Salmonson, T. Role of modelling and simulation: A European regulatory perspective. Clin. Pharmacokinet. 51, 69-76 (2012).
76. Edholm, M. Modelling and Simulation examples that failed to meet regulator's expectations November 30, 2011 EMA, London http://www.ema.europa.eu/docs/en- GB/document-library/Presentation/2011/11/WC500118266.pdf (2011)
77. Salmonson, T. & Rane, A. Clinical pharmacokinetics in the drug regulatory process. Clin. Pharmacokinet. 18, 177-183 (1990).
78. Rowland, M. Physiologic pharmacokinetic models and interanimal species scaling. Pharmacol Ther. 29, 49-68 (1985).
79. Rajman, I. PK/PD modelling and simulations: Utility in drug development. Drug Discov. Today 13, 341-346 (2008).
80. Sheiner, L.B. & Wakefield, J. Population modelling in drug development. Stat. Methods Med. Res. 8, 183-193 (1999).
81. Jonsson, E.N. & Sheiner, L.B. More efficient clinical trials through use of scientific model-based statistical tests. Clin. Pharmacol. Ther. 72, 603-614 (2002).
82. Bonate, P.L. et al. Guidelines for the quality control of population pharmacokineticpharmacodynamic analyses: An industry perspective. AAPS J. 14, 749-758 (2012).
83. Caro, J.J., Briggs, A.H., Siebert, U. & Kuntz, K.M.; ISPOR-SMDM Modeling Good Research Practices Task Force. Modeling good research practices-overview: A report of the ISPOR-SMDM Modeling Good Research Practices Task Force-1. Value Health 15, 796-803 (2012).
84. Kiang, T.K., Sherwin, C.M., Spigarelli, M.G. & Ensom, M.H. Fundamentals of population pharmacokinetic modelling: Modelling and software. Clin. Pharmacokinet. 51, 515-525.
85. Byon, W. et al. Establishing best practices and guidance in population modeling: An experience with an internal population pharmacokinetic analysis guidance. CPT Pharmacometrics Syst. Pharmacol. 2, e51 (2013).
86. Karlsson, M.O. & Mentr-e, F. Best practices in population modeling should always be evolving. CPT Pharmacometrics Syst. Pharmacol. 2, e52 (2013).
87. Eudralex Volume 2B Notice to Applicants Medicinal products for human use Presentation and format of the dossier Common Technical Document (CTD) http://ec. europa.eu/health/documents/eudralex/vol-2/index-en.htm (2008)
88. ICH. International Committee for Harmonisation Common Technical Document http://www.ich.org/products/ctd.html (2003)
89. Beal, S.L. Ways to fit a PK model with some data below the quantification limit. J. Pharmacokinet. Pharmacodyn. 28, 481-504 (2001).
90. Comets, E., Brendel, K. & Mentr-e, F. Computing normalised prediction distribution errors to evaluate nonlinear mixed-effect models: The npde add-on package for R. Comput. Methods Programs Biomed. 90, 154-166 (2008).
91. Comets, E., Brendel, K. & Mentr-e, F. Why should prediction discrepancies be renamed standardized visual predictive check? J. Clin. Pharmacol. 52, 1284-1285 (2012).
92. Yates, J.W., Jones, R.D., Walker, M. & Cheung, S.Y. Structural identifiability and indistinguishability of compartmental models. Expert Opin. Drug Metab. Toxicol. 5, 295-302 (2009).
93. Cheung, S.Y., Majid, O., Yates, J.W. & Aarons, L. Structural identifiability analysis and reparameterisation (parameter reduction) of a cardiovascular feedback model. Eur. J. Pharm. Sci. 46, 259-271 (2012).
94. Yano, Y., Beal, S.L. & Sheiner, L.B. Evaluating pharmacokinetic/pharmacodynamic models using the posterior predictive check. J. Pharmacokinet. Pharmacodyn. 28, 171-192 (2001).
95. Savic, R.M. & Karlsson, M.O. Importance of shrinkage in empirical Bayes estimates for diagnostics: Problems and solutions. AAPS J. 11, 558-569 (2009).
96. Brendel, K., Comets, E., Laffont, C. & Mentr-e, F. Evaluation of different tests based on observations for external model evaluation of population analyses. J. Pharmacokinet. Pharmacodyn. 37, 49-65 (2010).
97. Bergstrand, M., Hooker, A.C., Wallin, J.E. & Karlsson, M.O. Prediction-corrected visual predictive checks for diagnosing nonlinear mixed-effects models. AAPS J. 13, 143-151 (2011).
98. Kimko H.C & Duffull, S.B. (eds.) Simulation for designing clinical trials. A pharmacokinetic pharmacodynamic modeling perspective. Marcel Dekker, New York, 2003.
99. Goggin, T., Gieschke, R., Pillai, G., Fotteler, B., P. Jordan & J.-L.Steimer: Modeling and simulation of clinical trials: An industry perspective. In: Simulation for designing clinical trials. (eds. Kimko, H.C. & Duffull, S. B.) 227-244 (Marcel Dekker, Inc., New York, 2003
100. Holford, N.H., Kimko, H.C., Monteleone, J.P. & Peck, C.C. Simulation of clinical trials. Annu. Rev. Pharmacol. Toxicol. 40, 209-234 (2000).
101. Holford, N., Ma, S.C. & Ploeger, B.A. Clinical trial simulation: A review. Clin. Pharmacol. Ther. 88, 166-182 (2010).
102. Bonate, P.L. Clinical trial simulation in drug development. Pharm. Res. 17, 252-256 (2000).
103. Nyberg, J, Methods and software tools for design evaluation in population pharmacokinetics-pharmacodynamics studies. Br. J. Clin. Pharmacol. 79, 6-17 (2015).
104. National Research Council. Committee on Mathematical Foundations of Verification, Validation, and Uncertainty Quantification; Board on Mathematical Sciences and Their Applications, Division on Engineering and Physical Sciences, National Research Council: Assessing the Reliability of Complex Models: Mathematical and Statistical Foundations of Verification, Validation, and Uncertainty Quantification The National Academies http://www.espc.oar.noaa.gov/sites/ESPC/Documents/NAS%20-Report%20Uncertanty%20Quantification%2013395.pdf (2012)