When quality beats quantity: Decision theory, drug discovery, and the reproducibility crisis

PLoS ONE

Volumn 11, Issue 2, 2016, Pages

  Scannell, Jack W ^a,b,c   Bosley, Jim ^d  

^a UNIVERSITY OF OXFORD   (United Kingdom)

^b UNIVERSITY OF EDINBURGH   (United Kingdom)

^c JWScannell Analytics Ltd   (United Kingdom)

^d Clerbos LLC   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; BINDING AFFINITY; DECISION THEORY; DRUG BINDING; DRUG DETERMINATION; DRUG DEVELOPMENT; DRUG EFFICACY; DRUG RESEARCH; DRUG SCREENING; DRUG TARGETING; FALSE POSITIVE RESULT; HISTORY; PREDICTIVE VALIDITY; QUALITATIVE ANALYSIS; QUANTITATIVE ANALYSIS; REPRODUCIBILITY; THEORETICAL MODEL; COST BENEFIT ANALYSIS; ECONOMICS; HIGH THROUGHPUT SCREENING; HUMAN; PHARMACEUTICS; PROCEDURES; PRODUCTIVITY; QUALITY CONTROL; RESEARCH; TRENDS;

BIOPHARMACEUTICS; COST-BENEFIT ANALYSIS; DECISION THEORY; DRUG DISCOVERY; EFFICIENCY; FALSE POSITIVE REACTIONS; HIGH-THROUGHPUT SCREENING ASSAYS; HUMANS; MODELS, THEORETICAL; QUALITY CONTROL; REPRODUCIBILITY OF RESULTS; RESEARCH;

EID: 84959378832     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0147215     Document Type: Article

References (107)

1. Scannell J, Blanckley A, Boldon H, Warrington B. Diagnosing the decline in pharmaceutical R&D efficiency. Nat Rev Drug Discov. 2012; 11: p. 191-200. doi: 10.1038/nrd3681 PMID: 22378269
2. Hogan JC. Combinatorial chemistry in drug discovery. Nat Biotechnol. 1997; 15: p. 328-330. PMID: 9094132
3. Geysen HM, Schoenen F, Wagner D, Wagner R. Combinatorial compound libraries for drug discovery: an ongoing challenge. Nat Rev Drug Discov. 2003; 2: p. 222-230. PMID: 12612648
4. Nature Biotechnology. Combinatorial chemistry. Nat Biotechnol. 2000; 18 supplement: p. IT50-IT52.
5. Dolle RE. Historical overview of chemical library design. In Zhou JZ, editor. Chemical Library Design (Methods in Molecular Biology 685).: Springer Science; 2011. p. 3-25.
6. Sanger F. Sequences, sequences, and sequences. Annu Rev Biochem. 1988; 57: p. 1-28. PMID: 2460023
7. Powledge TM. How many genomes are enough? Genome Biology. 2003;: spotlight-20031118-01.
8. Brand D. From years to hours: calculating molecular structure speeds up, fueled by new X-ray sources. Ithaca, NY: Cornell University; 2004. Available: http://www.news.cornell.edu/stories/2004/02/new-x-ray-sources-speed-protein-crystallography. Accessed November 2015.
9. Van Brunt J. Protein architecture: designing from the ground up. Nat Biotechnol. 1986: p. 277-283.
10. PDB on-line database. Available: http://www.rcsb.org/pdb/statistics/holdings.do.
11. Mayr LM, Fuerst P. The future of high-throughput screening. J. Biomol. Screen. 2008; 13: p. 443-448. doi: 10.1177/1087057108319644 PMID: 18660458
12. Brown A, Casadevall A. Increasing disparities between resource inputs and outcomes, as measured by certain health deliverables, in biomedical research. Proc Natl Acad Sci U S A. 2015.
13. Ioannidis J. Why most published research findings are false. PLoS Med. 2005; 2(8): p. e124. PMID: 16060722
14. Prinz F, Schlange T, Asadullah K. Believe it or not: How much can we rely on published data on potential drug targets? Nat Rev Drug Discov. 2011; 10: p. 328-329.
15. Begley G, Ellis L. Drug development: Raise standards for preclinical cancer research. Nature. 2012; 483: p. 531-533. doi: 10.1038/483531a PMID: 22460880
16. Perrin S. Preclinical research: Make mouse studies work. Nature. 2014; 507: p. 423-425. PMID: 24678540
17. Di Masi J. Success rates for new drugs entering clinical testing in the United States. Clin Pharmacol Ther. 1995; 58: p. 1-14. PMID: 7628176
18. Hay M, Thomas J, Craighead J, Economides C, Rosenthal J. Clinical development success rates for investigational drugs. Nat Biotechnol. 2014; 32: p. 40-51. doi: 10.1038/nbt.2786 PMID: 24406927
19. Booth B, Zemmil R. Prospects for productivity. Nat Rev Drug Discov. 2004; 3: p. 451-456. PMID: 15136792
20. Munos B. Lessons from 60 years of pharmaceutical innovation. Nat Rev Drug Discov. 2010;: p. 959-968.
21. Paul S, Mytelka D, Dunwiddie C, Persinger C, Munos B, Lindborg SSA. How to improve R&D productivity: the pharmaceutical industry's grand challenge. Nat Rev Drug Discov. 2010; 9: p. 203-214. doi: 10.1038/nrd3078 PMID: 20168317
22. Berndt E, Nass D, Kleinrock M, Aitken M. Decline in economic returns from new drugs raises questions about sustaining innovation. Health Aff (Millwood). 2015; 34: p. 245-252.
23. Scannell J, Hinds S, Evans R. Financial returns on R&D: Looking back at history, looking forward to adaptive licensing. Rev Recent Clin Trials. 2015; 10: p. 28-43. PMID: 25925881
24. Le Fanu J. The Rise and Fall of Modern Medicine. London: Little Brown; 1999.
25. Horrobin D. Modern biomedical research: an internally self-consistent universe with little contact with medical reality? Nat Rev Drug Discov. 2003; 2: p. 151-154. PMID: 12563306
26. Horrobin D. Realism in drug discovery-could Cassandra be right. Nat Biotechnol. 2011; 19: p. 1099-1100.
27. Brown C, Davis H. Receiver operating characteristics curves and related decision measures: A tutorial. Chemometr Intell Lab Syst. 2006; 80: p. 24-38.
28. Fawcett T. An introduction to ROC analysis. Pattern Recognit Lett. 2006; 27: p. 861-874.
29. Colquhoun D. An investigation of the false discovery rate and the misinterpretation of P values. R Soc Open Sci. 2014; 1: p. 140216. doi: 10.1098/rsos.140216 PMID: 26064558
30. Edwards W, Miles R, von Winterfeldt D. Advances in Decision Analysis: From Foundations to Applications. New York: Cambridge Unversity Press; 2007.
31. Peterson M. An Introduction to Decision Theory. Cambridge: Cambridge University Press; 2009.
32. Spiegelhalter D, Abrams K, Myles J. Bayesian Approaches to Clinical Trials and Health-care Evaluation: John Wiley and Sons; 2004.
33. O'Hagan A, Stevens J, Campbell M. Assurance in clinical trial design. Pharm Stat. 2005; 4: p. 187-201.
34. Gasparini M, Di Scala L, Bretz F, Racine-Poon A. Predictive probability of success in clinical drug development. Epidemiol Biostat Public Health. 2013;: p. e8760 1-14.
35. Senn S. Statistical Issues in Drug Development. 2nd ed. Chichester: John Wiley & Sons, Ltd; 2007.
36. Willan A, Pinto E. The value of information and optimal clinical trial design. Stat Med. 2005; 24: p. 1791-1806. PMID: 15806619
37. Bacchetti P. Current sample size conventions: Flaws, harms, and alternatives. BMC Med. 2010; 8: p. 17. doi: 10.1186/1741-7015-8-17 PMID: 20307281
38. Bacchetti P, McCulloch C, Segal M. Simple, defensible sample sizes based on cost efficiency. Biometrics. 2008; 64(2): p. 577-585. doi: 10.1111/j.1541-0420.2008.01004-1.x PMID: 18482055
39. Simon R. Simple, defensible sample sizes based on cost efficiency-Discussions. Biometrics. 2008; 64(2): p. 589-591.
40. Mueller P, Rosner G. Simple, defensible sample sizes based on cost efficiency-Discussions. Biometrics. 2008; 64(2): p. 587-589.
41. Hanley J, Shapiro S. Simple, defensible sample sizes based on cost efficiency-Discussions. Biometrics. 2008; 64(2): p. 586-587.
42. Bacchetti P, M CE, Segal M. Simple, defensible sample sizes based on cost efficiency-Rejoinder. Biometrics. 2008; 64(2): p. 592-594.
43. Bacchetti P, Deeks S, McCune J. Breaking free of sample size dogma to perform innovative translational research. Sci Transl Med. 2011; 3: p. 87ps24. doi: 10.1126/scitranslmed.3001628 PMID: 21677197
44. Detsky A. Using cost-effectiveness analysis to improve the efficiency of allocating funds to clinical trials. Stat Med. 1990; 9: p. 173-184. PMID: 2111932
45. Berry D. Bayesian clinical trials. Nat Rev Drug Discov. 2006; 5: p. 26-36.
46. Drug Baron. Drug Baron. 2013. Available: http://www.tcpinnovations.com/drugbaron/monte-carlomodels-of-drug-rd-focus-attention-on-cutting-costs-part-1/. Accessed February 2015.
47. Leach A, Gillet V. An Introduction to Chemoinformatics. Dordrecht: Springer; 2005.
48. Ajay A, Walters W, Murcko M. Can we learn to distinguish between 'drug-like' and 'nondrug-like' molecules. J Med Chem. 1998; 41: p. 3314-3324. PMID: 9719583
49. Sadowski J, Kubinyl H. A scoring scheme for discriminating between drugs and nondrugs. J Med Chem. 1998; 41: p. 3325-3329. PMID: 9719584
50. Zhang J, Thomas D, Chung Y, Oldenburg K. A simple statistical parameter for use in evaluation and validation of high throughput screening assays. J Biomol Screen. 1999; 4: p. 67-73. PMID: 10838414
51. Pickett S, McLay I, Clark D. Enhancing the hit-to-lead properties of lead optimisation libraries. J Chem Inf Comput Sci. 2000; 40: p. 263-272. PMID: 10761127
52. Lipinski CA, Lombardo F, Dominy BW, Feeney PJ. Experimental and computational approaches to estimate solubility and permiability in drug discovery and development settings. Adv Drug Deliv Rev. 2001: p. 3-26.
53. Walters W, Namchuk M. Designing screens: How to make your hits a hit. Nat Rev Drug Discov. 2003; 2: p. 259-266. PMID: 12669025
54. Bender A, Bojanic D, Davies J, Crisman T, Mikhailov D, Scheiber J, et al. Which aspects of HTS are empirically correlated with downstream success? Curr Opin Drug Discov Devel. 2008; 11(3).
55. Langer T, Hoffman R, Bryant S, Lesur B. Hit finding: Towards 'smarter' approaches. Curr Opin Pharmacol. 2009; 9: p. 589-593. doi: 10.1016/j.coph.2009.06.001 PMID: 19576852
56. Bickerton G, P GV, Besnard J, Muresan S, Hopkins A. Quantifying the chemical beauty of drugs. Nature Chem. 2012; 4: p. 90-98.
57. Cumming J, Davis A, Muresan S, Haeberlein M, Chen H. Chemical predictive modelling to improve compound quality. Nat Rev Drug Discov. 2013; 12: p. 948-462.
58. Hopkins A, Kesuru G, Leeson P, Rees D, Reynolds C. The role of ligand efficiency metrics in drug discovery. Nat Rev Drug Discov. 2014; 13: p. 105-121. doi: 10.1038/nrd4163 PMID: 24481311
59. Lendrem D, Lendrem B, Peck R, Senn S, Day S, Isaacs J. Progression-seeking bias and rational optimism in research and development. Nat Rev Drug Discov. 2015; 14: p. 219-221. doi: 10.1038/nrd4320-c1 PMID: 25656595
60. Huang R, Wallqvist A, Covell D. Assessment of in vitro and in vivo activities in the National Cancer Institute's anticancer screen with respect to chemical structure, target specificity, and mechanism of action. J Med Chem. 2006; 49: p. 1964-1979. PMID: 16539384
61. Wilner P. The validity of animal models in depression. Psychopharmacology (Berl). 1984; 83: p. 1-16.
62. Baker S. Evaluating a new test using a reference test with estimated sensitivity and specificity. Communications in Statistics-Theory and Methods. 1989; 20: p. 2739-2752.
63. Gart J, Buck A. Comparison of a screening test and a reference test in epidemiologic studies. II. A probabilistic model for the comparison of diagnostic tests. Am J Epidemiol. 1966; 83: p. 593-602. PMID: 5932703
64. Howard R. The foundations of decision analysis revisited. In Edwards W, Miles R, von Winterfeldt D. Advances in Decision Analysis: From Foundations to Applications. New York: Cambridge University Press; 2007. p. 32-56.
65. Cook D, Brown D, Alexander R, March R, Morgan P, Satterthwaite G, et al. Lessons learned from the fate of Astra Zeneca's drug pipeline: a five-dimensional framework. Nat Rev Drug Discov. 2014.
66. Metcalf B, Dillon S. Target Validation in Drug Discovery: Academic Press; 2006.
67. Ioannidis J, Khoury M. Improving validation practices in "Omics" research. Science. 2011; 334: p. 1230-1232. doi: 10.1126/science.1211811 PMID: 22144616
68. Coma I, Clark L, Diez E, Harper G, Herranz J, Hofmann G, et al. Process validation and screen reproducibility in high-throughput screening. J Biomol Screen. 2009; 14: p. 66-76. doi: 10.1177/1087057108326664 PMID: 19171922
69. Sams-Dodd F. Strategies to optimize the validity of disease models in the drug discovery process. Drug Discov Today. 2006; 11: p. 355-363. PMID: 16580978
70. Henderson V, Kimmelman J, Fergusson D, Grimshaw J, Hackam D. Threats to validity in the design and conduct of preclinical efficacy studies: a systematic review of guidelines for in vivo animal experiments. PLoS Med. 2013; 7: p. e1001489.
71. Belzung C, Lemoine M. Criteria for validity for animal models of psychiatric disorders: focus on anxiety disorders and depression. Biol Mood Anxiety Disord. 2011; 1: p. 9. doi: 10.1186/2045-5380-1-9 PMID: 22738250
72. Button K, Ioannidis J, Mokrysz C, Nosek B, Flint J, Robinson E, et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci. 2013; 14(5): p. 365-376. doi: 10.1038/nrn3475 PMID: 23571845
73. Beck B. Model evaluation and performance. In El-Shaarawi A, Piegorsch W. Encyclopedia of Environmetrics. Chichester: John Wiley & Sons, Ltd; 2002. p. 1275-1279.
74. Lewis K. Plaforms for antibiotic discovery. Nature Rev Drug Discov. 2013; 12: p. 371-387.
75. Dranchak P, MacArthur R, Guha R, Zuercher W, Drewry D, Auld D, et al. Profile of the GSK published protein kinase inhibitor set across ATP-dependent and -independent luciferases: Implications for reporter-gene assays. PLoS One. 2013; 8: p. e57888. doi: 10.1371/journal.pone.0057888 PMID: 23505445
76. Pedró-Rosa L, Buckner F, Ranade R, Eberhart C, Madoux F, Gillespie J, et al. Identification of potent inhibitors of the Trypanosoma brucei methionyl-tRNA synthetase inhibitors via high throughput orthogonal screening. J Mol Screen. 2014; 20: p. 122-130.
77. La Mattina J. Forbes. 2012. Available: http://www.forbes.com/sites/johnlamattina/2012/03/02/therehas-never-really-been-low-hanging-fruit-in-pharma-rd/. Accessed March 2015.
78. Pammolli F, Magazzini L, Riccaboni M. The productivity crisis in pharmaceutical R&D. Nature Rev Drug Discov. 2011; 10: p. 428-438.
79. Horrobin DF. Innovation in the pharmaceutical industry. J R Soc Med. 2000; 93: p. 341-345. PMID: 10928019
80. Swinney D, Anthony J. How were new medicines discovered? Nat Rev Drug Discov. 2011; 10: p. 507-519. doi: 10.1038/nrd3480 PMID: 21701501
81. Sams-Dodd F. Target-based drug discovery: Is something wrong? Drug Discov Today. 2005; 10: p. 139-147. PMID: 15718163
82. Silver L. Challenges of antibacterial discovery. Clin Microbiol Rev. 2011; 24: p. 71-109. doi: 10.1128/CMR.00030-10 PMID: 21233508
83. Joyner M, Prendergast F. Chasing Mendel: five questions for personalized medicine. J. Physiol. 2014; 592: p. 2381-2388. doi: 10.1113/jphysiol.2014.272336 PMID: 24882820
84. Payne D, Gwynn M, Holmes D, Pompliano D. Drugs for bad bugs: confronting the challenges of antibacterial discovery. Nat Rev Drug Discov. 2007; 6: p. 29-40. PMID: 17159923
85. Sams-Dodd F. Is poor research the cause of the declining productivity of the pharmaceutical industry? An industry in need of a paradigm shift. Drug Discov Today. 2013; 18: p. 211-217. doi: 10.1016/j. drudis.2012.10.010 PMID: 23131208
86. Goldstein D. Common genetic variation and human traits. N Engl J Med. 2009; 360: p. 1696-1698. doi: 10.1056/NEJMp0806284 PMID: 19369660
87. C Do, Francke U, Eriksson N. Comparison of family history and SNPs for predicting risk of complex disease. PLoS Genetics. 2012; 8: p. e1002973. doi: 10.1371/journal.pgen.1002973 PMID: 23071447
88. Healy D. The Antidepressant Era: Harvard University Press; 1997.
89. Healy D. The Creation of Psychopharmacology. Cambridge, MA: New Directions Publishing Corporation; 2002.
90. De Monaco HJ, Ali A, Von Hippel E. The major role of clinicians in the discovery of off-label drug therapies. Pharmacotherapy. 2006; 26: p. 323-332. PMID: 16503712
91. Rask-Anderson M, Almen M, Schioth H. Trends in the exploitation of novel drug targets. Nat Rev Drug Discov. 2011; 10: p. 579-590. doi: 10.1038/nrd3478 PMID: 21804595
92. Landis S, Amara S, Asadulla K. A call for transparent reporting to optimize the predictive value of preclinical research. Nature. 2012; 490: p. 187-191. doi: 10.1038/nature11556 PMID: 23060188
93. Collins F, Tabak L. Policy: NIH plans to enhance reproducibility. Nature. 2014; 505: p. 612-613. PMID: 24482835
94. Alberts B, Kirschner M, Tilghman S, Varmuts H. Rescuing US biomedical research from its systematic flaws. Proc Natl Acad Sci U S A. 2014; 111: p. 5773-5777. doi: 10.1073/pnas.1404402111 PMID: 24733905
95. Young N, Ioannidis J, Al-Unaydli O. Why current publication practices may distort science. PLoS Med. 2008; 5: p. e201. doi: 10.1371/journal.pmed.0050201 PMID: 18844432
96. Fanelli D. Negative results are disappearing from most disciplines and countries. Scientometrics. 2012; 90: p. 891-904.
97. Benner S, Bains W, Seager S. Models and standards of proof in cross-disciplinary science: The case of arsenic DNA. Astrobiology. 2013; 13: p. 510-513. doi: 10.1089/ast.2012.0954 PMID: 23634974
98. McArthur R, Borsini F, editors. Animal and Translational Models for CNS Drug Discovery. Volume 1: Psychiatric Disorders Amsterdam: Elsevier; 2008.
99. Funtowicz S, Ravetz J. Uncertainty and Quality in Science for Policy. Dordrecht: Kluwer Academic Publishers; 1990.
100. Cronbach L, Meehl P. Construct validity in psychological tests. Psychol Bull. 1955; 52: p. 281-302. PMID: 13245896
101. Haibe-Kaine B, El-Hachem N, Birkbak N, Jin A, Beck A, Aerts H, et al. Inconsistency in large pharmacogenomic studies. Nature. 2013; 504: p. 389-393. doi: 10.1038/nature12831 PMID: 24284626
102. Eder J, Sedrani R, Wiesmann C. The discovery of first-in-class drugs: Origins and evolution. Nat Rev Drug Discov. 2014 July.
103. Brunke S, Quintin L, Kasper L, Jacobsen I, Richter M, Hiller EST, et al. Of mice, flies-and men? Comparing fungal infection models for large-scale screening efforts. Dis Model Mech. 2015; 8: p. 473-486 doi: 10.1242/dmm.019901 PMID: 25786415
104. Wong C, Cheng K, Rigas B. Preclinical predictors of anticancer drug efficacy: critical assessment with emphasis on whether nanomolar potency should be required of candidate agents. J Pharmacol Exp Ther. 2012; 341: p. 572-578. doi: 10.1124/jpet.112.191957 PMID: 22448039
105. Holbeck S, Collins J, Doroshow J. Analysis of Food and Drug Administration-approved anticancer agents in the NCI60 panel of human tumour cell lines. Mol Cancer Ther.; 9: p. 1451-1460. doi: 10.1158/1535-7163.MCT-10-0106 PMID: 20442306
106. Petrone P, Simms B, Nigsch F, Lounkine E, Kutchukian P, Cornett A, et al. Rethinking molecular similarity: comparing compounds on the basis of biological activity. ACS Chem Biol. 2012; 7: p. 1399-1409. doi: 10.1021/cb3001028 PMID: 22594495
107. Macleod M, Lawson McLean A, Kyriakopoulou A, Serghiou S, de Wilde A, Sherratt N, et al. Risk of bias in reports of in vivo research: A focus for improvement. PLOS Biol. 2015; 13(10): p. e1002273. doi: 10.1371/journal.pbio.1002273 PMID: 26460723