How to increase value and reduce waste when research priorities are set

The Lancet

Volumn 383, Issue 9912, 2014, Pages 156-165

  Correction Chalmers, Iain ^a   Bracken, Michael B ^b   Djulbegovic, Ben ^c,d   Garattini, Silvio ^e   Grant, Jonathan ^f   Gülmezoglu, A Metin ^g   Howells, David W ^h   Ioannidis, John P A ^i,j   Oliver, Sandy ^k  

^a James Lind Initiative   (United Kingdom)

^b YALE UNIVERSITY SCHOOL OF MEDICINE   (United States)

^c UNIVERSITY OF SOUTH FLORIDA   (United States)

^d H LEE MOFFITT CANCER CENTER AND RESEARCH INSTITUTE   (United States)

^e MARIO NEGRI INSTITUTE FOR PHARMACOLOGICAL RESEARCH   (Italy)

^f RAND CORPORATION   (United States)

^g WORLD BANK   (United States)

^h FLOREY INSTITUTE OF NEUROSCIENCE AND MENTAL HEALTH   (Australia)

ⁱ STANFORD UNIVERSITY SCHOOL OF MEDICINE   (United States)

^j STANFORD UNIVERSITY   (United States)

^k UNIVERSITY OF LONDON   (United Kingdom)

more..

Author keywords

[No Author keywords available]

Indexed keywords

APPLIED RESEARCH; BASIC RESEARCH; HUMAN; INVESTMENT; MEDICAL RESEARCH; PRIORITY JOURNAL; RECYCLING; RESEARCH PRIORITY; REVIEW;

EID: 84892142354     PISSN: 01406736     EISSN: 1474547X     Source Type: Journal    
DOI: 10.1016/S0140-6736(13)62229-1     Document Type: Review

References (111)

1. Stokes DE. Pasteur's quadrant-basic science and technological innovation. Washington, DC: Brookings Institution Press, 1997.
2. Altman D. The scandal of poor medical research. BMJ 1994; 308: 283-84.
3. Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet 2009; 374: 86-89.
4. Macleod R, Michie S, Roberts I, et al. Biomedical research: increasing value, reducing waste. Lancet 2014; published online Jan 8. http://dx.doi.org/ 10.1016/S0140-6736(13)62329-6.
5. Rottingen J-A, Regmi S, Eide M, et al. Mapping of available health research and development data: what's there, what's missing, and what role is there for a global observatory? Lancet 2013; 382: 1286-307.
6. Rothwell PM. Funding for practice-oriented clinical research. Lancet 2006; 368: 262-66.
7. UK Clinical Research Collaboration. UK health research analysis 2009/10. November, 2012. http://www.ukcrc.org/publications/reports/(accessed Nov 14, 2013).
8. Collins FS. NIH basics. Science 2012; 337: 503.
9. Comroe JH, Dripps RD. Scientifi c basis for the support of biomedical science. Science 1976; 192: 105-11.
10. Smith R. Comroe and Dripps revisited. BMJ 1987; 295: 1404-07.
11. Grant J, Green L, Mason B. Basic research and health: a reassessment of the scientifi c basis for the support of biomedical science. Res Eval 2003; 12: 217-24.
12. Chien A, Edgar DB, Trela JM. Deoxyribonucleic acid polymerase from the extreme thermophile Thermus aquaticus. J Bacteriol 1976; 127: 1550-57.
13. Rosenberg B, Van Camp L, Krigas T. Inhibition of cell division in Escherichia coli by electrolysis products from a platinum electrode. Nature 1965; 205: 698-99.
14. Endo A, Kuroda M, Tsujita Y. ML-236A, ML-236B, and ML-236C, new inhibitors of cholesterogenesis produced by Penicillium citrinium. J Antibiot (Tokyo) 1976; 29: 1346-48.
15. Contopoulos-Ioannidis DG, Alexiou GA, Gouvias TC, Ioannidis JPA. Life cycle of translational research for medical interventions. Science 2008; 321: 1298-99.
16. Wooding S, Hanney S, Buxton M, Grant J. The returns from arthritis research volume 1: approach, analysis and recommendations. 2004. http://www.rand.org/pubs/monographs/MG251.html (accessed Nov 14, 2013).
17. Wooding H, Hanney S, Pollitt A, Buxton M, Grant J. Project Retrosight: understanding the returns from cardiovascular and stroke research: the policy report. 2011. http://www.rand.org/pubs/monographs/MG1079.html (accessed Nov 14, 2013).
18. Wooding S, Pollitt A, Castle-Clark S, et al. Mental Health Retrosight: understanding the returns from research (lessons from schizophrenia): policy report. 2013. http://www.rand.org/pubs/research-reports/RR325.html (accessed Nov 14, 2013).
19. Morris ZS, Wooding S, Grant J. The answer is 17 years, what is the question: understanding time lags in translational research. J R Soc Med 2011; 104: 510-20.
20. Mestre-Ferrandiz J, Sussex J, Towse A. The R&D cost of a new medicine. London: Offi ce of Health Economics, 2012.
21. Chalmers I, Rounding C, Lock K. Descriptive survey of non-commercial randomised controlled trials in the United Kingdom, 1980-2002. BMJ 2003; 327: 1017-19.
22. Cooksey D. A review of UK health research funding. December, 2006. http://www.offi cial-documents.gov.uk/document/other/0118404881/0118404881.pdf (accessed Nov 14, 2013).
23. Institute of Medicine. Learning what works: infrastructure required for comparative eff ectiveness research: workshop summary. September, 2007. http://www.ncbi.nlm.nih.gov/books/NBK64784/(accessed Nov 25, 2013).
24. Italian Medicines Agency (AIFA) Research and Development Working Group. Feasibility and challenges of independent research on drugs: the Italian Medicines Agency (AIFA) experience. Eur J Clin Invest 2010; 40: 69-86.
25. Marburger JH. Wanted: better benchmarks. Science 2005; 308: 1087.
26. Ioannidis JPA. Fund people not projects. Nature 2011; 477: 529-31.
27. Mervis J. White House panel urges agencies to take more risks. Science 2012; 338: 1274.
28. Horrobin D. The philosophiocal basis of peer review and the suppression of innovation. JAMA 1990; 263: 1438-41.
29. Boudreau KJ, Guinan EC, Lakhani KR, Riedl C. The novelty paradox and bias for normal science: evidence from randomized medical grant proposal evaluations. Jan 10, 2013. http://hbswk.hbs.edu/item/7173.html (accessed Nov 25, 2013).
30. Nicholson JM, Ioannidis JPA. Research grants: conform and be funded. Nature 2012; 492: 34-36.
31. Nasser M, Welch V, Ueffi ng E, Crowe S, Oliver S, Carlo R. Evidence in agenda setting: new directions for the Cochrane Collaboration. J Clin Epidemiol 2013; 66: 469-71.
32. Gillum LA, Gouveia C, Dorsey ER, et al. NIH disease funding levels and burden of disease. PLoS One 2011; 6: e16837.
33. Liberati A. Need to realign patient-oriented and commercial and academic research. Lancet 2011; 378: 1777-78.
34. Tallon D, Chard J, Dieppe P. Relation between agendas of the research community and the research consumer. Lancet 2000; 355: 2037-40.
35. Corner J, Wright D, Hopkinson J, Gunaratnam Y, McDonald JW, Foster C. The research priorities of patients attending UK cancer treatment centres: fi ndings from a modifi ed nominal group study. Br J Cancer 2007; 96: 875-81.
36. Garattini S, Bertele V. Ethics in clinical research. J Hepatol 2009; 51: 792-97.
37. Milne R. Burden of disease and research funding. July 20, 2012. http://rm21.typepad.com/blog/2012/07/burden-of-disease.html (accessed Nov 14, 2013).
38. Vesterinen HM, Sena ES, ff rench-Constant C, Williams A, Chandran S, Macleod MR. Improving the translational hit of experimental treatments in multiple sclerosis. Mult Scler 2010; 16: 1044-55.
39. Vesterinen HM, Irvine CMJ, Macleod MR, et al. Meta-analysis of data from animal studies: a practical guide. J Neurosci Methods 2013; 221: 92-102.
40. Caron-Flinterman F, Broerse JEW, Teerling J, et al. Stakeholder participation in health research agenda setting: the case of asthma and COPD research in the Netherlands. Sci Public Policy 2006; 33: 291-304.
41. Cartwright J, Crowe S. Patient and public involvement toolkit. Oxford: Blackwell, 2011.
42. Djulbegovic B, Kumar A, Glasziou P, Miladinovic B, Chalmers I. Trial unpredictability yields predictable therapy gains. Nature 2013; 500: 395-96.
43. Ioannidis JPA, Greenland S, Hlatky MA, et al. Increasing value and reducing waste in research design, conduct, and analysis. Lancet 2014; published online Jan 8. http://dx.doi.org/10.1016/S0140- 6736(13)62227-8.
44. Prinz F, Schlange T, Asadullah K. Believe it or not: how much can we rely on published data on potential drug targets? Nat Rev Drug Discov 2011; 10: 712-13.
45. Begley CG, Ellis LM. Drug development: raise standards for preclinical cancer research. Nature 2012; 483: 531-33.
46. Osherovich L. Hedging against academic risk. April 14, 2011. http://www.nature.com/scibx/journal/v4/n15/full/scibx.2011.416.html (accessed Nov 26, 2013).
47. Ioannidis JPA. Expectations, validity, and reality in omics. J Clin Epidemiol 2010; 63: 945-49.
48. Ioannidis JPA, Trikalinos TA. Early extreme contradictory estimates may appear in published research: the Proteus phenomenon in molecular genetics research and randomized trials. J Clin Epidemiol 2005; 58: 543-49.
49. Nature Immunology. Raising standards. Nat Immunol 2013; 14: 415.
50. Chang SM, Carey T, Kato EU, Guise J-M, Sanders GD. Identifying research needs for improving health care. Ann Intern Med 2012; 157: 439-45.
51. Barnett K, Mercer SW, Norbury M, Watt G, Wyke S, Guthrie B. Epidemiology of multimorbidity and implications for health care, research, and medical education: a cross-sectional study. Lancet 2012; 380: 37-43.
52. Oliver S. Exploring lay perspectives on questions of eff ectiveness. In: Chalmers I, Maynard A, eds. Non-random refl ections on health services research. London: BMJ Publishing Group, 1997: 272-91.
53. Oliver S. Users of health services: following their agenda. In: Hood S, Mayall B, Oliver S, eds. Critical issues in social research: power and prejudice. Buckingham: Open University Press, 1999: 139-53.
54. McKevitt C, Redfern J, Mold F, Wolfe C. Qualitative studies of stroke: a systematic review. Stroke 2004; 35: 1499-1505.
55. Staley K, Hanley B. Scoping research priority setting, and the presence of patient and public involvement, with UK clinical research organisations and funders. http://www.lindalliance.org/Scoping-research-priority-setting-PPI.asp (accessed Nov 14, 2013).
56. Department of Health. Research governance framework for health and social care. Feb 28, 2001. http://webarchive.nationalarchives. gov.uk/+/www.dh.gov.uk/ en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/ DH-4008777 (accessed Nov 14, 2013).
57. Jones AP, Conroy E, Williamson PR, Clarke M, Gamble C. The use of systematic reviews in the planning, design and conduct of randomised trials: a retrospective cohort of NIHR HTA funded trials. BMC Med Res Methodol 2013; 13: 50.
58. Ioannidis JPA, Karassa FB. The need to consider the wider agenda in systematic reviews and meta-analyses: breadth, timing, and depth of the evidence. BMJ 2010; 341: c4875.
59. Fergusson D, Glass KC, Hutton B, Shapiro S. Randomized controlled trials of aprotinin in cardiac surgery: could clinical equipoise have stopped the bleeding? Clin Trials 2005; 2: 218-29.
60. Ker K, Edwards P, Perel P, Shakur H, Roberts I. Eff ect of tranexamic acid on surgical bleeding: systematic review and cumulative meta-analysis. BMJ 2012; 344: e3054.
61. Robinson KA, Goodman SN. A systematic examination of the citation of prior research in reports of randomized, controlled trials. Ann Intern Med 2011; 154: 50-55.
62. Cooper N, Jones D, Sutton A. The use of systematic reviews when designing studies. Clin Trials 2005; 2: 260-64.
63. Goudie AC, Sutton AJ, Jones DR, Donald A. Empirical assessment suggests that existing evidence could be used more fully in designing randomised controlled trials. J Clin Epidemiol 2010; 63: 983-91.
64. Clarke M, Hopewell S. Many reports of randomised trials still don't begin or end with a systematic review of the relevant evidence. J Bahrain Med Soc 2013; 24: 145-48.
65. Clark T, Berger U, Mansmann U. Sample size determinations in original research protocols for randomised clinical trials submitted to UK research ethics committees: review. BMJ 2013; 346: f1136.
66. Chalmers I, Matthews R. What are the implications of optimism bias in clinical research? Lancet 2006; 367: 449-50.
67. Djulbegovic B, Kumar A, Magazin A, et al. Optimism bias leads to inconclusive results-an empirical study. J Clin Epidemiol 2011; 64: 583-93.
68. Gotzsche P. Reference bias in reports of drug trials. BMJ 1987; 295: 654-56.
69. Greenberg SA. How citation distortions create unfounded authority: analysis of citation network. BMJ 2009; 339: b2680.
70. Sena ES, Briscoe CL, Howells DW, Donnan GA, Sandercock PA, Macleod MR. Factors aff ecting the apparent effi cacy and safety of tissue plasminogen activator in thrombotic occlusion models of stroke: systematic review and meta-analysis. J Cereb Blood Flow Metab 2010; 30: 1905-13.
71. Clark O, Clark L, Djulbegovic B. Is clinical research still too haphazard? Lancet 2001; 358: 1648.
72. Kenter MJH, Cohen AF. Establishing risk of human experimentation with drugs: lessons from TGN1412. Lancet 2006; 368: 1387-91.
73. Horn J, de Haan RJ, Vermeulen M, Luiten PGM, Limburg M. Nimodipine in animal model experiments of focal cerebral ischemia: a systematic review. Stroke 2001; 32: 2433-38.
74. Lee DS, Nguyen QT, Lapointe N, et al. Meta-analysis of ther efects of endothelin receptor blockade on survival in experimental heart failure. J Card Fail 2003; 9: 368-74.
75. Chalmers I. The lethal consequences of failing to make use of all relevant evidence about the eff ects of medical treatments: the need for systematic reviews. In: Rothwell P, ed. Treating individuals: from randomised trials to personalised medicine. London: Elsevier, 2007: 37-58.
76. Al-Shahi Salman R, Beller E, Kagan J, et al. Increasing value and reducing waste in biomedical research regulation and management. Lancet 2014; published online Jan 8. http://dx.doi.org/10.1016/S0140-6736(13)62297-7.
77. Sinclair JC. Meta-analysis of randomized controlled trials of antenatal corticosteroid for the prevention of respiratory distress syndrome: discussion. Am J Obstet Gynecol 1995; 173: 335-44.
78. Gilbert R, Salanti G, Harden M, See S. Infant sleeping position and the sudden infant death syndrome: systematic review of observational studies and historical review of recommendations from 1940 to 2002. Int J Epidemiol 2005; 34: 874-87.
79. Taylor R, Cumming R, Woodward A, Black M. Passive smoking and lung cancer: a cumulative meta-analysis. Aust N Z J Public Health 2001; 25: 203-11.
80. Rothwell PM, Robertson G. Meta-analyses of randomised controlled trials. Lancet 1997; 350: 1181-82.
81. Davey Smith G, Egger M. Meta-analyses of randomised controlled trials. Lancet 1997; 350: 1182.
82. Pereira TV, Horwitz RI, Ioannidis JP. Empirical evaluation of very large treatment eff ects of medical interventions. JAMA 2012; 308: 1676-84.
83. Rothwell PM. Treating individuals: from randomised trials to personalised medicine. London: Elsevier, 2007.
84. Chan A-W, Song F, Vickers A, et al. Increasing value and reducing waste: addressing inaccessible research. Lancet 2014; published online Jan 8. http://dx.doi.org/10.1016/S0140-6736(13)62296-5.
85. Hopewell S, Clarke MJ, Stewart L, Tierney J. Time to publication for results of clinical trials. Cochrane Database Syst Rev 2007; 2: MR000011.
86. Pereira TV, Ioannidis JPA. Statistically signifi cant meta-analyses of clinical trials have modest credibility and infl ated eff ects. J Clin Epidemiol 2011; 64: 1060-69.
87. Filley J. If a job is worth doing, it is worth doing twice. Nature 2013; 496: 7.
88. Stilgoe D. Nanodialogues: experiments in public engagement with science. London: Demos, 2007.
89. Williamson P, Altman DG, Blazeby JM, et al. Developing core outcome sets for clinical trials: issues to consider. Trials 2012; 13: 132.
90. Chalmers I, Atkinson P, Fenton M, Firkins L, Crowe S, Cowan K. Tackling treatment uncertainties together: the evolution of the James Lind Initiative (JLI), 2003-2013. J R Soc Med 2013; published online July 3. DOI:10.1177/0141076813493063.
91. Staley K. Exploring impact: public involvement in NHS, public health and social care research. Eastleigh: INVOLVE, 2009.
92. Lloyd K, White J. Democratizing clinical research. Nature 2011; 474: 277-78.
93. National Institute for Health Research. Improving memory after traumatic brain injury. July 26, 2012. http://www.nets.nihr.ac.uk/news/archive/2012/ improving-memory-after-Traumatic-brain-injury (accessed Nov 14, 2013).
94. Elwyn G, Crowe S, Fenton M, et al. Identifying and prioritizing uncertainties: patient and clinician engagement in the identifi cation of research questions. J Eval Clin Pract 2010; 16: 627-31.
95. Broerse JE, Zweekhorst MB, van Rensen AJ, de Haan MJ. Involving burn survivors in agenda setting on burn research: an added value? Burns 2012; 36: 217-31.
96. Lophatananon A, Tyndale-Biscoe S, Malcolm E, et al. The James Lind Alliance approach to priority setting for prostate cancer research: an integrative methodology based on patient and clinician participation. BJU Int 2011; 108: 1040-43.
97. Boote JD, Dalgleish M, Freeman J, Jones Z, Miles M, Rodgers H. But is it a question worth asking? A refl ective case study describing how public involvement can lead to researchers' ideas being abandoned. Health Expect 2012; published online May 31. DOI:10.1111/j.1369-7625.2012.00771.x.
98. Oliver S, Armes DG, Gyte G. Public involvement infl uences a national research agenda. Patient 2009; 2: 179-90.
99. Poliakoff E, Webb TL. What factors predict scientists' intentions to participate in public engagement of science activities? Sci Commun 2007; 29: 242-63.
100. Stewart RJ, Caird J, Oliver K, Oliver S. Patients' and clinicians' research priorities. Health Expect 2011; 14: 439-48.
101. Bausell BB. After the meta-analytic revolution. Eval Health Prof 1993; 16: 3-12.
102. Song F, Loke YK, Walsh T, et al. Methodological problems in the use of indirect comparisons for evaluating healthcare interventions: survey of published systematic reviews. BMJ 2009; 338: b1147.
103. Bracken MB. Risk, chance and causation: investigating the origins and causation of disease. New Haven: Yale University Press, 2013.
104. Ganann R, Ciliska D, Thomas H. Expediting systematic reviews: methods and implications of rapid reviews. Implement Sci 2010; 5: 56.
105. Tsafnat G, Dunn A, Glasziou P, Coiera E. The automation of systematic reviews. BMJ 2013; 346: f139.
106. Glasziou P, Altman DH, Bossuyt P, et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet 2014; published online Jan 8. http://dx.doi.org/10.1016/S0140- 6736(13)62228-X.
107. Booth A, Clarke M, Ghersi D, Moher D, Petticrew M, Stewart L. An international registry of systematic review protocols. Lancet 2011; 377: 108-09.
108. Bracken MB. Preregistration of epidemiology protocols: a commentary in support. Epidemiology 2011; 22: 135-37.
109. Vandenbroucke JP. Registering observational research: second thoughts. Lancet 2010; 375: 982-83.
110. Hooijmans CR, Ritskes-Hotinga M. Progress in using systematic reviews of animal studies to improve translational research. PLoS Med 2013; 10: e1001482.
111. Chalmers I. Using systematic reviews and registers of ongoing trials for scientifi c and ethical trial design, monitoring, and reporting. In: Egger M, Davey Smith G, Altman D, eds. Systematic reviews in health care: meta-analysis in context, 2nd edn. London: BMJ Books, 2001: 429-43.