Artificial intelligence versus clinicians: Systematic review of design, reporting standards, and claims of deep learning studies in medical imaging

The BMJ

Volumn 368, Issue , 2020, Pages

  Nagendran, Myura ^a   Chen, Yang ^b   Lovejoy, Christopher A ^c   Gordon, Anthony C ^a,d   Komorowski, Matthieu ^a   Harvey, Hugh ^e   Topol, Eric J ^f   Ioannidis, John P A ^g   Collins, Gary S ^h,i   Maruthappu, Mahiben ^c  

^a IMPERIAL COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY COLLEGE LONDON   (United Kingdom)

^c CERA CARE   (United Kingdom)

^d IMPERIAL COLLEGE HEALTHCARE NHS TRUST   (United Kingdom)

^e Hardian Health   (United Kingdom)

^f SCRIPPS RESEARCH TRANSLATIONAL INSTITUTE   (United States)

^g STANFORD UNIVERSITY SCHOOL OF MEDICINE   (United States)

^h UNIVERSITY OF OXFORD   (United Kingdom)

ⁱ OXFORD UNIVERSITY HOSPITALS NHS FOUNDATION TRUST   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; ARTIFICIAL INTELLIGENCE; CATARACT; CLINICIAN; COLONOSCOPY; COMPARATIVE STUDY; CONVOLUTIONAL NEURAL NETWORK; DEEP LEARNING; DIAGNOSTIC ACCURACY; DIAGNOSTIC IMAGING; DIAGNOSTIC TEST ACCURACY STUDY; DISEASE CLASSIFICATION; HUMAN; INFORMATION PROCESSING; METHODOLOGY; PATTERN RECOGNITION; POLYP; PREDICTION; PRIORITY JOURNAL; PROTOCOL COMPLIANCE; RANDOMIZED CONTROLLED TRIAL (TOPIC); REPORTING STANDARD; REPRODUCIBILITY; RISK; SENSITIVITY AND SPECIFICITY; SYSTEMATIC REVIEW; ALGORITHM; IMAGE PROCESSING; PHYSICIAN; STATISTICAL BIAS;

ALGORITHMS; BIAS; DEEP LEARNING; DIAGNOSTIC IMAGING; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; PHYSICIANS; RANDOMIZED CONTROLLED TRIALS AS TOPIC; RESEARCH DESIGN;

EID: 85082505268     PISSN: 09598146     EISSN: 17561833     Source Type: Journal    
DOI: 10.1136/bmj.m689     Document Type: Article

References (41)

1. Carson E. IBM Watson Health computes a pair of new solutions to improve healthcare data and security. 2015. https://www. Techrepublic. com/article/ibm-watson-health-computesa-pair-of-new-solutions-to-improve-healthcare-data-and-security/.
2. Topol EJ. High-performance medicine: The convergence of human and artificial intelligence. Nat Med 2019;25:44-56. 10. 1038/s41591-018-0300-7 30617339
3. ReportLinker. Artificial intelligence in healthcare market by offering, technology, end-use application, end user and geography-global forecast to 2025. 2018. https://www. reportlinker. com/p04897122/Artificial-Intelligence-in-Healthcare-Market-by-Offering-Technology-Application-End-User-Industry-and-Geography-Global-Forecast-to. html.
4. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015;521:436-44. 10. 1038/nature14539 26017442
5. Esteva A, Robicquet A, Ramsundar B, et al. A guide to deep learning in healthcare. Nat Med 2019;25:24-9. 10. 1038/s41591-018-0316-z 30617335
6. NCBI. PubMed search for deep learning. 2019. https://www. ncbi. nlm. nih. gov/pubmed/ term=deep+learning+or+%22Deep+Learning%22%5BMesh%5D.
7. Murphy M. Google says its AI can spot lung cancer a year before doctors. 2019. https:// www. Telegraph. co. uk/technology/2019/05/07/google-says-ai-can-spot-lung-cancer-yeardoctors/.
8. Price E. AI Is better at diagnosing skin cancer than your doctor, study finds. 2018. https: //fortune. com/2018/05/30/ai-skin-cancer-diagnosis/.
9. Kappen TH, van Klei WA, van Wolfswinkel L, Kalkman CJ, Vergouwe Y, Moons KGM. Evaluating the impact of prediction models: lessons learned, challenges, and recommendations. Diagn Progn Res 2018;2:11. 10. 1186/s41512-018-0033-6 31093561
10. Psaty BM, Furberg CD. COX-2 inhibitors-lessons in drug safety. N Engl J Med 2005;352:1133-5. 10. 1056/NEJMe058042 15713946
11. Schulz KF, Altman DG, Moher DCONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ 2010;340:c332. 10. 1136/bmj. c332 20332509
12. Wallace E, Smith SM, Perera-Salazar R, et al. International Diagnostic and Prognosis Prediction (IDAPP) group. Framework for the impact analysis and implementation of Clinical Prediction Rules (CPRs). BMC Med Inform Decis Mak 2011;11:62. 10. 1186/1472-6947-11-62 21999201
13. Moher D, Liberati A, Tetzlaff J, Altman DGPRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. BMJ 2009;339:b2535. 10. 1136/bmj. b2535 19622551
14. Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD statement. BMJ 2015;350:g7594. 10. 1136/bmj. g7594 25569120
15. Moons KG, Altman DG, Reitsma JB, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 2015;162:W1-73. 10. 7326/M14-0698 25560730
16. Heus P, Damen JAAG, Pajouheshnia R, et al. Uniformity in measuring adherence to reporting guidelines: The example of TRIPOD for assessing completeness of reporting of prediction model studies. BMJ Open 2019;9:e025611. 10. 1136/bmjopen-2018-025611 31023756
17. Wolff RF, Moons KGM, Riley RD, et al. PROBAST Group. PROBAST: A tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med 2019;170:51-8. 10. 7326/M18-1376 30596875
18. Moons KGM, Wolff RF, Riley RD, et al. PROBAST: A tool to assess risk of bias and applicability of prediction model studies: explanation and elaboration. Ann Intern Med 2019;170:W1-33. 10. 7326/M18-1377 30596876
19. Higgins JP, Altman DG, Gøtzsche PC, et al. Cochrane Bias Methods GroupCochrane Statistical Methods Group. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ 2011;343:d5928. 10. 1136/bmj. d5928 22008217
20. Lin H, Li R, Liu Z, et al. Diagnostic efficacy and therapeutic decision-making capacity of an artificial intelligence platform for childhood cataracts in eye clinics: A multicentre randomized controlled trial. EClinicalMedicine 2019;9:52-9. 10. 1016/j. eclinm. 2019. 03. 001 31143882
21. Wang P, Berzin TM, Glissen Brown JR, et al. Real-time automatic detection system increases colonoscopic polyp and adenoma detection rates: A prospective randomised controlled study. Gut 2019;68:1813-9. 10. 1136/gutjnl-2018-317500 30814121
22. Han SS, Park GH, Lim W, et al. Deep neural networks show an equivalent and often superior performance to dermatologists in onychomycosis diagnosis: Automatic construction of onychomycosis datasets by region-based convolutional deep neural network. PLoS One 2018;13:e0191493. 10. 1371/journal. pone. 0191493 29352285
23. Glasziou P, Altman DG, Bossuyt P, et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet 2014;383:267-76. 10. 1016/S0140-6736(13)62228-X 24411647
24. Ioannidis JP, Greenland S, Hlatky MA, et al. Increasing value and reducing waste in research design, conduct, and analysis. Lancet 2014;383:166-75. 10. 1016/S0140-6736(13)62227-8 24411645
25. Carfagno J. 5 FDA Approved Uses of AI in Healthcare. 2019. https://www. docwirenews. com/docwire-pick/future-of-medicine-picks/fda-approved-uses-of-ai-in-healthcare/.
26. Fleming TR, DeMets DL. Surrogate end points in clinical trials: Are we being misledAnn Intern Med 1996;125:605-13. 10. 7326/0003-4819-125-7-199610010-00011 8815760
27. Section FDA. 510(k) premarket notification of intent to market the device. 2018. https:// www. Accessdata. fda. gov/cdrh_docs/pdf18/K180647. pdf.
28. FDA. Artificial Intelligence and Machine Learning in Software as a Medical Device. 2019. https://www. fda. gov/medical-devices/software-medical-device-samd/artificial-intelligenceand-machine-learning-software-medical-device.
29. Camerer CF, Dreber A, Holzmeister F, et al. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nat Hum Behav 2018;2:637-44. 10. 1038/s41562-018-0399-z 31346273
30. Ebrahim S, Sohani ZN, Montoya L, et al. Reanalyses of randomized clinical trial data. JAMA 2014;312:1024-32. 10. 1001/jama. 2014. 9646 25203082
31. Wallach JD, Boyack KW, Ioannidis JPA. Reproducible research practices, transparency, and open access data in the biomedical literature, 2015-2017. PLoS Biol 2018;16:e2006930. 10. 1371/journal. pbio. 2006930 30457984
32. Van Calster B, Steyerberg EW, Collins GS. Artificial Intelligence Algorithms for Medical Prediction Should Be Nonproprietary and Readily Available. JAMA Intern Med 2019;179:731. 10. 1001/jamainternmed. 2019. 0597 31058938
33. Hwang D-K, Hsu C-C, Chang K-J, et al. Artificial intelligence-based decision-making for age-related macular degeneration. Theranostics 2019;9:232-45. 10. 7150/thno. 28447 30662564
34. Sumner P, Vivian-Griffiths S, Boivin J, et al. Exaggerations and caveats in press releases and health-related science news. PLoS One 2016;11:e0168217. 10. 1371/journal. pone. 0168217 27978540
35. Boutron I, Haneef R, Yavchitz A, et al. Three randomized controlled trials evaluating the impact of "spin" in health news stories reporting studies of pharmacologic treatments on patients'/caregivers' interpretation of treatment benefit [correction in: BMC Med 2019;17:147]. BMC Med 2019;17:105. 10. 1186/s12916-019-1330-9 31159786
36. Vosoughi S, Roy D, Aral S. The spread of true and false news online. Science 2018;359:1146-51. 10. 1126/science. Aap9559 29590045
37. Vollmer S, Mateen BA, Bohner G, et al. Machine learning and AI research for patient benefit: 20 critical questions on transparency, replicability, ethics and effectiveness. 2018. https://arxiv. org/abs/1812. 10404.
38. Collins GS, Moons KGM. Reporting of artificial intelligence prediction models. Lancet 2019;393:1577-9. 10. 1016/S0140-6736(19)30037-6 31007185
39. De Angelis C, Drazen JM, Frizelle FA, et al. International Committee of Medical Journal Editors. Clinical trial registration: A statement from the International Committee of Medical Journal Editors. N Engl J Med 2004;351:1250-1. 10. 1056/NEJMe048225 15356289
40. Allen B. The role of the FDA in ensuring the safety and efficacy of artificial intelligence software and devices. J Am Coll Radiol 2019;16:208-10. 10. 1016/j. jacr. 2018. 09. 007 30389329
41. Ratner M. FDA backs clinician-free AI imaging diagnostic tools. Nat Biotechnol 2018;36:673-4. 10. 1038/nbt0818-673a 30080822