Grader Variability and the Importance of Reference Standards for Evaluating Machine Learning Models for Diabetic Retinopathy

Ophthalmology

Volumn 125, Issue 8, 2018, Pages 1264-1272

  Krause, Jonathan ^a   Gulshan, Varun ^a   Rahimy, Ehsan ^b   Karth, Peter ^c   Widner, Kasumi ^a   Corrado, Greg S ^a   Peng, Lily ^a   Webster, Dale R ^a  

^a GOOGLE INC   (United States)

^b PALO ALTO MEDICAL FOUNDATION   (United States)

^c Oregon Cardiology PC   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; AREA UNDER THE CURVE; ARTICLE; ARTIFACT; CLINICAL EVALUATION; CLINICAL PROTOCOL; CONSENSUS; DIABETIC RETINOPATHY; DIAGNOSTIC TEST ACCURACY STUDY; DISEASE SEVERITY; EYE FUNDUS; FEMALE; HUMAN; IMAGE QUALITY; LEARNING ALGORITHM; MACHINE LEARNING; MACULAR EDEMA; MAJOR CLINICAL STUDY; MALE; MEDICAL SPECIALIST; MICROANEURYSM; MIDDLE AGED; OPHTHALMOLOGIST; PREDICTION; PRIORITY JOURNAL; RETINA; RETINA EXUDATE; RETROSPECTIVE STUDY; SCREENING; SENSITIVITY AND SPECIFICITY; STANDARD; UNITED STATES; ALGORITHM; CLINICAL COMPETENCE; MASS SCREENING; RECEIVER OPERATING CHARACTERISTIC; STANDARDS;

ALGORITHMS; CLINICAL COMPETENCE; DIABETIC RETINOPATHY; FEMALE; HUMANS; MACHINE LEARNING; MALE; MASS SCREENING; MIDDLE AGED; OPHTHALMOLOGISTS; REFERENCE STANDARDS; RETROSPECTIVE STUDIES; ROC CURVE;

EID: 85043470011     PISSN: 01616420     EISSN: 15494713     Source Type: Journal    
DOI: 10.1016/j.ophtha.2018.01.034     Document Type: Article

References (29)

1. Early Treatment Diabetic Retinopathy Study Research Group. Grading diabetic retinopathy from stereoscopic color fundus photographs–an extension of the modified Airlie House classification. ETDRS report number 10. Ophthalmology 98 (1991), 786–806.
2. Diabetic Retinopathy Screening Services in Scotland: A Training Handbook – July 2003: page 17. Available at: http://www.ndrs-wp.scot.nhs.uk/?page_id=1609 Accessed June 21, 2017.
3. American Academy of Ophthalmology. International clinical diabetic retinopathy disease severity scale, detailed table. Available at: http://www.icoph.org/dynamic/attachments/resources/diabetic-retinopathy-detail.pdf Accessed October 14, 2016.
4. Scott, I.U., Bressler, N.M., Bressler, S.B., et al. Agreement between clinician and reading center gradings of diabetic retinopathy severity level at baseline in a phase 2 study of intravitreal bevacizumab for diabetic macular edema. Retina 28 (2008), 36–40.
5. Li, H.K., Hubbard, L.D., Danis, R.P., et al. Digital versus film fundus photography for research grading of diabetic retinopathy severity. Invest Ophthalmol Vis Sci 51 (2010), 5846–5852.
6. Gangaputra, S., Lovato, J.F., Hubbard, L., et al. Comparison of standardized clinical classification with fundus photograph grading for the assessment of diabetic retinopathy and diabetic macular edema severity. Retina 33 (2013), 1393–1399.
7. Ruamviboonsuk, P., Teerasuwanajak, K., Tiensuwan, M., Yuttitham, K., Thai Screening for Diabetic Retinopathy Study Group. Interobserver agreement in the interpretation of single-field digital fundus images for diabetic retinopathy screening. Ophthalmology 113 (2006), 826–832.
8. Elmore, J.G., Wells, C.K., Lee, C.H., et al. Variability in radiologists’ interpretations of mammograms. N Engl J Med 331 (1994), 1493–1499.
9. Elmore, J.G., Longton, G.M., Carney, P.A., et al. Diagnostic concordance among pathologists interpreting breast biopsy specimens. JAMA 313 (2015), 1122–1132.
10. LeCun, Y., Yoshua, B., Geoffrey, H., Deep learning. Nature 521 (2015), 436–444.
11. Esteva, A., Kuprel, B., Novoa, R.A., et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542 (2017), 115–118.
12. Liu Y, Gadepalli K, Norouzi M, et al. Detecting cancer metastases on Gigapixel Pathology Images. arXiv [csCV] 2017. Available at: http://arxiv.org/abs/1703.02442.
13. Bejnordi, B.E., Veta, M., van Diest, P.J., et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA 318 (2017), 2199–2210.
14. Gulshan, V., Peng, L., Coram, M., et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316 (2016), 2402–2410.
15. Gargeya, R., Leng, T., Automated identification of diabetic retinopathy using deep learning. Ophthalmology 124 (2017), 962–969.
16. Ting, D.S.W., Cheung, C.Y.-L., Lim, G., et al. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA 318 (2017), 2211–2223.
17. Decencière, E., Etienne, D., Xiwei, Z., et al. Feedback on a publicly distributed image database: the Messidor Database. Image Anal Stereol, 2014, 0 Available at: http://dx.doi.org/10.5566/ias.1155.
18. Quellec, G., Lamard, M., Josselin, P.M., et al. Optimal wavelet transform for the detection of microaneurysms in retina photographs. IEEE Trans Med Imaging 27 (2008), 1230–1241.
19. EyePACS. EyePACS digital retinal image grading protocol narrative. Available at: https://www.eyepacs.org/consultant/Clinical/grading/EyePACS-DIGITAL-RETINAL-IMAGE-GRADING.pdf.
20. Lecun, Y., Bottou, L., Bengio, Y., Haffner, P., Gradient-based learning applied to document recognition. Proc IEEE 86 (1998), 2278–2324.
21. Cuadros, J., Bresnick, G., EyePACS: an adaptable telemedicine system for diabetic retinopathy screening. J Diabetes Sci Technol 3 (2009), 509–516.
22. Golovin D, Solnik B, Moitra S, et al. Google Vizier: A Service for Black-Box Optimization. Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 2017:1487–1495. Accessed June 30, 2017.
23. Kolb, H., Facts and figures concerning the human retina. Kolb, H., Fernandez, E., Nelson, R., (eds.) Webvision: The Organization of the Retina and Visual System, 2005, University of Utah Health Sciences Center, Salt Lake City, UT.
24. A Practical Manual of Diabetic Retinopathy Management. Clin Exp Optom 92 (2009), 527–528.
25. Szegedy C, Vanhouke V, Ioffe S, et al. Rethinking the inception architecture for computer vision. arXiv preprint arXiv. 2015;1502.03167.
26. Szegedy C, Ioffe S, Vanhoucke V, Alemi A. Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning. 2016. Thirty-First AAAI Conference on Artificial Intelligence 2016. Available at: http://www.aaai.org/ocs/index.php/AAAI/AAAI17/paper/download/14806/14311. Accessed December 13, 2017.
27. Cohen, J.A., Coefficient of agreement for nominal scales. Educ Psychol Meas 20 (1960), 37–46.
28. Gonzalez, M.E., Gonzalez, C., Stern, M.P., et al. Concordance in diagnosis of diabetic retinopathy by fundus photography between retina specialists and a standardized reading center. Mexico City Diabetes Study Retinopathy Group. Arch Med Res 26 (1995), 127–131.
29. Abràmoff, M.D., Folk, J.C., Han, D.P., et al. Automated analysis of retinal images for detection of referable diabetic retinopathy. JAMA Ophthalmol 131 (2013), 351–357.