Rapid grading of fundus photographs for diabetic retinopathy using crowdsourcing

Journal of Medical Internet Research

Volumn 16, Issue 10, 2014, Pages

  Brady, Christopher J ^a   Villanti, Andrea C ^b   Pearson, Jennifer L ^a   Kirchner, Thomas R ^b   Gupta, Omesh P ^a   Shah, Chirag P ^c  

^a JOHNS HOPKINS UNIVERSITY   (United States)

^b Schroeder Institute for Tobacco Research and Policy Studies   (United Kingdom)

^c Ophthalmic Consultants of Boston   (United Kingdom)

Author keywords

Amazon mechanical turk; Crowdsourcing; Diabetic retinopathy; Fundus photography; Telemedicine

Indexed keywords

CROWDSOURCING; DIABETIC RETINOPATHY; ECONOMICS; EYE FUNDUS; HUMAN; INTERNET; PATHOLOGY; PHOTOGRAPHY; PROCEDURES; REPRODUCIBILITY; TELEMEDICINE; VISUAL SYSTEM EXAMINATION;

CROWDSOURCING; DIABETIC RETINOPATHY; DIAGNOSTIC TECHNIQUES, OPHTHALMOLOGICAL; FUNDUS OCULI; HUMANS; INTERNET; PHOTOGRAPHY; REPRODUCIBILITY OF RESULTS; TELEMEDICINE;

EID: 84987648115     PISSN: None     EISSN: 14388871     Source Type: Journal    
DOI: 10.2196/jmir.3807     Document Type: Article

References (37)

1. Beck RW. The burgeoning public health impact of diabetes: The role of the ophthalmologist. Arch Ophthalmol 2011 Feb;129(2):225-229. [doi:10.1001/archophthalmol.2010.331] [Medline: 21320972].
2. Javitt JC, Canner JK, Frank RG, Steinwachs DM, Sommer A. Detecting and treating retinopathy in patients with type I diabetes mellitus. A health policy model. Ophthalmology 1990 Apr;97(4):483-94; discussion 494. [Medline: 2109299].
3. Javitt JC, Aiello LP. Cost-effectiveness of detecting and treating diabetic retinopathy. Ann Intern Med 1996 Jan 1;124(1 Pt2):164-169. [Medline: 8554212].
4. Jones S, Edwards RT. Diabetic retinopathy screening: A systematic review of the economic evidence. Diabet Med 2010 Mar;27(3):249-256. [doi:10.1111/j.1464-5491.2009.02870.x] [Medline: 20536486].
5. Schoenfeld ER, Greene JM, Wu SY, Leske MC. Patterns of adherence to diabetes vision care guidelines: Baseline findings from the Diabetic Retinopathy Awareness Program. Ophthalmology 2001 Mar;108(3):563-571. [Medline: 11237912].
6. Paz SH, Varma R, Klein R, Wu J, Azen SP, Los Angeles Latino Eye Study Group. Noncompliance with vision care guidelines in Latinos with type 2 diabetes mellitus: The Los Angeles Latino Eye Study. Ophthalmology 2006 Aug;113(8):1372-1377. [doi:10.1016/j.ophtha.2006.04.018] [Medline: 16769120].
7. Lee PP, Feldman ZW, Ostermann J, Brown DS, Sloan FA. Longitudinal rates of annual eye examinations of persons with diabetes and chronic eye diseases. Ophthalmology 2003 Oct;110(10):1952-1959. [doi:10.1016/S0161-6420(03)00817-0] [Medline: 14522771].
8. Fong DS, Sharza M, Chen W, Paschal JF, Ariyasu RG, Lee PP. Vision loss among diabetics in a group model Health Maintenance Organization (HMO). Am J Ophthalmol 2002 Feb;133(2):236-241. [Medline: 11812428].
9. Sharp PF, Olson J, Strachan F, Hipwell J, Ludbrook A, O'Donnell M, et al. The value of digital imaging in diabetic retinopathy. Health Technol Assess 2003;7(30): 1-119 [FREE Full text] [Medline: 14604499].
10. Scanlon PH. The English national screening programme for sight-Threatening diabetic retinopathy. J Med Screen 2008;15(1):1-4. [doi:10.1258/jms.2008.008015] [Medline: 18416946].
11. Ng M, Nathoo N, Rudnisky CJ, Tennant MT. Improving access to eye care: Teleophthalmology in Alberta, Canada. J Diabetes Sci Technol 2009 Mar;3(2):289-296 [FREE Full text] [Medline: 20144360].
12. Teng T, Lefley M, Claremont D. Progress towards automated diabetic ocular screening: A review of image analysis and intelligent systems for diabetic retinopathy. Med Biol Eng Comput 2002 Jan;40(1):2-13. [Medline: 11954703].
13. Abràmoff M, Folk JC, Han DP, Walker JD, Williams DF, Russell SR, et al. Automated analysis of retinal images for detection of referable diabetic retinopathy. JAMA Ophthalmol 2013 Mar;131(3):351-357. [doi:10.1001/jamaophthalmol.2013.1743] [Medline: 23494039].
14. Trucco E, Ruggeri A, Karnowski T, Giancardo L, Chaum E, Hubschman JP, et al. Validating retinal fundus image analysis algorithms: issues and a proposal. Invest Ophthalmol Vis Sci 2013 May;54(5):3546-3559 [FREE Full text] [doi:10.1167/iovs.12-10347] [Medline: 23794433].
15. Brabham DC. Crowdsourcing. The MIT Press Essential Knowledge Series. Cambridge, MA: MIT Press; 2013.
16. Brabham DC, Ribisl KM, Kirchner TR, Bernhardt JM. Crowdsourcing applications for public health. Am J Prev Med 2014 Feb;46(2):179-187. [doi:10.1016/j.amepre.2013.10.016] [Medline: 24439353].
17. Luengo-Oroz MA, Arranz A, Frean J. Crowdsourcing malaria parasite quantification: An online game for analyzing images of infected thick blood smears. J Med Internet Res 2012;14(6):e167 [FREE Full text] [doi:10.2196/jmir.2338] [Medline: 23196001].
18. Mitry D, Peto T, Hayat S, Morgan JE, Khaw KT, Foster PJ. Crowdsourcing as a novel technique for retinal fundus photography classification: Analysis of images in the EPIC Norfolk cohort on behalf of the UK Biobank Eye and Vision Consortium. Plos One 2013;8(8):e71154 [FREE Full text] [doi:10.1371/journal.pone.0071154] [Medline: 23990935].
19. Li HK, Horton M, Bursell SE, Cavallerano J, Zimmer-Galler I, Tennant M, American Telemedicine Association Diabetic Retinopathy Telehealth Practice Recommendations Working Group, et al. Telehealth practice recommendations for diabetic retinopathy, second edition. Telemed J E Health 2011 Dec;17(10):814-837. [doi:10.1089/tmj.2011.0075] [Medline: 21970573].
20. Amazon Mechanical Turk. URL: https://www.mturk.com/mturk/welcome [accessed 2014-08-25] [WebCite Cache ID 6S5lYMqkr].
21. Ilakkuvan V, Tacelosky M, Ivey KC, Pearson JL, Cantrell J, Vallone DM, et al. Cameras for Public Health Surveillance: A Methods Protocol for Crowdsourced Annotation of Point-of-Sale Photographs. JMIR Res Protoc 2014;3(2):e22 [FREE Full text] [doi:10.2196/resprot.3277] [Medline: 24717168].
22. Turner AM, Kirchhoff K, Capurro D. Using crowdsourcing technology for testing multilingual public health promotion materials. J Med Internet Res 2012;14(3):e79 [FREE Full text] [doi:10.2196/jmir.2063] [Medline: 22664384].
23. Henshaw EJ. Too sick, not sick enough? Effects of treatment type and timing on depression stigma. J Nerv Ment Dis 2014 Apr;202(4):292-299. [doi:10.1097/NMD.0000000000000121] [Medline: 24647215].
24. Halkjelsvik T. Do disgusting and fearful anti-smoking advertisements increase or decrease support for tobacco control policies? Int J Drug Policy 2014 Jul;25(4):744-747. [doi:10.1016/j.drugpo.2014.01.021] [Medline: 24630079].
25. Carter RR, DiFeo A, Bogie K, Zhang GQ, Sun J. Crowdsourcing awareness: Exploration of the ovarian cancer knowledge gap through Amazon Mechanical Turk. Plos One 2014;9(1):e85508 [FREE Full text] [doi:10.1371/journal.pone.0085508] [Medline: 24465580].
26. Swan M. Crowdsourced health research studies: An important emerging complement to clinical trials in the public health research ecosystem. J Med Internet Res 2012;14(2):e46 [FREE Full text] [doi:10.2196/jmir.1988] [Medline: 22397809].
27. Good BM, Loguercio S, Griffith OL, Nanis M, Wu C, Su AI. The Cure: Design and Evaluation of a Crowdsourcing Game for Gene Selection for Breast Cancer Survival Prediction. JMIR Serious Games 2014 Jul 29;2(2):e7. [doi:10.2196/games.3350].
28. Dasgupta N, Freifeld C, Brownstein JS, Menone CM, Surratt HL, Poppish L, et al. Crowdsourcing black market prices for prescription opioids. J Med Internet Res 2013;15(8):e178 [FREE Full text] [doi:10.2196/jmir.2810] [Medline: 23956042].
29. Martin D, Hanrahan BV, O'Neill J. Being a turker. In: Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. 2014 Presented at: 17th ACM conference on Computer supported cooperative work & social computing; February 15-19, 2014; Baltimore, Maryland, USA. [doi:10.1145/2531602.2531663].
30. Ipeirotis PG. CeDER Working Papers-10-01. 2010. Demographics of mechanical turk URL: http://hdl.handle.net/2451/29585 [accessed 2014-09-26] [WebCite Cache ID 6StCwI3A8].
31. Shotliff K, Duncan G. Diabetic retinopathy: summary of grading and management criteria. Pract Diab Int 2006 Nov;23(9):418-420. [doi:10.1002/pdi.1029].
32. Quenouille MH. Notes on bias in estimation. Biometrika 1956;43(3-4):353-360. [doi:10.1093/biomet/43.3-4.353].
33. Amazon Mechanical Turk Frequently Asked Questions. URL: https://requester.mturk.com/help/faq [accessed 2014-08-25] [WebCite Cache ID 6S5ljfanT].
34. Friedman DS, Ali F, Kourgialis N. Diabetic retinopathy in the developing world: how to approach identifying and treating underserved populations. Am J Ophthalmol 2011 Feb;151(2):192-4.e1. [doi:10.1016/j.ajo.2010.10.014] [Medline: 21251492].
35. Horton JJ, Chilton LB. The labor economics of paid crowdsourcing. In: Proceedings of the 11th ACM conference on Electronic commerce. 2010 Presented at: 11th ACM conference on Electronic commerce; June 7-11, 2010; Cambridge, Massachusetts, USA. [doi:10.1145/1807342.1807376].
36. Mason W, Watts DJ. Financial incentives and the "performance of crowds". In: Proceedings of the ACM SIGKDD Workshop on Human Computation. 2009 Presented at: 2009 ACM SIGKDD Workshop on Human Computation; June 28-July 1, 2009; Paris, France. [doi:10.1145/1600150.1600175].
37. Reed J, Raddick MJ, Lardner A. An exploratory factor analysis of motivations for participating in Zooniverse, a collection of virtual citizen science projects. In: 46th Hawaii International Conference on System Sciences (HICSS). 2013 Presented at: 46th Hawaii International Conference on System Sciences (HICSS); January 7-10, 2013; Wailea, Maui, HI. [doi:10.1109/HICSS.2013.85].