Strategies for improved interpretation of computer-aided detections for CT colonography utilizing distributed human intelligence

Medical Image Analysis

Volumn 16, Issue 6, 2012, Pages 1280-1292

  McKenna, Matthew T ^a   Wang, Shijun ^a   Nguyen, Tan B ^a   Burns, Joseph E ^a,b   Petrick, Nicholas ^c   Summers, Ronald M ^a  

^a NATIONAL INSTITUTES OF HEALTH   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c CENTER FOR DEVICES AND RADIOLOGICAL HEALTH   (United States)

Author keywords

Computed tomography colonography; Crowdsourcing; Distributed human intelligence; Observer performance study; Video analysis

Indexed keywords

COMPUTED TOMOGRAPHY COLONOGRAPHY; CROWDSOURCING; HUMAN INTELLIGENCE; OBSERVER PERFORMANCE STUDY; VIDEO ANALYSIS;

COMPUTER AIDED DESIGN; COMPUTERIZED TOMOGRAPHY; THREE DIMENSIONAL;

COMPUTER AIDED ANALYSIS;

ADULT; AGED; ARTICLE; CLINICAL ARTICLE; COMPUTED TOMOGRAPHIC COLONOGRAPHY; COMPUTER ASSISTED DIAGNOSIS; CONTROLLED STUDY; DIAGNOSTIC IMAGING; FALSE POSITIVE RESULT; FEMALE; HISTOPATHOLOGY; HUMAN; INTELLIGENCE; MALE; POLYP; PRIORITY JOURNAL; RADIOLOGIST; SENSITIVITY AND SPECIFICITY; SIGNAL NOISE RATIO; THREE DIMENSIONAL IMAGING;

AGED; ALGORITHMS; ARTIFICIAL INTELLIGENCE; COLONIC POLYPS; COLONOGRAPHY, COMPUTED TOMOGRAPHIC; COLORECTAL NEOPLASMS; FEMALE; HUMANS; INTESTINAL POLYPS; MALE; MIDDLE AGED; PATTERN RECOGNITION, AUTOMATED; RADIOGRAPHIC IMAGE ENHANCEMENT; RADIOGRAPHIC IMAGE INTERPRETATION, COMPUTER-ASSISTED; REPRODUCIBILITY OF RESULTS; SENSITIVITY AND SPECIFICITY;

EID: 84866128466     PISSN: 13618415     EISSN: 13618423     Source Type: Journal    
DOI: 10.1016/j.media.2012.04.007     Document Type: Article

References (37)

1. Burgess A.E. Comparison of receiver operating characteristic and forced choice observer performance measurement methods. Med. Phys. 1995, 22:643-655.
2. Cooper S., Khatib F., Treuille A., Barbero J., Lee J., Beenen M., Leaver-Fay A., Baker D., Popovic Z., Players F. Predicting protein structures with a multiplayer online game. Nature 2010, 466:756-760.
3. Cotton P.B., Durkalski V.L., Benoit P.C., Palesch Y.Y., Mauldin P.D., Hoffman B., Vining D.J., Small W.C., Affronti J., Rex D., Kopecky K.K., Ackerman S., Burdick J.S., Brewington C., Turner M.A., Zfass A., Wright A.R., Iyer R.B., Lynch P., Sivak M.V., Butler H. Computed tomographic colonography (virtual colonoscopy) - a multicenter comparison with standard colonoscopy for detection of colorectal neoplasia. JAMA-J. Am. Med. Assoc. 2004, 291:1713-1719.
4. Dachman A.H., Obuchowski N.A., Hoffmeister J.W., Hinshaw J.L., Frew M.I., Winter T.C., Van Uitert R.L., Periaswamy S., Summers R.M., Hillman B.J. Effect of computer-aided detection for CT colonography in a multireader, multicase trial. Radiology 2010, 256:827-835.
5. Dawid A.P., Skene A.M. Maximum likelihood estimation of observer error-rates using the em algorithm. J. R. Stat. Soc. Ser. C (Appl. Stat.) 1979, 28:20-28.
6. Dijkstra E.W. A note on two problems in connexion with graphs. Numer. Math. 1959, 1:269-271.
7. Gur D., Bandos A.I., Cohen C.S., Hakim C.M., Hardesty L.A., Ganott M.A., Perrin R.L., Poller W.R., Shah R., Sumkin J.H., Wallace L.P., Rockette H.E. The "Laboratory" effect: comparing radiologists' performance and variability during prospective clinical and laboratory mammography interpretations. Radiology 2008, 249:47-53.
8. Ipeirotis, P.G., Provost, F., Wang, J., 2010. Quality management on Amazon Mechanical Turk. In: Proceedings of the ACM SIGKDD Workshop on Human Computation. ACM, Washington DC.
9. Jemal A., Siegel R., Xu J., Ward E. Cancer statistics. CA: Cancer J. Clin. 2010, 60:277-300.
10. Johnson C.D. Accuracy of CT colonography for detection of large adenomas and cancers. N. Engl. J. Med. 2008, 359:1207-1217.
11. Johnson C.D., Toledano A.Y., Herman B.A., Dachman A.H., McFarland E.G., Barish M.A., Brink J.A., Ernst R.D., Fletcher J.G., Halvorsen R.A., Hara A.K., Hopper K.D., Koehler R.E., Lu D.S., Macari M., Maccarty R.L., Miller F.H., Morrin M., Paulson E.K., Yee J., Zalis M. Computerized tomographic colonography: performance evaluation in a retrospective multicenter setting. Gastroenterology 2003, 125:688-695.
12. Li J., Huang A., Yao J., Liu J.M., Van Uitert R.L., Petrick N., Summers R.M. Optimizing computer-aided colonic polyp detection for CT colonography by evolving the Pareto front. Med. Phys. 2009, 36:201-212.
13. Metz C.E., Herman B.A., Shen J.H. Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data. Stat. Med. 1998, 17:1033-1053.
14. Nappi J., Yoshida H. Virtual tagging for laxative-free CT colonography: Pilot evaluation. Med. Phys. 2009, 36:1830-1838.
15. Nguyen T., Wang S., Anugu V., Rose N., Burns J., McKenna M., Petrick N., Summers R.M. Distributed human intelligence for colonic polyp classification in computer-aided detection for CT colonography. Radiology 2012, 262:824-833.
16. Nodine C.F., Mello-Thoms C., Kundel H.L., Weinstein S.P. Time course of perception and decision making during mammographic interpretation. Am. J. Roentgenol. 2002, 179:917-923.
17. Obuchowski N.A., Meziane M., Dachman A.H., Lieber M.L., Mazzone P.J. What's the control in studies measuring the effect of computer-aided detection (CAD) on observer performance?. Acad. Radiol. 2010, 17:761-767.
18. Park S.H., Lee S.S., Kim J.K., Kim M.-J., Kim H.J., Kim S.Y., Kim M.-Y., Kim A.Y., Ha H.K. Volume rendering with color coding of tagged stool during endoluminal fly-through CT colonography: effect on reading efficiency. Radiology 2008, 248:1018-1027.
19. Petrick N., Haider M., Summers R.M., Yeshwant S.C., Brown L., Iuliano E.M., Louie A., Choi J.R., Pickhardt P.J. CT colonography and computer-aided detection as a second reader: observer performance study. Radiology 2008, 246:148-156.
20. Pickhardt P.J., Choi J.H. Electronic cleansing and stool tagging in CT colonography: advantages and pitfalls with primary three-dimensional evaluation. AJR: Am. J. Roentgenol. 2003, 181:799-805.
21. Pickhardt P.J., Choi J.R., Hwang I., Butler J.A., Puckett M.L., Hildebrandt H.A., Wong R.K., Nugent P.A., Mysliwiec P.A., Schindler W.R. Computed tomographic virtual colonoscopy to screen for colorectal neoplasia in asymptomatic adults. N. Engl. J. Med. 2003, 349:2191-2200.
22. Raykar V.C., Yu S., Zhao L.H., Valadez G.H., Florin C., Bogoni L., Moy L. Learning from crowds. J. Mach. Learn. Res. 2010, 11:1297-1322.
23. Rockey D.C., Paulson E., Niedzwiecki D., Davis W., Bosworth H.B., Sanders L., Yee J., Henderson J., Hatten P., Burdick S., Sanyal A., Rubin D.T., Sterling M., Akerkar G., Bhutani M.S., Binmoeller K., Garvie J., Bini E.J., McQuaid K., Foster W.L., Thompson W.M., Dachman A., Halvorsen R. Analysis of air contrast barium enema, computed tomographic colonography, and colonoscopy: prospective comparison. Lancet 2005, 365:305-311.
24. Saunders R.S., Samei E. Improving mammographic decision accuracy by incorporating observer ratings with interpretation time. Brit. J. Radiol. 2006, Spec No. 79(2):S117-S122.
25. Schroeder W., Martin K., Lorensen B. The Visualization Toolkit 2003, Kitware, Inc. third ed.
26. Smith R.A., Cokkinides V., Brooks D., Saslow D., Brawley O.W. Cancer Screening in the United States, 2010: a review of current American Cancer Society guidelines and issues in cancer screening. CA: Cancer J. Clin. 2010, 60:99-119.
27. Snow R., O'Connor B., Jurafsky D., Ng A.Y. Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks. Proceedings of the Conference on Empirical Methods in Natural Language Processing 2008, Association for Computational Linguistics, Honolulu, Hawaii.
28. Summers R.M., Yao J., Pickhardt P.J., Franaszek M., Bitter I., Brickman D., Krishna V., Choi J.R. Computed tomographic virtual colonoscopy computer-aided polyp detection in a screening population. Gastroenterology 2005, 129:1832-1844.
29. Suzuki K., Zhang J., Xu J.W. Massive-training artificial neural network coupled with Laplacian-Eigenfunction-based dimensionality reduction for computer-aided detection of polyps in CT colonography. IEEE Trans. Med. Imag. 2010, 29:1907-1917.
30. Taylor S.A., Robinson C., Boone D., Honeyfield L., Halligan S. Polyp characteristics correctly annotated by computer-aided detection software but ignored by reporting radiologists during CT colonography. Radiology 2009, 253:715-723.
31. von Ahn, L., Dabbish, L., 2004. Labeling images with a computer game. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Vienna, Austria.
32. von Ahn L., Maurer B., McMillen C., Abraham D., Blum M. ReCAPTCHA: human-based character recognition via web security measures. Science 2008, 321:1465.
33. Wang, S., Anugu, V., Nguyen, T., Rose, N., Burns, J., McKenna, M., Petrick, N., Summers, R.M., 2011. Fusion of machine intelligence and human intelligence for colonic polyp detection in CT colonography. In: IEEE International Symposium on Biomedical Imaging: From Nano to Macro, pp. 160-164.
34. Wang S., Yao J., Petrick N., Summers R.M. Combining statistical and geometric features for colonic polyp detection in CTC based on multiple kernel learning. Int. J. Comput. Intell. Appl. 2010, 9:1-15.
35. Whitehill J., Ruvolo P., Wu T.-f., Bergsma J., Movellan J. Whose vote should count more: optimal integration of labels from labelers of unknown expertise. Adv. Neur. Inform. Process. Syst. 2009, 22.
36. Yao J., Miller M., Franaszek M., Summers R.M. Colonic polyp segmentation in CT colonography based on fuzzy clustering and deformable models. IEEE Trans. Med. Imag. 2004, 23:1344-1352.
37. Zhu H.B., Liang Z.R., Pickhardt P.J., Barish M.A., You J.S., Fan Y., Lu H.B., Posniak E.J., Richards R.J., Cohen H.L. Increasing computer-aided detection specificity by projection features for CT colonography. Med. Phys. 2010, 37:1468-1481.