Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?

IEEE Transactions on Medical Imaging

Volumn 35, Issue 5, 2016, Pages 1299-1312

  Tajbakhsh, Nima ^a   Shin, Jae Y ^a   Gurudu, Suryakanth R ^b   Hurst, R Todd ^b   Kendall, Christopher B ^b   Gotway, Michael B ^b   Liang, Jianming ^a  

^a ARIZONA STATE UNIVERSITY   (United States)

^b MAYO CLINIC   (United States)

Author keywords

Carotid intima media thickness; computer aided detection; convolutional neural networks; deep learning; fine tuning; medical image analysis; polyp detection; pulmonary embolism detection; video quality assessment

Indexed keywords

COMPUTER AIDED ANALYSIS; COMPUTER AIDED INSTRUCTION; CONVOLUTION; IMAGE ANALYSIS; IMAGE SEGMENTATION; KNOWLEDGE MANAGEMENT; NEURAL NETWORKS; QUALITY CONTROL;

CAROTID INTIMA-MEDIA THICKNESS; COMPUTER AIDED DETECTION; CONVOLUTIONAL NEURAL NETWORK; DEEP LEARNING; FINE TUNING; POLYP DETECTION; PULMONARY EMBOLISM; VIDEO QUALITY ASSESSMENT;

MEDICAL IMAGING;

ARTERIAL WALL THICKNESS; ARTICLE; ARTIFICIAL NEURAL NETWORK; COLON POLYP; COLONOSCOPY; COMPUTER ASSISTED TOMOGRAPHY; CONVOLUTIONAL NEURAL NETWORK; CONVOLUTIONAL NEURAL NETWORK TRAINED FROM SCRATCH; DIAGNOSTIC IMAGING; ECHOGRAPHY; HUMAN; IMAGE ANALYSIS; IMAGE QUALITY; LUNG ANGIOGRAPHY; LUNG EMBOLISM; PRE TRAINED CONVOLUTIONAL NEURAL NETWORK FINED TUNED; QUALITY CONTROL; COMPUTED TOMOGRAPHIC ANGIOGRAPHY; COMPUTER ASSISTED DIAGNOSIS; MACHINE LEARNING; RECEIVER OPERATING CHARACTERISTIC;

COLONIC POLYPS; COLONOSCOPY; COMPUTED TOMOGRAPHY ANGIOGRAPHY; DIAGNOSTIC IMAGING; HUMANS; IMAGE INTERPRETATION, COMPUTER-ASSISTED; MACHINE LEARNING; NEURAL NETWORKS (COMPUTER); PULMONARY EMBOLISM; ROC CURVE;

EID: 84968649810     PISSN: 02780062     EISSN: 1558254X     Source Type: Journal    
DOI: 10.1109/TMI.2016.2535302     Document Type: Article

References (76)

1. K. Fukushima, "Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position," Biol. Cybern., vol. 36, no. 4, pp. 193-202, 1980.
2. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proc. IEEE, vol. 86, no. 11, pp. 2278-2324, Nov. 1998.
3. Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," Nature, vol. 521, no. 7553, pp. 436-444, 2015.
4. MIT Technol. Rev. "10 breakthrough technologies," [Online]. Available: http://www.technologyreview.com/featuredstory/513696/deep-learning/
5. C. Szegedy et al., Going deeper with convolutions ArXiv, 2014 [Online]. Available: arXiv:1409.4842, to be published
6. K. Simonyan and A. Zisserman, Very deep convolutional networks for large-scale image recognition ArXiv, 2014 [Online]. Available: arXiv:1409.1556, to be published
7. M. D. Zeiler and R. Fergus, "Visualizing and understanding convolutional networks," in Comput. Visi. ECCV, 2014, pp. 818-833.
8. D. Eigen, J. Rolfe, R. Fergus, and Y. LeCun, Understanding deep architectures using a recursive convolutional network ArXiv, 2013 [Online]. Available: arXiv:1312.1847, to be published
9. D. Erhan, P.-A. Manzagol, Y. Bengio, S. Bengio, and P. Vincent, "The difficulty of training deep architectures and the effect of unsupervised pre-training," in Proc. Int. Conf. Artif. Intell. Stat., 2009, pp. 153-160.
10. A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson, "CNN features off-the-shelf: An astounding baseline for recognition," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops, 2014, pp. 512-519.
11. H. Azizpour, A. S. Razavian, J. Sullivan, A. Maki, and S. Carlsson, From generic to specific deep representations for visual recognition ArXiv, 2014 [Online]. Available: arXiv:1406.5774, to be published
12. O. A. Penatti, K. Nogueira, and J. A. dos Santos, "Do deep features generalize from everyday objects to remote sensing and aerial scenes domains?," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops, Boston, MA, pp. 44-51.
13. W. Zhang et al., "Computerized detection of clustered microcalcifications in digital mammograms using a shift-invariant artificial neural network," Med. Phys., vol. 21, no. 4, pp. 517-524, 1994.
14. H.-P. Chan, S.-C. B. Lo, B. Sahiner, K. L. Lam, and M. A. Helvie, "Computer-aided detection of mammographic microcalcifications: Pattern recognition with an artificial neural network," Med. Phys., vol. 22, no. 10, pp. 1555-1567, 1995.
15. S.-C. B. Lo et al., "Artificial convolution neural network techniques and applications for lung nodule detection," IEEE Trans. Med. Imag., vol. 14, no. 4, pp. 711-718, Dec. 1995.
16. N. Tajbakhsh, S. R. Gurudu, and J. Liang, "A comprehensive computer-aided polyp detection system for colonoscopy videos," in Information Processing in Medical Imaging. New York: Springer, 2015, pp. 327-338.
17. N. Tajbakhsh, S. R. Gurudu, and J. Liang, "Automatic polyp detection in colonoscopy videos using an ensemble of convolutional neural networks," in Proc. IEEE 12th Int. Symp. o Biomed. Imag., 2015, pp. 79-83.
18. N. Tajbakhsh and J. Liang, "Computer-aided pulmonary embolism detection using a novel vessel-aligned multi-planar image representation and convolutional neural networks," in Proc. MICCAI, 2015.
19. D. C. Cireşan, A. Giusti, L. M. Gambardella, and J. Schmidhuber, "Mitosis detection in breast cancer histology images with deep neural networks," in Proc. MICCAI, 2013, pp. 411-418.
20. H. Roth et al., "A new 2.5D representation for lymph node detection using random sets of deep convolutional neural network observations," in Proc. MICCAI, P. Goll, N. Hata, C. Barillot, J. Hornegger, and R. Howe, Eds., 2014, vol. 8673, LNCS, pp. 520-527.
21. Y. Zheng, D. Liu, B. Georgescu, H. Nguyen, and D. Comaniciu, "3d deep learning for efficient and robust landmark detection in volumetric data," in Proc. MICCAI, 2015, pp. 565-572.
22. J. Y. Shin, N. Tajbakhsh, R. T. Hurst, C. B. Kendall, and J. Liang, "Automating carotid intima-media thickness video interpretation with convolutional neural networks," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Las Vegas, NV, 2016.
23. H. R. Roth, A. Farag, L. Lu, E. B. Turkbey, and R. M. Summers, "Deep convolutional networks for pancreas segmentation in CT imaging," in Proc. SPIE Med. Imag., 2015, pp. 94 131G-94 131G.
24. M. Havaei et al., Brain tumor segmentation with deep neural networks ArXiv, 2015 [Online]. Available: arXiv:1505.03540, to be published
25. W. Zhang et al., "Deep convolutional neural networks for multi-modality isointense infant brain image segmentation," NeuroImage, vol. 108, pp. 214-224, 2015.
26. D. Ciresan, A. Giusti, L. M. Gambardella, and J. Schmidhuber, "Deep neural networks segment neuronal membranes in electron microscopy images," in Advances in Neural Information Processing Systems, F. Pereira, C. Burges, L. Bottou, and K. Weinberger, Eds. Red Hook, NY: Curran, 2012, vol. 25, pp. 2843-2851.
27. A. Prasoon, K. Petersen, C. Igel, F. Lauze, E. Dam, and M. Nielsen, "Deep feature learning for knee cartilage segmentation using a tri-planar convolutional neural network," in Proc. MICCAI, 2013, pp. 246-253.
28. Y. Bar, I. Diamant, L. Wolf, and H. Greenspan, "Deep learning with non-medical training used for chest pathology identification," in Proc. SPIE Med. Imag., 2015, pp. 94 140V-94 140V.
29. B. van Ginneken, A. A. Setio, C. Jacobs, and F. Ciompi, "Off-the-shelf convolutional neural network features for pulmonary nodule detection in computed tomography scans," in Proc. IEEE 12th Int. Symp. Biomed. Imag., Apr. 2015, pp. 286-289.
30. J. Arevalo, F. Gonzalez, R. Ramos-Pollan, J. Oliveira, and M. G. Lopez, "Convolutional neural networks for mammography mass lesion classification," in Proc. 37th Annu. Int. Conf. IEEE EMBC, Aug. 2015, pp. 797-800.
31. T. Schlegl, J. Ofner, and G. Langs, "Unsupervised pre-training across image domains improves lung tissue classification," in Medical Computer Vision: Algorithms for Big Data. New York: Springer, 2014, pp. 82-93.
32. H. Chen et al., "Standard plane localization in fetal ultrasound via domain transferred deep neural networks," IEEE J. Biomed. Health Informat., vol. 19, no. 5, pp. 1627-1636, Sep. 2015.
33. G. Carneiro, J. Nascimento, and A. Bradley, "Unregistered multiview mammogram analysis with pre-trained deep learning models," in Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015, N. Navab, J. Hornegger, W. M. Wells, and A. F. Frangi, Eds., 2015, vol. 9351, LNCS, pp. 652-660.
34. H.-C. Shin et al., "Interleaved text/image deep mining on a very large-scale radiology database," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2015, pp. 1090-1099.
35. M. Gao et al., "Holistic classification of CT attenuation patterns for interstitial lung diseases via deep convolutional neural networks," in 1st Workshop Deep Learn. Med. Image Anal., Int. Conf. Med. Image Comput. Comput. Assist. Intervent., Munich, Germany, 2015 [Online]. Available: www.research.rutgers.edu/~minggao/files/MingchenGao-MICCAIworkshop2015.pdf
36. J. Margeta, A. Criminisi, R. C. Lozoya, D. C. Lee, and N. Ayache, "Fine-tuned convolutional neural nets for cardiac MRI acquisition plane recognition," Computer Methods Biomechan. Biomed. Eng., Imag. Visualizat., pp. 1-11, 2015.
37. D. H. Hubel and T. N. Wiesel, "Receptive fields of single neurones in the cat's striate cortex," J. Physiol., vol. 148, no. 3, pp. 574-591, 1959.
38. D. C. Edwards, M. A. Kupinski, C. E. Metz, and R. M. Nishikawa, "Maximum likelihood fitting of FROC curves under an initial-detection-and-candidate-analysis model," Med. Phys., vol. 29, no. 12, pp. 2861-2870, 2002.
39. Y. Jia et al., Caffe: Convolutional architecture for fast feature embedding ArXiv, 2014 [Online]. Available: arXiv:1408.5093, to be published
40. X. Glorot and Y. Bengio, "Understanding the difficulty of training deep feedforward neural networks," in Proc. Int. Conf. Artif. Intell. Stat., 2010, pp. 249-256.
41. K. He, X. Zhang, S. Ren, and J. Sun, Delving deep into rectifiers: Surpassing human-level performance on imagenet classification ArXiv, 2015 [Online]. Available: arXiv:1502.01852, to be published
42. N. Tajbakhsh, S. Gurudu, and J. Liang, "Automated polyp detection in colonoscopy videos using shape and context information," IEEE Trans. Med. Imag., vol. 35, no. 2, pp. 630-644, Feb. 2016.
43. A. Pabby et al., "Analysis of colorectal cancer occurrence during surveillance colonoscopy in the dietary polyp prevention trial," Gastrointest Endosc., vol. 61, no. 3, pp. 385-91, 2005.
44. J. van Rijn et al., "Polyp miss rate determined by tandem colonoscopy: A systematic review," Am. Jo. Gastroenterol., vol. 101, no. 2, pp. 343-350, 2006.
45. D. H. Kim et al., "CT colonography versus colonoscopy for the detection of advanced neoplasia," N. Eng. J. Med., vol. 357, no. 14, pp. 1403-12, 2007.
46. D. Heresbach et al., "Miss rate for colorectal neoplastic polyps: A prospective multicenter study of back-to-back video colonoscopies," Endoscopy, vol. 40, no. 4, pp. 284-290, 2008.
47. A. Leufkens, M. van Oijen, F. Vleggaar, and P. Siersema, "Factors influencing the miss rate of polyps in a back-to-back colonoscopy study," Endoscopy, vol. 44, no. 05, pp. 470-475, 2012.
48. L. Rabeneck, H. El-Serag, J. Davila, and R. Sandler, "Outcomes of colorectal cancer in the united states: No change in survival (1986-1997)," Am. J. Gastroenterol., vol. 98, no. 2, p. 471, 2003.
49. S. A. Karkanis, D. K. Iakovidis, D. E. Maroulis, D. A. Karras, and M. Tzivras, "Computer-aided tumor detection in endoscopic video using color wavelet features," IEEE Trans. Inf. Technol. Biomed., vol. 7, no. 3, pp. 141-152, Sep. 2003.
50. D. K. Iakovidis, D. E. Maroulis, S. A. Karkanis, and A. Brokos, "A comparative study of texture features for the discrimination of gastric polyps in endoscopic video," in Proc. 18th IEEE Symp. Comput.-Based Med. Syst., 2005, pp. 575-580.
51. L. A. Alexandre, N. Nobre, and J. Casteleiro, "Color and position versus texture features for endoscopic polyp detection," in Proc. Int. Conf. BioMedical Eng. Informat., 2008, vol. 2, pp. 38-42.
52. S. Hwang, J. Oh, W. Tavanapong, J. Wong, and P. de Groen, "Polyp detection in colonoscopy video using elliptical shape feature," in IEEE Int. Conf. Image Process., 2007, vol. 2, pp. II-465-II-468.
53. J. Bernal, J. Sánchez, and F. Vilariño, "Towards automatic polyp detection with a polyp appearance model," Pattern Recognit., vol. 45, no. 9, pp. 3166-3182, 2012.
54. J. Bernal, J. Sánchez, and F. Vilarino, "Impact of image preprocessing methods on polyp localization in colonoscopy frames," in Proc. 35th Annu. Int. Conf. IEEE EMBC, 2013, pp. 7350-7354.
55. Y. Wang, W. Tavanapong, J. Wong, J. Oh, and P. de Groen, "Part-based multi-derivative edge cross-section profiles for polyp detection in colonoscopy," IEEE J. Biomed. Health Informat., vol. 18, no. 4, pp. 1379-1389, Jul. 2014.
56. S. Y. Park, D. Sargent, I. Spofford, K. Vosburgh, and Y. A-Rahim, "A colon video analysis framework for polyp detection," IEEE Trans. Biomed. Eng., vol. 59, no. 5, pp. 1408-1418, May 2012.
57. N. Tajbakhsh, S. Gurudu, and J. Liang, "A classification-enhanced vote accumulation scheme for detecting colonic polyps," in Abdominal Imaging. Computation and Clinical Applications. New York: Springer, 2013, vol. 8198, LNCS, pp. 53-62.
58. N. Tajbakhsh, C. Chi, S. R. Gurudu, and J. Liang, "Automatic polyp detection from learned boundaries," in Proc. IEEE 10th Int. Symp. Biomed. Imag., 2014, pp. 97-100.
59. N. Tajbakhsh, S. R. Gurudu, and J. Liang, "Automatic polyp detection using global geometric constraints and local intensity variation patterns," in Proc. MICCAI, 2014, pp. 179-187.
60. J. Liang and J. Bi, "Computer aided detection of pulmonary embolism with tobogganing and multiple instance classification in CT pulmonary angiography," in Information Processing in Medical Imaging. New York: Springer, 2007, pp. 630-641.
61. K. K. Calder, M. Herbert, and S. O. Henderson, "The mortality of untreated pulmonary embolism in emergency department patients," Ann. Emerg. Med., vol. 45, no. 3, pp. 302-310, 2005.
62. G. Sadigh, A. M. Kelly, and P. Cronin, "Challenges, controversies, and hot topics in pulmonary embolism imaging," Am. J. Roentgenol., vol. 196, no. 3, 2011.
63. J. Fairfield, "Toboggan contrast enhancement for contrast segmentation," in Proc. 10th Int. Conf. Pattern Recognit., 1990, vol. 1, pp. 712-716.
64. R. M. Haralick, K. Shanmugam, and I. H. Dinstein, "Textural features for image classification," IEEE Trans. Syst., Man Cybern., vol. 3, no. 6, pp. 610-621, Nov. 1973.
65. N. Tajbakhsh, "Automatic assessment of image informativeness in colonoscopy," in Abdominal Imaging. Computational and Clinical Applications. New York: Springer, 2014, pp. 151-158.
66. M. Arnold, A. Ghosh, G. Lacey, S. Patchett, and H. Mulcahy, "Indistinct frame detection in colonoscopy videos," in Proc. 13th Int. Mach. Vis. Image Process. Conf., 2009, pp. 47-52.
67. J. Oh, S. Hwang, J. Lee, W. Tavanapong, J. Wong, and P. C. de Groen, "Informative frame classification for endoscopy video," Med. Image Anal., vol. 11, no. 2, pp. 110-127, 2007.
68. R.-M. Menchón-Lara and J.-L. Sancho-Gómez, "Fully automatic segmentation of ultrasound common carotid artery images based on machine learning," Neurocomputing, vol. 151, pp. 161-167, 2015.
69. R.-M. Menchón-Lara, M.-C. Bastida-Jumilla, A. González-López, and J. L. Sancho-Gómez, "Automatic evaluation of carotid intima-media thickness in ultrasounds using machine learning," in Natural and Artificial Computation in Engineering and Medical Applications. New York: Springer, 2013, pp. 241-249.
70. S. Petroudi, C. Loizou, M. Pantziaris, and C. Pattichis, "Segmentation of the common carotid intima-media complex in ultrasound images using active contours," IEEE Trans. Biomed. Eng., vol. 59, no. 11, pp. 3060-3069, Nov. 2012.
71. X. Xu, Y. Zhou, X. Cheng, E. Song, and G. Li, "Ultrasound intimamedia segmentation using Hough transform and dual snake model," Comput. Med. Imag. Graphics, vol. 36, no. 3, pp. 248-258, 2012.
72. J. Liang, T. McInerney, and D. Terzopoulos, "United snakes," Med. Image Anal., vol. 10, no. 2, pp. 215-233, 2006.
73. H. Sharma et al., "ECG-based frame selection and curvature-based ROI detection for measuring carotid intima-media thickness," in Proc. SPIE Med. Imag., 2014, pp. 904 016-904 016.
74. H. Lee, R. Grosse, R. Ranganath, and A. Y. Ng, "Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations," in Proc. 26th Annu. Int. Conf. Mach. Learn., 2009, pp. 609-616.
75. O. Abdel-Hamid et al., "Convolutional neural networks for speech recognition," IEEE/ACM Trans. Audio, Speech, Language Process., vol. 22, no. 10, pp. 1533-1545, Oct. 2014.
76. D. Wulsin, J. Gupta, R. Mani, J. Blanco, and B. Litt, "Modeling electroencephalography waveforms with semi-supervised deep belief nets: Fast classification and anomaly measurement," J. Neural Eng., vol.8, no. 3, p. 036015, 2011.