Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation

Medical Image Analysis

Volumn 36, Issue , 2017, Pages 61-78

  Kamnitsas, Konstantinos ^a   Ledig, Christian ^a   Newcombe, Virginia F J ^b,c   Simpson, Joanna P ^b   Kane, Andrew D ^b   Menon, David K ^b,c   Rueckert, Daniel ^a   Glocker, Ben ^a  

^a IMPERIAL COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY OF CAMBRIDGE   (United Kingdom)

^c UNIVERSITY OF CAMBRIDGE   (United Kingdom)

Author keywords

3D convolutional neural network; Brain lesions; Deep learning; Fully connected CRF; Segmentation

Indexed keywords

BENCHMARKING; CLINICAL RESEARCH; CONVOLUTION; DEEP LEARNING; DEEP NEURAL NETWORKS; HOSPITAL DATA PROCESSING; IMAGE SEGMENTATION; MEDICAL IMAGING; NETWORK ARCHITECTURE; RANDOM PROCESSES;

BRAIN LESIONS; COMPUTATIONAL BURDEN; COMPUTATIONALLY EFFICIENT; CONDITIONAL RANDOM FIELD; CONTEXTUAL INFORMATION; FULLY CONNECTED CRF; LESION SEGMENTATIONS; TRAUMATIC BRAIN INJURIES;

CONVOLUTIONAL NEURAL NETWORKS;

ARTICLE; BRAIN DAMAGE; BRAIN ISCHEMIA; BRAIN MAPPING; BRAIN TUMOR; HUMAN; IMAGE PROCESSING; IMAGE SEGMENTATION; NERVE CELL NETWORK; NEUROIMAGING; NUCLEAR MAGNETIC RESONANCE IMAGING; PRIORITY JOURNAL; RANDOM FOREST; RECEPTIVE FIELD; TRAUMATIC BRAIN INJURY; ARTIFICIAL NEURAL NETWORK; BRAIN; DIAGNOSTIC IMAGING; PATHOLOGY; REPRODUCIBILITY; SENSITIVITY AND SPECIFICITY;

BRAIN; BRAIN INJURIES, TRAUMATIC; BRAIN ISCHEMIA; BRAIN NEOPLASMS; HUMANS; NEURAL NETWORKS (COMPUTER); REPRODUCIBILITY OF RESULTS; SENSITIVITY AND SPECIFICITY;

EID: 84995784237     PISSN: 13618415     EISSN: 13618423     Source Type: Journal    
DOI: 10.1016/j.media.2016.10.004     Document Type: Article

References (73)

1. Bakas, S., Zeng, K., Sotiras, A., Rathore, S., Akbari, H., Gaonkar, B., Rozycki, M., Pati, S., Davatzikos, C., GLISTRboost: Combining multimodal MRI segmentation, registration, and biophysical tumour growth modelling with gradient boosting machines for glioma segmentation. International Workshop on Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, 2015, Springer, 144–155.
2. Bastien, F., Lamblin, P., Pascanu, R., Bergstra, J., Goodfellow, I.J., Bergeron, A., Bouchard, N., Bengio, Y., Theano: new features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop, 2012.
3. Brebisson, A., Montana, G., Deep neural networks for anatomical brain segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2015, 20–28.
4. Brosch, T., Yoo, Y., Tang, L.Y., Li, D.K., Traboulsee, A., Tam, R., Deep convolutional encoder networks for multiple sclerosis lesion segmentation. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015, 2015, Springer, 3–11.
5. Cardoso, M.J., Sudre, C.H., Modat, M., Ourselin, S., Template-based multimodal joint generative model of brain data. Information Processing in Medical Imaging, 2015, Springer, 17–29.
6. Carey, L.M., Seitz, R.J., Parsons, M., Levi, C., Farquharson, S., Tournier, J.-D., Palmer, S., Connelly, A., Beyond the lesion: neuroimaging foundations for post-stroke recovery. Future Neurol 8:5 (2013), 507–527.
7. Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A. L., 2014. Semantic image segmentation with deep convolutional nets and fully connected CRFs. arXiv preprint arXiv: 1412.7062.
8. Choromanska, A., Henaff, M., Mathieu, M., Arous, G.B., LeCun, Y., The loss surfaces of multilayer networks. AISTATS, 2015.
9. Ciresan, D., Giusti, A., Gambardella, L.M., Schmidhuber, J., Deep neural networks segment neuronal membranes in electron microscopy images. Advances in neural information processing systems, 2012, 2843–2851.
10. Cireşan, D.C., Giusti, A., Gambardella, L.M., Schmidhuber, J., Mitosis detection in breast cancer histology images with deep neural networks. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2013, 2013, Springer, 411–418.
11. Ding, K., de la Plata, C.M., Wang, J.Y., Mumphrey, M., Moore, C., Harper, C., Madden, C.J., McColl, R., Whittemore, A., Devous, M.D., et al. Cerebral atrophy after traumatic white matter injury: correlation with acute neuroimaging and outcome. J. Neurotrauma 25:12 (2008), 1433–1440.
12. Doyle, S., Vasseur, F., Dojat, M., Forbes, F., Fully automatic brain tumour segmentation from multiple MR sequences using hidden markov fields and variational EM. Procs. NCI-MICCAI BRATS, 2013, 18–22.
13. Erihov, M., Alpert, S., Kisilev, P., Hashoul, S., A cross saliency approach to asymmetry-based tumour detection. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015, 2015, Springer, 636–643.
14. Feng, C., Zhao, D., Huang, M., Segmentation of ischemic stroke lesions in multi-spectral MR images using weighting suppressed FCM and three phase level set. International Workshop on Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, 2015, Springer, 233–245.
15. Gal, Y., Ghahramani, Z., 2015. Dropout as a bayesian approximation: representing model uncertainty in deep learning. arXiv preprint arXiv: 1506.02142.
16. Geremia, E., Menze, B.H., Clatz, O., Konukoglu, E., Criminisi, A., Ayache, N., Spatial decision forests for MS lesion segmentation in multi-channel MR images. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2010, 2010, Springer, 111–118.
17. Glorot, X., Bengio, Y., Understanding the difficulty of training deep feedforward neural networks. International conference on artificial intelligence and statistics, 2010, 249–256.
18. Gooya, A., Pohl, K.M., Bilello, M., Biros, G., Davatzikos, C., Joint segmentation and deformable registration of brain scans guided by a tumour growth model. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2011, 2011, Springer, 532–540.
19. Grabner, G., Janke, A.L., Budge, M.M., Smith, D., Pruessner, J., Collins, D.L., Symmetric atlasing and model based segmentation: an application to the hippocampus in older adults. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2006, 2006, Springer, 58–66.
20. Halme, H.-L., Korvenoja, A., Salli, E., ISLES (SISS) challenge 2015: segmentation of stroke lesions using spatial normalization, Random Forest classification and contextual clustering. International Workshop on Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, 2015, Springer, 211–221.
21. Havaei, M., Davy, A., Warde-Farley, D., Biard, A., Courville, A., Bengio, Y., Pal, C., Jodoin, P.-M., Larochelle, H., 2015. Brain tumour segmentation with deep neural networks. arXiv preprint arXiv: 1505.03540.
22. He, K., Zhang, X., Ren, S., Sun, J., Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. Proceedings of the IEEE International Conference on Computer Vision, 2015, 1026–1034.
23. Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R. R., 2012. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv: 1207.0580.
24. Iglesias, J.E., Liu, C.-Y., Thompson, P.M., Tu, Z., Robust brain extraction across datasets and comparison with publicly available methods. Med. Imaging IEEE Trans. 30:9 (2011), 1617–1634.
25. Ikram, M.A., Vrooman, H.A., Vernooij, M.W., den Heijer, T., Hofman, A., Niessen, W.J., van der Lugt, A., Koudstaal, P.J., Breteler, M.M., Brain tissue volumes in relation to cognitive function and risk of dementia. Neurobiol. Aging 31:3 (2010), 378–386.
26. Ioffe, S., Szegedy, C., 2015. Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 1502.03167.
27. Irimia, A., Wang, B., Aylward, S.R., Prastawa, M.W., Pace, D.F., Gerig, G., Hovda, D.A., Kikinis, R., Vespa, P.M., Van Horn, J.D., Neuroimaging of structural pathology and connectomics in traumatic brain injury: toward personalized outcome prediction. NeuroImage: Clinical 1:1 (2012), 1–17.
28. Jarrett, K., Kavukcuoglu, K., Ranzato, M., LeCun, Y., What is the best multi-stage architecture for object recognition?. Computer Vision, 2009 IEEE 12th International Conference on, 2009, IEEE, 2146–2153.
29. Kamnitsas, K., Chen, L., Ledig, C., Rueckert, D., Glocker, B., 2015. Multi-scale 3D convolutional neural networks for lesion segmentation in brain MRI. in proc of ISLES-MICCAI.
30. Kappos, L., Freedman, M.S., Polman, C.H., Edan, G., Hartung, H.-P., Miller, D.H., Montalbán, X., Barkhof, F., Radü, E.-W., Bauer, L., et al. Effect of early versus delayed interferon beta-1b treatment on disability after a first clinical event suggestive of multiple sclerosis: a 3-year follow-up analysis of the BENEFIT study. Lancet 370:9585 (2007), 389–397.
31. Krähenbühl, P., Koltun, V., Efficient inference in fully connected CRFs with gaussian edge potentials. Adv. Neural Inf. Process. Syst 24 (2011), 109–117.
32. Krizhevsky, A., Sutskever, I., Hinton, G.E., ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 2012, 1097–1105.
33. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., Gradient-based learning applied to document recognition. Proc. IEEE 86:11 (1998), 2278–2324.
34. Ledig, C., Heckemann, R.A., Hammers, A., Lopez, J.C., Newcombe, V.F., Makropoulos, A., Lötjönen, J., Menon, D.K., Rueckert, D., Robust whole-brain segmentation: application to traumatic brain injury. Med. Image Anal. 21:1 (2015), 40–58.
35. Lehtonen, S., Stringer, A.Y., Millis, S., Boake, C., Englander, J., Hart, T., High, W., Macciocchi, S., Meythaler, J., Novack, T., et al. Neuropsychological outcome and community re-integration following traumatic brain injury: the impact of frontal and non-frontal lesions. Brain Inj. 19:4 (2005), 239–256.
36. Li, R., Zhang, W., Suk, H.-I., Wang, L., Li, J., Shen, D., Ji, S., Deep learning based imaging data completion for improved brain disease diagnosis. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2014, 2014, Springer, 305–312.
37. Liu, X., Niethammer, M., Kwitt, R., McCormick, M., Aylward, S., Low-rank to the rescue – atlas-based analyses in the presence of pathologies. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2014, 2014, Springer, 97–104.
38. Long, J., Shelhamer, E., Darrell, T., Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, 3431–3440.
39. Lyksborg, M., Puonti, O., Agn, M., Larsen, R., An ensemble of 2D convolutional neural networks for tumor segmentation. Image Analysis, 2015, Springer, 201–211.
40. Maas, A.I., Menon, D.K., Steyerberg, E.W., Citerio, G., Lecky, F., Manley, G.T., Hill, S., Legrand, V., Sorgner, A., Participants, C.-T., Investigators, et al. Collaborative european neurotrauma effectiveness research in traumatic brain injury (CENTER-TBI): a prospective longitudinal observational study. Neurosurgery 76:1 (2015), 67–80.
41. Maier, O., Menze, B.H., et al. ISLES 2015 – A public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI. Med. Image Anal. 35 (2017), 250–269.
42. Menze, B.H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., et al. The multimodal brain tumor image segmentation benchmark (BRATS). Med. Imaging IEEE Trans. 34:10 (2015), 1993–2024.
43. Mitra, J., Bourgeat, P., Fripp, J., Ghose, S., Rose, S., Salvado, O., Connelly, A., Campbell, B., Palmer, S., Sharma, G., et al. Lesion segmentation from multimodal MRI using random forest following ischemic stroke. Neuroimage 98 (2014), 324–335.
44. Moen, K.G., Skandsen, T., Folvik, M., Brezova, V., Kvistad, K.A., Rydland, J., Manley, G.T., Vik, A., A longitudinal MRI study of traumatic axonal injury in patients with moderate and severe traumatic brain injury. J. Neurol. Neurosurg. Psychiatry 83 (2012), 1193–1200 jnnp–2012.
45. Noh, H., Hong, S., Han, B., Learning deconvolution network for semantic segmentation. Proceedings of the IEEE International Conference on Computer Vision, 2015, 1520–1528.
46. Parisot, S., Duffau, H., Chemouny, S., Paragios, N., Joint tumor segmentation and dense deformable registration of brain MR images. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2012, 2012, Springer, 651–658.
47. Pereira, S., Pinto, A., Alves, V., Silva, C.A., Deep convolutional neural networks for the segmentation of gliomas in multi-sequence MRI. International Workshop on Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, 2015, Springer, 131–143.
48. Prasoon, A., Petersen, K., Igel, C., Lauze, F., Dam, E., Nielsen, M., Deep feature learning for knee cartilage segmentation using a triplanar convolutional neural network. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2013, 2013, Springer, 246–253.
49. Prastawa, M., Bullitt, E., Ho, S., Gerig, G., A brain tumor segmentation framework based on outlier detection. Med. Image Anal. 8:3 (2004), 275–283.
50. Rao, A., Ledig, C., Newcombe, V., Menon, D., Rueckert, D., Contusion segmentation from subjects with traumatic brain injury: a random forest framework. Biomedical Imaging (ISBI), 2014 IEEE 11th International Symposium on, 2014, IEEE, 333–336.
51. Ronneberger, O., Fischer, P., Brox, T., U-Net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015, 2015, Springer, 234–241.
52. Roth, H.R., Lu, L., Seff, A., Cherry, K.M., Hoffman, J., Wang, S., Liu, J., Turkbey, E., Summers, R.M., A new 2.5D representation for lymph node detection using random sets of deep convolutional neural network observations. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2014, 2014, Springer, 520–527.
53. Rovira, À., León, A., MR in the diagnosis and monitoring of multiple sclerosis: an overview. Eur. J. Radiol. 67:3 (2008), 409–414.
54. Schmidt, P., Gaser, C., Arsic, M., Buck, D., Förschler, A., Berthele, A., Hoshi, M., Ilg, R., Schmid, V.J., Zimmer, C., et al. An automated tool for detection of FLAIR-hyperintense white-matter lesions in multiple sclerosis. Neuroimage 59:4 (2012), 3774–3783.
55. Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., LeCun, Y., Overfeat: Integrated recognition, localization and detection using convolutional networks. International Conference on Learning Representations, 2014.
56. Sharp, D.J., Beckmann, C.F., Greenwood, R., Kinnunen, K.M., Bonnelle, V., De Boissezon, X., Powell, J.H., Counsell, S.J., Patel, M.C., Leech, R., Default mode network functional and structural connectivity after traumatic brain injury. Brain 134:8 (2011), 2233–2247.
57. Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. Mastering the game of Go with deep neural networks and tree search. Nature 529:7587 (2016), 484–489.
58. Simonyan, K., Zisserman, A., 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409.1556.
59. Springenberg, J. T., Dosovitskiy, A., Brox, T., Riedmiller, M., 2014. Striving for simplicity: the all convolutional net. arXiv preprint arXiv: 1412.6806.
60. Studholme, C., Hill, D.L., Hawkes, D.J., An overlap invariant entropy measure of 3D medical image alignment. Pattern Recognit. 32:1 (1999), 71–86.
61. Sutskever, I., Martens, J., Dahl, G., Hinton, G., On the importance of initialization and momentum in deep learning. Proceedings of the 30th international conference on machine learning (ICML-13), 2013, 1139–1147.
62. Tieleman, T., Hinton, G., Lecture 6.5-RMSprop: divide the gradient by a running average of its recent magnitude. COURSERA: Neural Netw. Mach. Learn, 2012.
63. Tustison, N., Wintermark, M., Durst, C., Brian, A., ANTs and Arboles. in proc of BRATS-MICCAI, 2013.
64. Urban, G., Bendszus, M., Hamprecht, F., Kleesiek, J., Multi-modal brain tumor segmentation using deep convolutional neural networks. in proc of BRATS-MICCAI, 2014.
65. Van Leemput, K., Maes, F., Vandermeulen, D., Suetens, P., Automated model-based tissue classification of MR images of the brain. Med. Imaging IEEE Trans. 18:10 (1999), 897–908.
66. Warner, M.A., de la Plata, C.M., Spence, J., Wang, J.Y., Harper, C., Moore, C., Devous, M., Diaz-Arrastia, R., Assessing spatial relationships between axonal integrity, regional brain volumes, and neuropsychological outcomes after traumatic axonal injury. J. Neurotrauma 27:12 (2010), 2121–2130.
67. Weiss, N., Rueckert, D., Rao, A., Multiple sclerosis lesion segmentation using dictionary learning and sparse coding. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2013, 2013, Springer, 735–742.
68. Wen, P.Y., Macdonald, D.R., Reardon, D.A., Cloughesy, T.F., Sorensen, A.G., Galanis, E., DeGroot, J., Wick, W., Gilbert, M.R., Lassman, A.B., et al. Updated response assessment criteria for high-grade gliomas: response assessment in neuro-oncology working group. J. Clinical Oncol. 28:11 (2010), 1963–1972.
69. Ye, D.H., Zikic, D., Glocker, B., Criminisi, A., Konukoglu, E., Modality propagation: Coherent synthesis of subject-specific scans with data-driven regularization. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2013, 2013, Springer, 606–613.
70. Yuh, E.L., Cooper, S.R., Ferguson, A.R., Manley, G.T., Quantitative CT improves outcome prediction in acute traumatic brain injury. J. Neurotrauma 29:5 (2012), 735–746.
71. Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., Torr, P.H., Conditional random fields as recurrent neural networks. Proceedings of the IEEE International Conference on Computer Vision, 2015, 1529–1537.
72. Zikic, D., Glocker, B., Konukoglu, E., Criminisi, A., Demiralp, C., Shotton, J., Thomas, O., Das, T., Jena, R., Price, S., Decision forests for tissue-specific segmentation of high-grade gliomas in multi-channel MR. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2012, 2012, Springer, 369–376.
73. Zikic, D., Ioannou, Y., Brown, M., Criminisi, A., 2014. Segmentation of brain tumor tissues with convolutional neural networks. in proc of BRATS-MICCAI.