Learning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotation

Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition

Volumn 2016-December, Issue , 2016, Pages 2497-2506

  Shin, Hoo Chang ^a   Roberts, Kirk ^b   Lu, Le ^a   Demner Fushman, D ^b   Yao, Jianhua ^a   Summers, Ronald M ^a  

^a NATIONAL INSTITUTES OF HEALTH   (United States)

^b NATIONAL INSTITUTES OF HEALTH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER VISION; IMAGE ANALYSIS; IMAGE RETRIEVAL; NEURAL NETWORKS; PATTERN RECOGNITION; STREAM FLOW;

AUTOMATED IMAGE ANNOTATIONS; COMPOSITE IMAGES; CONVOLUTIONAL NEURAL NETWORK; DOMAIN SPECIFIC; IMAGE ANNOTATION; NATURAL IMAGES; RECURRENT NEURAL NETWORK (RNNS); REGULARIZATION TECHNIQUE;

RECURRENT NEURAL NETWORKS;

EID: 84986277510     PISSN: 10636919     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CVPR.2016.274     Document Type: Conference Paper

References (65)

1. Mesh: Medical subject headings. https://www.nlm. nih.gov/mesh/meshhome.html.
2. Open-i: An open access biomedical search engine. https: //openi.nlm.nih.gov.
3. U. Avni, H. Greenspan, E. Konen, M. Sharon, and J. Goldberger. X-ray categorization and retrieval on the organ and pathology level, using patch-based visual words. Medical Imaging, IEEE Transactions on, 2011.
4. S. Bird, E. Klein, and E. Loper. Natural language processing with Python. " O'Reilly Media, Inc.", 2009.
5. H. Boussaid and I. Kokkinos. Fast and exact: Admm-based discriminative shape segmentation with loopy part models. 2014.
6. X. Chen and C. Lawrence Zitnick. Mind's eye: A recurrent visual representation for image caption generation. In CVPR, 2015.
7. K. Cho, B. Van Merrïenboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.
8. J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
9. J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. Gated feedback recurrent neural networks. arXiv preprint arXiv:1502.02367, 2015.
10. S. J. Darmoni, L. F. Soualmia, C. Letord, M.-C. Jaulent, N. Griffon, B. Thirion, and A. Névéol. Improving information retrieval using medical subject headings concepts: a test case on rare and chronic diseases. Journal of the Medical Library Association: JMLA, 100(3):176, 2012.
11. D. Demner-Fushman, M. D. Kohli, M. B. Rosenman, S. E. Shooshan, L. Rodriguez, S. Antani, G. R. Thoma, and C. J. McDonald. Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association, page ocv080, 2015.
12. D. Demner-Fushman, S. E. Shooshan, L. Rodriguez, S. Antani, and G. R. Thoma. Annotation of chest radiology reports for indexing and retrieval. Multimodal Retrieval in the Medical Domain (MRMD) 2015.
13. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
14. J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. In CVPR, 2015.
15. H. Fang, S. Gupta, F. Iandola, R. K. Srivastava, L. Deng, P. Dollar, J. Gao, X. He, M. Mitchell, J. C. Platt, et al. From captions to visual concepts and back. In CVPR, 2015.
16. A. Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth. Every picture tells a story: Generating sentences from images. In ECCV. 2010.
17. Y. Feng and M. Lapata. How many words is a picture worth? automatic caption generation for news images. In Proceedings of the 48th annual meeting of the Association for Computational Linguistics, 2010.
18. A. Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013.
19. A. Graves, A.-r. Mohamed, and G. Hinton. Speech recognition with deep recurrent neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 2013.
20. R. B. Haynes, N.Wilczynski, K. A. McKibbon, C. J.Walker, and J. C. Sinclair. Developing optimal search strategies for detecting clinically sound studies in medline. Journal of the American Medical Informatics Association, 1(6):447, 1994.
21. S. Hermann. Evaluation of scan-line optimization for 3d medical image registration. In CVPR, 2014.
22. W. Hersh and E. Voorhees. Trec genomics special issue overview. Information Retrieval, 12(1):1-15, 2009.
23. G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012.
24. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.
25. M. Hodosh, P. Young, and J. Hockenmaier. Framing image description as a ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research, 2013.
26. J. Hofmanninger and G. Langs. Mapping visual features to semantic profiles for retrieval in medical imaging. In CVPR, 2015.
27. S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
28. O. Irsoy and C. Cardie. Opinion mining with deep recurrent neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.
29. S. Jaeger, A. Karargyris, S. Candemir, L. Folio, J. Siegelman, F. Callaghan, Z. Xue, K. Palaniappan, R. K. Singh, S. Antani, et al. Automatic tuberculosis screening using chest radiographs. Medical Imaging, IEEE Transactions on, 2014.
30. R. Jozefowicz, W. Zaremba, and I. Sutskever. An empirical exploration of recurrent network architectures. In ICML 2015, 2015.
31. A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In CVPR, 2015.
32. A. Karpathy, J. Johnson, and F.-F. Li. Visualizing and understanding recurrent networks. arXiv preprint arXiv:1506.02078, 2015.
33. G. Kim, S. Moon, and L. Sigal. Joint photo stream and blog post summarization and exploration. In CVPR, 2015.
34. G. Kim, S. Moon, and L. Sigal. Ranking and retrieval of image sequences from multiple paragraph queries. In CVPR, 2015.
35. G. Kim, L. Sigal, and E. P. Xing. Joint summarization of large-scale collections of web images and videos for storyline reconstruction. In CVPR, 2014.
36. R. Kiros, R. Salakhutdinov, and R. S. Zemel. Unifying visual-semantic embeddings with multimodal neural language models. Transactions of the Association for Computational Linguistics, 2015.
37. B. Klein, G. Lev, G. Sadeh, and L. Wolf. Associating neural word embeddings with deep image representations using fisher vectors. In CVPR, 2015.
38. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. 2012.
39. G. Kulkarni, V. Premraj, V. Ordonez, S. Dhar, S. Li, Y. Choi, A. C. Berg, and T. Berg. Babytalk: Understanding and generating simple image descriptions. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 35(12):2891-2903, 2013.
40. C. Ledig, W. Shi, W. Bai, and D. Rueckert. Patch-based evaluation of image segmentation. In CVPR, 2014.
41. M. Lin, Q. Chen, and S. Yan. Network in network. Proceedings of the International Conference on Learning Representations (ICLR), 2014.
42. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV. 2014.
43. M.-T. Luong, I. Sutskever, Q. V. Le, O. Vinyals, and W. Zaremba. Addressing the rare word problem in neural machine translation. In Proceedings of ACL, 2015.
44. J. Mao, W. Xu, Y. Yang, J. Wang, and A. L. Yuille. Explain images with multimodal recurrent neural networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2015.
45. J. Melendez, B. van Ginneken, P. Maduskar, R. H. Philipsen, K. Reither, M. Breuninger, I. M. Adetifa, R. Maane, H. Ayles, C. Sanchez, et al. A novel multiple-instance learning-based approach to computer-aided detection of tuberculosis on chest x-rays. Medical Imaging, IEEE Transactions on, 2015.
46. T. A. Ngo and G. Carneiro. Fully automated non-rigid segmentation with distance regularized level set evolution initialized and constrained by deep-structured inference. In CVPR, 2014.
47. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, 2002.
48. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 2011.
49. C. Rupprecht, L. Peter, and N. Navab. Image segmentation in twenty questions. In CVPR, 2015.
50. Y. Shi, H.-I. Suk, Y. Gao, and D. Shen. Joint coupled-feature representation and coupled boosting for ad diagnosis. In CVPR, 2014.
51. H.-C. Shin, L. Lu, L. Kim, A. Seff, J. Yao, and R. M. Summers. Interleaved text/image deep mining on a very largescale radiology database. In CVPR, 2015.
52. H.-C. Shin, L. Lu, L. Kim, A. Seff, J. Yao, and R. M. Summers. Interleaved text/image deep mining on a large-scale radiology database for automated image interpretation. Journal of Machine Learning Research (JMLR), 2016.
53. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
54. R. Socher, A. Karpathy, Q. V. Le, C. D. Manning, and A. Y. Ng. Grounded compositional semantics for finding and describing images with sentences. Transactions of the Association for Computational Linguistics, 2014.
55. N. Subbanna, D. Precup, and T. Arbel. Iterative multilevel mrf leveraging context and voxel information for brain tumour segmentation in mri. In CVPR, 2014.
56. I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In NIPS 2014.
57. T. Syeda-Mahmood, R. Kumar, and C. Compas. Learning the correlation between images and disease labels using ambiguous learning. In MICCAI. 2015.
58. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. 2015.
59. L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 2008.
60. S. Venugopalan, M. Rohrbach, J. Donahue, R. Mooney, T. Darrell, and K. Saenko. Sequence to sequence-video to text. ICCV, 2015.
61. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. In CVPR, 2015.
62. K. Xu, J. Ba, R. Kiros, A. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. 2015.
63. Z. Xue, D. You, S. Candemir, S. Jaeger, S. Antani, L. R. Long, and G. R. Thoma. Chest x-ray image view classification. In Computer-Based Medical Systems (CBMS), 2015 IEEE 28th International Symposium on, 2015.
64. L. Yao, A. Torabi, K. Cho, N. Ballas, C. Pal, H. Larochelle, and A. Courville. Describing videos by exploiting temporal structure. ICCV, 2015.
65. P. Young, A. Lai, M. Hodosh, and J. Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2014.