Interleaved text/image deep mining on a large-scale radiology database for automated image interpretation

Journal of Machine Learning Research

Volumn 17, Issue , 2016, Pages 1-31

  Shin, Hoo Chang ^a   Lu, Le ^a   Kim, Lauren ^a   Seff, Ari ^a   Yao, Jianhua ^a   Summers, Ronald M ^a  

^a NATIONAL INSTITUTES OF HEALTH   (United States)

Author keywords

Convolutional neural networks; Deep learning; Medical imaging; Natural language processing; Topic models

Indexed keywords

ARTIFICIAL INTELLIGENCE; COMPUTER VISION; DIAGNOSIS; ECONOMIC AND SOCIAL EFFECTS; LEARNING ALGORITHMS; LEARNING SYSTEMS; MEDICAL IMAGING; NATURAL LANGUAGE PROCESSING SYSTEMS; NEURAL NETWORKS; PICTURE ARCHIVING AND COMMUNICATION SYSTEMS; RADIATION; RADIOLOGY; SEMANTICS;

AUTOMATED IMAGE INTERPRETATIONS; CONVOLUTIONAL NEURAL NETWORK; DEEP LEARNING; ELECTRONIC PATIENT RECORD; MEDICAL IMAGE DATABASE; NATURAL LANGUAGE PROCESSING; TOPIC MODEL; TWO DIMENSIONAL IMAGES;

IMAGE PROCESSING;

EID: 84989187487     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (60)

1. Openi - an open access biomedical image search engine. http://openi.nlm.nih.gov. Lister Hill National Center for Biomedical Communications, U.S. National Library of Medicine.
2. Kobus Barnard, Pinar Duygulu, David Forsyth, Nando de Freitas, David M Blei, and Michael I Jordan. Matching words and pictures. Journal of machine learning research, 3 (Feb):1107-1135, 2003.
3. Tamara L Berg, Alexander C Berg, and Jonathan Shih. Automatic attribute discovery and characterization from noisy web data. In European Conference on Computer Vision, pages 663-676. Springer, 2010.
4. Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python. O'Reilly Media, Inc., 2009.
5. David M Blei and Michael I Jordan. Modeling annotated data. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, pages 127-134. ACM, 2003.
6. David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. Journal of machine Learning research, 3:993-1022, 2003.
7. Luke Carrivick, Sanjay Prabhu, Paul Goddard, and Jonathan Rossiter. Unsupervised learning in radiology using novel latent variable models. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 2, pages 854-859. IEEE, 2005.
8. Wendy W Chapman, Will Bridewell, Paul Hanbury, Gregory F Cooper, and Bruce G Buchanan. A simple algorithm for identifying negated findings and diseases in discharge summaries. Journal of biomedical informatics, 34(5):301-310, 2001.
9. Wendy W Chapman, Dieter Hilert, Sumithra Velupillai, Maria Kvist, Maria Skeppstedt, Brian E Chapman, Michael Conway, Melissa Tharp, Danielle L Mowery, and Louise Deleger. Extending the negex lexicon for multiple languages. Studies in health technology and informatics, 192:677, 2013.
10. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In Computer Vision and Pattern Recognition, pages 248-255. IEEE, 2009.
11. Thomas Deselaers and Hermann Ney. Deformations, patches, and discriminative models for automatic annotation of medical radiographs. Pattern Recognition Letters, 29(15):2003-2010, 2008.
12. Chris Ding, Tao Li, and Wei Peng. Nonnegative matrix factorization and probabilistic latent semantic indexing: Equivalence chi-square statistic, and a hybrid method. In Proceedings of the national conference on artificial intelligence, volume 21, page 342. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 2006.
13. Jeff Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional networks for visual recognition and description. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015.
14. Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Tomas Mikolov, et al. Devise: A deep visual-semantic embedding model. In Advances in neural information processing systems, pages 2121-2129, 2013.
15. Eric Gaussier and Cyril Goutte. Relation between plsa and nmf and implications. In Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pages 601-602. ACM, 2005.
16. Mark Girolami and Ata Kabán. On an equivalence between plsi and lda. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, pages 433-434. ACM, 2003.
17. Ashish Gupta, Murat Ayhan, and Anthony Maida. Natural image bases to represent neuroimaging data. In Proceedings of the International Conference on Machine Learning, 2013.
18. Saurabh Gupta, Ross Girshick, Pablo Arbeláez, and Jitendra Malik. Learning rich features from rgb-d images for object detection and segmentation. In European Conference on Computer Vision, pages 345-360. Springer, 2014.
19. Micah Hodosh, Peter Young, and Julia Hockenmaier. Framing image description as a ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research, pages 853-899, 2013.
20. Thomas Hofmann. Probabilistic latent semantic indexing. In Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, pages 50-57. ACM, 1999.
21. Betsy L Humphreys, Donald AB Lindberg, Harold M Schoolman, and G Octo Barnett. The unified medical language system an informatics research collaboration. Journal of the American Medical Informatics Association, 5(1):1-11, 1998.
22. Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Deep features for text spotting. In European conference on computer vision, pages 512-528. Springer, 2014.
23. Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM international conference on Multimedia, pages 675-678. ACM, 2014.
24. Andrej Karpathy, Armand Joulin, and Fei Fei F Li. Deep fragment embeddings for bidirectional image sentence mapping. In Advances in Neural Information Processing Systems, pages 1889-1897, 2014.
25. M Hadi Kiapour, Kota Yamaguchi, Alexander C Berg, and Tamara L Berg. Hipster wars: Discovering elements of fashion styles. In European conference on computer vision, pages 472-488. Springer, 2014.
26. Gunhee Kim, Leonid Sigal, and Eric P Xing. Joint summarization of large-scale collections of web images and videos for storyline reconstruction. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages 4225-4232. IEEE, 2014.
27. Gunhee Kim, Seungwhan Moon, and Leonid Sigal. Joint photo stream and blog post summarization and exploration. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015a.
28. Gunhee Kim, Seungwhan Moon, and Leonid Sigal. Ranking and retrieval of image sequences from multiple paragraph queries. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015b.
29. Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Computer Science Department, University of Toronto, Tech. Rep, 2009.
30. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1105, 2012.
31. G. Kulkarni, V. Premraj, V. Ordonez, S. Dhar, S. Li, Y. Choi, A. Berg, and T. Berg. Babytalk: Understanding and generating simple image descriptions. IEEE Trans. Pattern Anal. Mach. Intell., 35(12):2891-2903, 2013.
32. Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Attribute-based classification for zero-shot visual object categorization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(3):453-465, 2014.
33. Curtis P Langlotz. Radlex: A new method for indexing online educational materials 1. Radiographics, 26(6):1595-1597, 2006.
34. Daniel D Lee and H Sebastian Seung. Learning the parts of objects by non-negative matrix factorization. Nature, 401(6755):788-791, 1999.
35. S. Li, G. Kulkarni, T. Berg, A. Berg, and Y. Choi. Composing simple image descriptions using web-scale n-grams. In ACM CoNLL, pages 220-228, 2011.
36. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pages 740-755. Springer, 2014.
37. Donald A Lindberg, Betsy L Humphreys, and Alexa T McCray. The unified medical language system. Methods of information in Medicine, 32(4):281-291, 1993.
38. Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, and Alan Yuille. Deep captioning with multimodal recurrent neural networks (m-rnn). Proceedings of the International Conference on Learning Representations (ICLR), 2015.
39. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013a.
40. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111-3119, 2013b.
41. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In HLT-NAACL, pages 746-751. Citeseer, 2013c.
42. George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38 (11):39-41, 1995.
43. Vicente Ordonez and Tamara L Berg. Learning high-level judgments of urban perception. In European Conference on Computer Vision, pages 494-510. Springer, 2014.
44. Vicente Ordonez, Girish Kulkarni, and Tamara L Berg. Im2text: Describing images using 1 million captioned photographs. In Advances in Neural Information Processing Systems, pages 1143-1151, 2011.
45. Cyrus Rashtchian, Peter Young, Micah Hodosh, and Julia Hockenmaier. Collecting image annotations using amazon's mechanical turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, pages 139-147. Association for Computational Linguistics, 2010.
46. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-252, 2015.
47. Walter J Scheirer, Neeraj Kumar, Peter N Belhumeur, and Terrance E Boult. Multiattribute spaces: Calibration for attribute fusion and similarity search. In Computer Vision and Pattern Recognition, 2012 IEEE Conference on, pages 2933-2940. IEEE, 2012.
48. Lynn Marie Schriml, Cesar Arze, Suvarna Nadendla, Yu-Wei Wayne Chang, Mark Mazaitis, Victor Felix, Gang Feng, and Warren Alden Kibbe. Disease ontology: a backbone for disease semantic integration. Nucleic acids research, 40(D1):D940-D946, 2012.
49. Peri L Schuyler, William T Hole, Mark S Tuttle, and David D Sherertz. The umls metathesaurus: representing different views of biomedical concepts. Bulletin of the Medical Library Association, 81(2):217, 1993.
50. Hoo-Chang Shin, Matthew R Orton, David J Collins, Simon J Doran, and Martin O Leach. Stacked autoencoders for unsupervised feature learning and multiple organ detection in a pilot study using 4d patient data. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 35(8):1930-1943, 2013.
51. Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. Proceedings of the International Conference on Learning Representations (ICLR), 2015.
52. Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. Zero-shot learning through cross-modal transfer. In Advances in Neural Information Processing Systems, pages 935-943, 2013.
53. Keith Stevens, Philip Kegelmeyer, David Andrzejewski, and David Buttler. Exploring topic coherence over many models and many topics. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 952-961. Association for Computational Linguistics, 2012.
54. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015.
55. Joseph Turian, Lev Ratinov, and Yoshua Bengio. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th annual meeting of the association for computational linguistics, pages 384-394. Association for Computational Linguistics, 2010.
56. Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096-1103. ACM, 2008.
57. Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. The Journal of Machine Learning Research, 11:3371-3408, 2010.
58. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015.
59. Kelvin Xu, Jimmy Ba, Ryan Kiros, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. Proceedings of the 31st International Conference on Machine Learning, 2015.
60. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.