Medical image retrieval: A multimodal approach

Cancer Informatics

Volumn 2014, Issue , 2014, Pages 125-136

  Cao, Yu ^a   Steffey, Shawn ^a   He, Jianbiao ^b   Xiao, Degui ^c   Tao, Cui ^d   Chen, Ping ^e   Müller, Henning ^f  

^a University of Massachusetts Lowell   (United States)

^b CENTRAL SOUTH UNIVERSITY   (China)

^c HUNAN UNIVERSITY   (China)

^d UNIVERSITY OF TEXAS   (United States)

^e UNIVERSITY OF MASSACHUSETTS BOSTON   (United States)

^f UNIVERSITY OF APPLIED SCIENCES WESTERN SWITZERLAND   (Switzerland)

Author keywords

Content based image retrieval; Deep boltzmann machine; Deep learning; Extended probabilistic latent semantic analysis; Multi modal and content based medical image retrieval

Indexed keywords

DIAGNOSTIC IMAGING; EXPERIMENTAL MODEL; IMAGE RETRIEVAL; JOINT; LEARNING; MACHINE; STATISTICAL MODEL;

EID: 84981714705     PISSN: None     EISSN: 11769351     Source Type: Journal    
DOI: 10.4137/CIN.S14053     Document Type: Article

References (73)

1. Marcia A, Kassirer P, Relman A. Looking back on the Millennium in medicine. N Engl J Med. 2000; 342: 42-9.
2. Diagnostic imaging market to increase to $26. 6 billion by 2016 by Companies & Markets, a leading global aggregator of business information. 2013. Available at: http://www.companiesandmarkets.com/News/Healthcare-and-Medical/Diagnostic-imaging-market-to-increase-to-26-6-billion-by-2016/NI6386.
3. Roobottom C, Mitchell G, Morgan-Hughes G. Radiation-reduction strategies in cardiac computed tomographic angiography. Clin Radiol. 2010; 65 (11): 859-67.
4. AT&T, Accenture Service Stores Medical Images in Cloud. Available at: http://www.informationweek.com/news/healthcare/interoperability/232200581.
5. Morin RL. Transforming the radiological interpretation process (TRIP). J Digital Imaging. 2004; 17 (2): 78-9.
6. Medical imaging in cancer care: charting the progress, prepared by Polidais LLC for Philips Healthcare. Available at: http://www.healthcare.philips.com/pwc_hc/us_en/about/Reimbursement/assets/docs/cancer_white_paper.pdf.
7. Brinkley JF, Greenes RA. Imaging and structural informatics. In: Shortliffe EH, Cimino JJ, eds. Biomedical Informatics: Computer Applications in Health Care and Biomedicine. 3rd ed. New York: Springer; 2006: 344-78.
8. Aisen AM, Broderick LS, Winer-Muram H, et al. Automated storage and retrieval of thin-section CT images to assist diagnosis: system description and preliminary assessment. Radiology. 2003; 228 (1): 265-70.
9. Müller H, Michoux N, Bandon D, Geissbuhler A. A review of content-based image retrieval systems in medicine-clinical benefits and future directions. Int J Med Inform. 2004; 73: 1-23.
10. Lehmann TM, Güld MO, Thies C, et al. Content-based image retrieval in medical applications. Methods Inf Med. 2004; 43 (4): 354-61.
11. Ruiz ME. Combining image features, case descriptions and UMLS concepts to improve retrieval of medical images. Paper presented at: Proceedings of American Medical Informatics Association Annual Symposium; 2006.
12. Deserno TM, Antani S, Long R. Exploring access to scientific literature using content-based image retrieval. Paper presented at: Proceedings of the SPIE, Medical Imaging 2007: PACS and Imaging Informatics; 2007; San Diego, California, USA.
13. Müller H, Kalpathy-Cramer J. Analyzing the content out of context-features and gaps in medical image retrieval. J Healthcare Inf Sys Inf. 2009; 4 (1): 88-98.
14. Névéol A, Deserno TM, Darmoni SJ, Güld MO, Aronson AR. Natural language processing versus content-based image analysis for medical document retrieval. J Am Soc Inf Sci Technol. 2009; 60 (1): 123-34.
15. Kalpathy-Cramer J, Hersh W. Medical image retrieval and automatic annotation: OHSU at ImageCLEF 2007. Paper presented at: Proceedings of 8th Workshop of the Cross-Language Evaluation Forum, CLEF; 2007; Budapest, Hungary.
16. Lindberg DA, Humphreys BL, McCray AT. The unified medical language system. Methods Inf Med. 1993; 32 (4): 281-91.
17. Demner-Fushman D, Antani S, Siadat M-R, Soltanian-Zadeh H, Fotouhi F, Elisevich AK. Automatically finding images for clinical decision support. Paper presented at: Proceedings of the Seventh IEEE International Conference on Data Mining (ICDM) Workshops; 2007; Omaha, Nebraska, USA.
18. Atmosukarto I, Travillian R, Franklin J, et al. A unifying framework for combining content-based image retrieval with relational database queries for biomedical applications. Paper presented at: Proceedings of Annual Meeting of the Society for Imaging Informatics in Medicine; 2008.
19. Syeda-Mahmood T, Wang F, Beymer D, Amir A, Richmond M, Hashmi S. AALIM: Multi-modal Mining for healthcare decision support. Paper presented at: Proceedings of IEEE Conference on Computers in Cardiology (CinC), 2007-2009; Durham, North Carolina, USA.
20. Müller H, Geissbuhler A. Medical Multimedia Retrieval 2. 0 In: Year book of Medical Informatics-Methods of Information in Medicine. Murray P, Ed. Stuttgart, Germany: Schattauer Publishers. 2008; 3: 55-64.
21. Cao Y, Kalpathy-Cramer J, Ünay D. Medical multimedia analysis and retrieval. Paper presented at: Proceedings of the 19th ACM international conference on Multimedia (ACM MM 2011); 2011; Phoenix, AZ, U. S. A.
22. Müller H, Greenspan H. Overview of the Third Workshop on Medical Content-Based Retrieval for Clinical Decision Support (MCBR—CDS 2012). Medical Content-Based Retrieval for Clinical Decision Support. Berlin: Springer; 2012: 1-9.
23. Benois-Pineau J, Briassouli A, Hauptmann A. ACM MM MIIRH 2013: workshop on multimedia indexing and information retrieval for healthcare. Paper presented at: Proceedings of the 21st ACM international conference on Multimedia; 2013.
24. Pham T-T, Maillot NE, Lim J-H, Chevallet J-P. Latent semantic fusion model for image retrieval and annotation. Paper presented at: Proceedings of the sixteenth ACM conference on Conference on information and knowledge management (CIKM); 2007; Lisbon, Portugal.
25. Lienhart R, Romberg S, Hörster E. Multilayer pLSA for multimodal image retrieval. Paper presented at: Proceedings of the ACM International Conference on Image and Video Retrieval (CIVR); 2009; Island of Santorini, Greece.
26. Bengio Y. Learning Deep Architectures for AI. Foundations and trends® in Machine Learning. Vol 2. Boston: Now publishers inc; 2009: 1-127.
27. Salakhutdinov R, Hinton GE. Deep Boltzmann machines. Paper presented at: International Conference on Artificial Intelligence and Statistics; 2009.
28. Datta R, Joshi D, Li J, Wang JZ. Image retrieval: Ideas, influences, and trends of the new age. ACM Comput Surv. 2008; 40 (2): 5, 1-60.
29. Hidki A, Depeursinge A, Iavindrasana J, Pitkanen MJ, Zhou X, Müller H. The medGIFT project: perspective of a medical doctor. Jpn J Med Imaging Technol. 2007; 25: 356-61.
30. Greenspan H, Pinhas AT. Medical image categorization and retrieval for PACS using the GMM-KL framework. IEEE Trans Inf Technol Biomed. 2007; 11 (2): 190-202.
31. Napel SA, Beaulieu CF, Rodriguez C, et al. Automated retrieval of CT images of liver lesions on the basis of image similarity: method and preliminary results 1. Radiology. 2010; 256 (1): 243-52.
32. Rahman MM, Antani SK, Long RL, Demner-Fushman D, Thoma GR. Multi-Modal Query Expansion Based on Local Analysis for Medical Image Retrieval. Medical Content-Based Retrieval for Clinical Decision Support. Berlin: Springer; 2010: 110-9.
33. Quellec G, Lamard M, Cazuguel G, Roux C, Cochener B. Case retrieval in medical databases by fusing heterogeneous information. IEEE Trans Med Imaging. 2011; 30 (1): 108-18.
34. Siegel E, Reiner B. The Radiological Society of North America’s medical image resource center: an update. J Digital Imaging. 2001; 14 (1): 77-9.
35. Kalpathy-Cramer J, de Herrera AGS, Demner-Fushman D, Antani S, Bedrick S, Müller H. Evaluating performance of biomedical image retrieval systems-an overview of the medical image retrieval task at ImageCLEF 2004-2013. Comput Med Imaging Graph. 2015; 39: 55-61.
36. Simpson MS, You D, Rahman MM, Antani SK, Thoma GR, Demner-Fushman D. Towards the creation of a visual ontology of biomedical imaging entities. Paper presented at: AMIA Annual Symposium Proceedings; 2012.
37. Cheng B, Stanley RJ, De S, Antani S, Thoma GR. Automatic detection of arrow annotation overlays in biomedical images. IJHISI. 2011; 6 (4): 23-41.
38. You D, Simpson M, Antani S, Demner-Fushman D, Thoma GR. Annotating image ROIs with text descriptions for multimodal biomedical document retrieval. Paper presented at: IS&T/SPIE Electronic Imaging; 2013.
39. Charles E, Kahn J, Thao C. GoldMiner: a radiology image search engine. Am J Roentgenol. 2007; 188: 1475-8.
40. Kumar A, Kim J, Cai W, Fulham M, Feng D. Content-based medical image retrieval: a survey of applications to multidimensional and multimodality data. J Digital Imaging. 2013; 26 (6): 1025-39.
41. Hinton GE, Salakhutdinov RR. Reducing the dimensionality of data with neural networks. Science. 2006; 313 (5786): 504-7.
42. Scientists see promise in deep-learning programs, by John Markoff. New York Times. 2012. Available at: http://www.nytimes.com/2012/11/24/science/scientists-see-advances-in-deep-learning-a-part-of-artificial-intelligence.html.
43. Deep learning makes MIT Tech review list of top-10 breakthroughs of 2013 by MIT Tech Review. 2013. Available at: http://www.technologyreview.com/featuredstory/513696/deep-learning/.
44. NYU Deep Learning Professor LeCun Will Head Facebooks New Artificial Intelligence Lab, by Josh Constine, Ntechcrunch. com; 2013. Available at: http://techcrunch.com/2013/12/09/facebook-artificialintelligence-lab-lecun/.
45. Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. Paper presented at: NIPS; 2012.
46. Eigen D, Rolfe J, Fergus R, LeCun Y. Understanding Deep Architectures Using a Recursive Convolutional Network. Ithaca: arXiv; 2013. [preprint arXiv:1312.1847].
47. Vincent P, Larochelle H, Bengio Y, Manzagol P-A. Extracting and composing robust features with denoising autoencoders. Paper presented at: Proceedings of the 25th international conference on Machine learning; 2008.
48. Bengio Y, Yao L, Alain G, Vincent P. Generalized denoising auto-encoders as generative models. Paper presented at: Advances in Neural Information Processing Systems; 2013.
49. Salakhutdinov R, Mnih A, Hinton G. Restricted Boltzmann machines for collaborative filtering. Paper presented at: Proceedings of the 24th international conference on Machine learning; 2007.
50. Hinton GE. A Practical Guide to Training Restricted Boltzmann Machines. Neural Networks: Tricks of the Trade. Berlin: Springer; 2012: 599-619.
51. Hinton GE, Sejnowski TJ, Ackley DH. Boltzmann Machines: Constraint Satisfaction Networks that Learn. Pittsburgh, PA: Carnegie-Mellon University, Department of Computer Science; 1984.
52. Guillaumin M, Verbeek J, Schmid C. Multimodal semi-supervised learning for image classification. Paper presented at: Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference; 2010.
53. Srivastava N, Salakhutdinov R. Multimodal learning with deep Boltzmann machines. Paper presented at: NIPS; 2012.
54. The MIRFLICKR retrieval evaluation. Available at: http://press.liacs.nl/mirflickr/.
55. Zheng Y-T, Zhao M, Neo S-Y, Chua T-S, Tian Q. Visual Synset: Towards a higher-level visual representation. Paper presented at: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2008; Anchorage, AK, USA.
56. Liu D, Hua G, Viola P, Chen T. Integrated feature selection and higher-order spatial feature extraction for object categorization. Paper presented at: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2008; Anchorage, AK.
57. Zhang S, Tian Q, Hua G, Huang Q, Li S. Descriptive visual words and visual phrases for image applications. Paper presented at: Proceedings of the seventeen ACM international conference on Multimedia (ACM MM); 2009; Beijing, China.
58. Lazebnik S, Schmid C, Ponce J. Spatial pyramid matching. In: Dickinson S, Leonardis A, Schiele B, Tarr M, eds. Object Categorization: Computer and Human Vision Perspectives. Cambridge, UK: Cambridge University Press; 2009: 401-5.
59. Lazebnik S, Raginsky M. Supervised learning of quantizer codebooks by information loss minimization. IEEE Trans Pattern Anal Mach Intell. 2009; 31 (7): 1294-309.
60. Moosmann F, Nowak E, Jurie F. Randomized clustering forests for image classification. Trans Pattern Anal Mach Intell. 2008; 30 (9): 1632-46.
61. Lowe DG. Distinctive image features from scale-invariant keypoints. Int J Comput Vis. 2004; 60 (2): 91-110.
62. Manning CD, Surdeanu M, Bauer J, Finkel J, Bethard SJ, McClosky D. The Stanford CoreNLP natural language processing toolkit. Paper presented at: Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations 2014; 2014.
63. Sivic J, Russell B, Efros A, Zisserman A, Freeman W. Discovering object categories in image collections. Paper presented at: Proceedings of the IEEE International Conference on Computer Vision (ICCV); 2005; Beijing, P. R. China.
64. Hinton GE, Osindero S, Teh Y-W. A fast learning algorithm for deep belief nets. Neural Comput. 2006; 18 (7): 1527-54.
65. Arel I, Rose DC, Karnowski TP. Deep machine learning-a new frontier in artificial intelligence research [research frontier]. IEEE Comput Intell Mag. 2010; 5 (4): 13-8.
66. Casella G, George EI. Explaining the Gibbs sampler. Am Stat. 1992; 46 (3): 167-74.
67. Ngiam J, Khosla A, Kim M, Nam J, Lee H, Ng AY. Multimodal deep learning. Paper presented at: Proceedings of the 28th International Conference on Machine Learning (ICML-11); 2011.
68. Srivastava N, Salakhutdinov R. Multimodal learning with deep Boltzmann machines. Paper presented at: Advances in Neural Information Processing Systems 25; 2012.
69. Lazebnik S, Schmid C, Ponce J. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. Paper presented at: Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR); 2006; New York, NY, USA.
70. Akgül CB, Napel DLRS, Beaulieu CF, Greenspan H, Acar B. Content-based image retrieval in radiology: current status and future directions. J Digital Imaging. 2011; 24 (2): 208-22.
71. trec_eval. A standard tool used by the TREC community for evaluating an ad hoc retrieval run. 2010. Available at: http://trec.nist.gov/trec_eval/2010. Accessed February 1, 2010.
72. de Herrera AGS, Kalpathy—Cramer J, Demner-Fushman D, Antani S, Müller H. Overview of the ImageCLEF 2013 medical tasks. In: Working notes of CLEF; 2013.
73. Simpson MS, You D, Rahman MM, Demner-Fushman D, Antani S, Thoma G. ITI’s participation in the 2013 medical track of ImageCLEF. In: Working Notes of CLEF; 2013.