Bridging the semantic gap via functional brain imaging

IEEE Transactions on Multimedia

Volumn 14, Issue 2, 2012, Pages 314-325

  Hu, Xintao ^a   Li, Kaiming ^a   Han, Junwei ^a   Hua, Xiansheng ^b   Guo, Lei ^a   Liu, Tianming ^c  

^a NORTHWESTERN POLYTECHNICAL UNIVERSITY   (China)

^b MICROSOFT RESEARCH ASIA   (China)

^c University of Georgia ^*  (United States)

Author keywords

brain imaging; Brain computer interface; high level features; low level features; semantic gap

Indexed keywords

BRAIN IMAGING; BRAIN REGIONS; CATEGORY CLASSIFICATION; COGNITIVE NEUROSCIENCES; COMPUTATIONAL FRAMEWORK; DATA SETS; FEATURE PROJECTION; FUNCTIONAL BRAIN IMAGING; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HIGH LEVEL SEMANTICS; HIGH-LEVEL FEATURES; HUMAN BRAIN; LOW-LEVEL FEATURES; MULTIMEDIA CONTENT ANALYSIS; SEMANTIC FEATURES; SEMANTIC GAP; TASK-BASED; TRECVID; VIDEO CLASSIFICATION; VIDEO DATASETS;

BRAIN; CLASSIFICATION (OF INFORMATION); FUNCTIONAL NEUROIMAGING; KNOWLEDGE MANAGEMENT; MAGNETIC RESONANCE IMAGING;

SEMANTICS;

EID: 84859016333     PISSN: 15209210     EISSN: None     Source Type: Journal    
DOI: 10.1109/TMM.2011.2172201     Document Type: Article

References (57)

1. A. W. M. Smeulders, M. Worring, S. Santini, A. Gupta, and R. Jain, "Content-based image retrieval at the end of the early years," IEEE Trans. Pattern Anal.Mach. Intell., vol. 22, no. 12, pp. 1349-1380, Dec. 2000. (Pubitemid 32088465)
2. Y. Liu, D. Zhang, G. Lu, and W.-Y.Ma, "A survey of content-based image retrieval with high-level semantics," Pattern Recognit., vol. 40, no. 1, pp. 262-282, 2007. (Pubitemid 44486285)
3. F. Fotouhi,W. I.Grosky, and P. Stanchev, "Guest editors' introduction: The many faces of semantics," IEEE Multimedia, vol. 16, no. 2, pp. 16-17, Apr.-Jun. 2009.
4. R. Arndt, R. Troncy, S. Staab, and L. Hardman, "COMM: A core ontology for multimedia," in Handbook on Ontologies, R. Studer, Ed. Berlin, Germany: Springer, 2009.
5. A. Hauptmann, R. Yan, andW.-H Lin, "How many high-level concepts will fill the semantic gap in news video retrieval?," in Proc. 6th ACM Int. Conf. Image and Video Retrieval, pp. 627-634.
6. [Online]. Available: http://www.lscom.org
7. A. Martin and L. L. Chao, "Semantic memory and the brain: Structure and processes," Curr. Opin. Neurobiol., vol. 11, no. 2, pp. 194-201, 2001. (Pubitemid 32289406)
8. K. J. Friston, "Modalities, modes and models in functional neuroimaging," Science, vol. 326, no. 5951, pp. 399-403, 2009.
9. N. K. Logothetis, "What we can do and what we cannot do with fMRI," Nature, vol. 453, no. 7197, pp. 869-878, 2008. (Pubitemid 351832564)
10. T. M. Mitchell, S. V. Shinkareva, A. Carlson, K. M. Chang, V. L. Malave, R. A. Mason, and M. A. Just, "Predicting human brain activity associated with the meanings of nouns," Science, vol. 320, no. 5880, pp. 1191-1195, 2008. (Pubitemid 351929502)
11. U. Hasson, R.Malach, and D. J. Heeger, "Reliability of cortical activity during natural stimulation," Trends Cogn. Sci., vol. 14, no. 1, pp. 40-8, 2010.
12. M. B. Schippers et al., "Mapping the information flow from one brain to another during gestural communication," in Proc. Nat. Acad. Sci. USA, 2010, pp. 9388-9393.
13. S. Lu, P. Piang, Y. Yang, and K. Li, "Recruitment of the pre-motor area in human inductive reasoning: An fMRI study," Cogn. Syst. Res., vol. 11, no. 1, pp. 74-80, 2010.
14. E. Redcay et al., "Live face-to-face interaction during fMRI: A new tool for social cognitive neuroscience," Neuroimage, vol. 50, no. 4, pp. 1639-47, 2010.
15. A. F. Smeaton, P. Over, and W. Kraaij, "Evaluation campaigns and TRECVid," in Proc. ACM Int. Multimedia Conf. Exh., 2006, pp. 321-330. (Pubitemid 47168230)
16. G. K. Wallace, "The JPEG still picture compression standard," Commun. ACM, vol. 34, no. 4, pp. 30-44, 1991.
17. B. G. Haskell, A. Puri, and A. N. Netravali, Digital Video: An Introduction to MPEG-2. New York: Chapman-Hall, 1997.
18. N. Jayant, J. Johnston, and R. Safranek, "Signal compression based on models of human perception," Proc. IEEE, vol. 81, no. 10, pp. 1385-1422, Oct. 1993.
19. T. Liu, H. Zhang, and F. Qi, "Perceptual frame dropping in adaptive video streaming," in Proc. IEEE Int. Symp. Circuits Syst., 2002.
20. C.-H. Yeh, S.-M. Chen, and S.-J. Chern, "Content-aware video transcoding via visual attention model analysis," in Proc. 4th Int. Conf. Intell. Inf. Hiding and Multimedia Signal Process., 2008, pp. 429-432.
21. Y. Rui et al., "Relevance feedback: A power tool for interactive content-based image retrieval," IEEE Trans. Circuits Syst. Video Technol., vol. 8, no. 5, pp. 644-655, Oct. 1998. (Pubitemid 128743730)
22. J.Han, K. N. Ngan,M. Li, andH.-J. Zhang, "Amemory learning framework for effective image retrieval," IEEE Trans. Image Process., vol. 14, no. 4, pp. 511-524, Apr. 2005. (Pubitemid 40552383)
23. J. Wang, E. Pohlmeyer, B. Hanna, Y.-G. Jiang, P. Sajda, and S.-F. Chang, "Brain state decoding for rapid image retrieval," in Proc. ACM Multimedia Conf., 2009, pp. 945-954.
24. N. Bigdely-Shamlo, A. Vankov, R. R. Ramirez, and S. Makeig, "Brain activity-based image classification from rapid serial visual presentation," IEEE Trans. Neural Syst. Rehab. Eng., vol. 16, no. 5, pp. 432-441, 2008.
25. A. D. Gerson, L. C. Parra, and P. Sajda, "Cortically coupled computer vision for rapid image search," IEEE Trans. Neural Syst. Rehab. Eng., vol. 14, no. 2, pp. 174-179, 2006. (Pubitemid 44150235)
26. A. Kapoor, P. Shenoy, and D. Tan, "Combining brain computer interfaces with vision for object categorization," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2008, vol. 1-12, pp. 2150-57.
27. D. B. Walther, E. Caddigan, F.-F. Li, and D. M. Beck, "Natural scene categories revealed in distributed patterns of activity in the human brain," J. Neurosci., vol. 29, no. 34, pp. 10573-10581, 2009.
28. X. Hu, F. Deng, K. Li, T. Zhang, H. Chen, X. Jiang, J. Lv, D. Zhu, C. Faraco, D. Zhang, A.Meshah, J. Han, X. Hua, L. Xie, S. Miller, L. Guo, and T. Liu, "Bridging low-level features and high-level semantics via fMRI brain imaging for video classification," in Proc. ACM Multimedia Int. Conf., 2010, pp. 451-460.
29. S. L. Bressler and V.Menon, "Large-scale brain networks in cognition: Emerging methods and principles," Trends Cogn Sci, vol. 14, no. 6, pp. 277-90, 2010.
30. A.M.Martinez and A. C. Kak, "PCA versus LDA," IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 2, pp. 228-233, Feb. 2001. (Pubitemid 32254716)
31. X. F. He, S. C. Yan, Y. X. Hu, P. Niyogi, and H. J. Zhang, "Face recognition using Laplacian faces," IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 3, pp. 328-340, Mar. 2005. (Pubitemid 40377326)
32. J. B. Tenenbaum, V. de Silva, and J. C. Langford, "Aglobal geometric framework for nonlinear dimensionality reduction," Science, vol. 290, no. 5500, pp. 2319-2323, Dec. 2000. (Pubitemid 32041577)
33. P. Hart, "Nearest neighbor pattern clsssificaton," IEEE Trans. Inf. Theory, vol. IT-13, no. 1, pp. 21-27, Jan. 1967.
34. J. M. Leller, M. R. Gray, and J. A. Givens, "A fussy K-nearest neighbor algorithm," IEEE Trans. Syst. Man Cybern., vol. SMC-15, no. 4, pp. 580-584, 1985.
35. C. Cortes and V. Vapnik, "Support-vector network," Mach. Learning, no. 20, pp. 273-297, 1995.
36. Y. Dudai, "Enslaving central executives: Toward a brain theory of cinema," Projections, vol. 2, no. 2, pp. 12-24, 2008.
37. T. V. Sewards, "Neural structures and mechanisms involved in scene recognition: A review and interpretation," Neuropsychologia, vol. 49, no. 3, pp. 277-298, 2010.
38. C. Faraco, D. Smith, J. Langley, Q. Zhao, and L. Miller, "Mapping the working memory network using the OSPAN task," Neuroimage, vol. 47, no. 1, p. S105, 2009.
39. J. P. Hornak, The Basic of MRI. Rochester, NY: Rochester Inst. Technol., 2002.
40. B. A. Wandell, A. A. Brewer, and R. F. Dougherty, "Visual field map clusters in human cortex," Philos. Trans. Roy. Soc. London B Biol. Sci., vol. 360, no. 1456, pp. 693-707, 2005. (Pubitemid 40885660)
41. G. Femandez, A. de Greiff, J. von Oertzen, M. Reuber, S. Lun, P. Klaver, J. Ruhlmann, J. Reul, and C. E. Elger, "Language mapping in less than 15 minutes: Real-time functional MRI during routine clinical investigation," Neuroimage, vol. 14, no. 3, pp. 585-594, 2001. (Pubitemid 32801706)
42. S. M. Smith, M. Jenkinson, M. W. Woolrich, C. F. Beckmann, T. E. Behrens, H. Johansen-Berg, P. R. Bannister, M. De Luca, I. Drobnjak, D. E. Flitney, R. K. Niazy, J. Saunders, J. Vickers, Y. Zhang, N. De Stefano, J. M. Brady, and P. M. Matthews, "Advances in functional and structural MR image analysis and implementation as FSL," Neuroimage, vol. 23, no. Suppl 1, pp. S208-219, 2004. (Pubitemid 39421720)
43. U. Hasson, Y. Nir, I. Levy, G. Fuhrmann, and R. Malach, "Intersubject synchronization of cortical activity during natural vision," Science, vol. 303, no. 5664, pp. 1634-40, 2004. (Pubitemid 38338315)
44. [Online]. Available: http://www.pstnet.com
45. A. Amir, J. Argillander, and M. Campbell, "IBM research TRECVID-2005 video retrieval system," in Proc. NIST TRECVID Workshop, Gaithersburg, MD, 2005, pp. 1-20.
46. J. Huang, S. R. Kumar, M. Mitra, W. Zhu, and R. Zabih, "Image indexing using color correlograms," in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 2007, pp. 762-768.
47. J. R. Smith and S.-F. Chang, "Automated binary texture feature sets for image retrieval," in Proc. ICASSP, 1996, pp. 2239-2242.
48. T. D. Albright and G. R. Stoner, "Visual motion perception," Proc. Natl. Acad. Sci. USA, vol. 92, no. 7, pp. 2433-40, 1995.
49. F. Pasemann, "Complex dynamics and the structure of small neural networks," Network, vol. 13, no. 2, pp. 195-216, 2002.
50. H. Hotelling, "Relations between two sets of variates," Biometrika, no. 28, pp. 321-377, 1936.
51. O. Friman, J. Cedafamn, P. Lundberg, M. Borga, and H. Knutsson, "Detection of neural activity in functional MRI using canonical correlation analysis," Magn. Reson. Med., vol. 45, no. 2, pp. 323-30, 2001. (Pubitemid 32119016)
52. Q. S. Sun, S. G. Zeng, Y. Liu, P. A. Heng, and D. S. Xia, "A new method of feature fusion and its application in image recognition," Pattern Recognit., vol. 38, no. 12, pp. 2437-2448, 2005. (Pubitemid 41336713)
53. T. Barnett, "Origins and levels of monthly and seasonal forecast skill for united states surface air temperatures determined by canonical correlation analysis," Monthly Weather Rev., no. 115, pp. 1825-1850, 1987.
54. L. Fei-Fei and P. Perona, "A Bayesian hierarchical model for learning natural scene categories," in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 2005, vol. 2, pp. 524-531.
55. D. G. Lowe, "Object recognition from local scale-invariant features," in Proc. IEEE Int. Conf. Comput. Vis., 1999, pp. 1150-1157.
56. K. J. Friston, L. Harrison, andW. Penny, "Dynamic causal modelling," Neuroimage, vol. 19, no. 4, pp. 1273-302, 2003. (Pubitemid 37025831)
57. L. Itti and C. Koch, "A saliency-based search mechanism for overt and covert shifts of visual attention," Vis. Res., vol. 40, no. 10-12, pp. 1489-1506, 2000. (Pubitemid 30216462)