Video abstraction based on fMRI-driven visual attention model

Information Sciences

Volumn 281, Issue , 2014, Pages 781-796

  Li, Kaiming ^a   Shao, Ling ^d   Hu, Xintao ^a   He, Sheng ^a   Guo, Lei ^a   Han, Jungong ^c   Liu, Tianming ^b   Han, Junwei ^a  

^a NORTHWESTERN POLYTECHNICAL UNIVERSITY   (China)

^b Department of Computer Science and Bioimaging Research Center University of Georgia ^*  (United States)

^c Civolution   (Netherlands)

^d UNIVERSITY OF SHEFFIELD   (United Kingdom)

Author keywords

Bayesian surprise model; Functional magnetic resonance imaging; Propensity for synchronization; Video abstraction; Visual attention

Indexed keywords

ABSTRACTING; BENCHMARKING; BRAIN; CURVE FITTING; GRAPH THEORY; MAGNETIC RESONANCE IMAGING; VIDEO RECORDING; VIDEO STREAMING;

BAYESIAN; DIGITAL VIDEO DATA; FUNCTIONAL MAGNETIC RESONANCE IMAGING; MULTIMEDIA CONTENTS; SPECTRAL GRAPH THEORY; VIDEO ABSTRACTION; VISUAL ATTENTION; VISUAL ATTENTION MODEL;

FUNCTIONAL NEUROIMAGING;

EID: 84904761891     PISSN: 00200255     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.ins.2013.12.039     Document Type: Article

References (63)

1. R. Aler, I. Galván, and J. Valls Applying evolution strategies to preprocessing EEG signals for brain-computer interfaces Inform. Sci. 215 2012 53 66
2. A. Baddeley Working Memory, Thought, and Action 2007 Oxford University Press
3. P. Baldi, and L. Itti Of bits and wows: a Bayesian theory of surprise with applications to attention Neural Networks 23 2010 649 666
4. L. Bo, and C. Sminchisescu Twin Gaussian processes for structured prediction Int. J. Comput. Vision 87 2010 28 52
5. S. Boccaletti, V. Latora, Y. Moreno, M. Chavez, and D.U. Hwang Complex networks: structure and dynamics Phys. Rep. 424 2006 175 308
6. B.W. Chen, J.C. Wang, and J.F. Wang A novel video summarization based on mining the story-structure and semantic relations among concept entities IEEE Trans. Multimedia 11 2009 295 312
7. F.R.K. Chung Spectral Graph Theory 1997 American Mathematical Society
8. Y. Dudai Enslaving central executives: toward a brain theory of cinema Projections 2 2008 21 42
9. G. Evangelopoulos, A. Zlatintsi, G. Skoumas, K. Rapantzikos, A. Potamianos, P. Maragos, Y. Avrithis, Video event detection and summarization using audio, visual and text saliency, in: Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing, ICASSP'09, Taipei, Taiwan, 2009, pp. 3553-3556.
10. K.J. Friston Modalities, modes, and models in functional neuroimaging Science 326 2009 399 403
11. K.J. Friston, A.P. Holmes, K.J. Worsley, J.P. Poline, C.D. Frith, and R.S.J. Frackowiak Statistical parametric maps in functional imaging: a general linear approach Hum. Brain Mapp. 2 1994 189 210
12. N. Guan, D. Tao, Z. Luo, and B. Yuan Online non-negative matrix factorization with robust stochastic approximation IEEE Trans. Neural Networks Learn. Syst. 23 2012 1087 1099
13. N. Guan, D. Tao, Z. Luo, and B. Yuan NeNMF: an optimal gradient method for non-negative matrix factorization IEEE Trans. Signal Process. 60 2012 2882 2898
14. J. Han Object segmentation from consumer videos: a unified framework based on visual attention IEEE Trans. Consum. Electron. 55 2009 1597 1605
15. J. Han, D. Farin, and P. de With Broadcast court-net sports video analysis using fast 3-D camera modeling IEEE Trans. Circuits Syst. Video Technol. 18 2008 1628 1638
16. J. Han, X. Ji, X. Hu, D. Zhu, K. Li, X. Jiang, G. Cui, L. Guo, and T. Liu Representing and retrieving video shots in human-centric brain imaging space IEEE Trans. Image Process. 22 2013 2723 2736
17. J. Han, K.N. Ngan, M. Li, and H.J. Zhang Unsupervised extraction of visual attention objects in color images IEEE Trans. Circuits Syst. Video Technol. 16 2006 141 145
18. J. Han, E. Pauwels, and P. Zeeuw Fast saliency-aware multi-modality image fusion Neurocomputing 111 2013 70 80
19. U. Hasson, R. Malach, and D.J. Heeger Reliability of cortical activity during natural stimulation Trends Cogn. Sci. 14 2010 40 48
20. U. Hasson, Y. Nir, I. Levy, G. Fuhrmann, and R. Malach Intersubject synchronization of cortical activity during natural vision Science 303 2004 1634 1640
21. M.T. Horstmann, S. Bialonski, N. Noennig, H. Mai, J. Prusseit, J. Wellmer, H. Hinrichs, and K. Lehnertz State dependent properties of epileptic brain networks: comparative graph-theoretical analyses of simultaneously recorded EEG and MEG Clin. Neurophysiol. 121 2010 172 185
22. X. Hu, L. Guo, D. Zhang, K. Li, T. Zhang, J. Lv, J. Han, T. Liu, Assessing the dynamics on functional brain networks using spectral graphy theory, in: Proceedings of the 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro, ISBI'11, Chicago, Illinois, USA, 2011, pp. 2144-2149.
23. X. Hu, K. Li, J. Han, X. Hua, L. Guo, and T. Liu Bridging the semantic gap via functional brain imaging IEEE Trans. Multimedia 14 2012 314 325
24. L. Itti, C. Koch, and E. Niebur A model of saliency-based visual attention for rapid scene analysis IEEE Trans. Pattern Anal. Mach. Intell. 20 1998 1254 1259
25. X. Ji, J. Han, X. Hu, K. Li, F. Deng, J. Fang, L. Guo, T. Liu, Retrieving video shots in semantic brain imaging space using manifold-ranking, in: Proc. 18th IEEE Int. Conf. on Image Processing, ICIP'11, Brussels, Belgium, 2011, pp. 3633-3636.
26. P. Jiang, and X. Qin Keyframe-based video summary using visual attention clues IEEE Multimedia 17 2010 64 73
27. S. Jones, and L. Shao Content-based retrieval of human actions from realistic video databases Inform. Sci. 236 2013 56 65
28. T. Judd, K. Ehinger, F. Durand, A. Torralba, Learning to predict where humans look, in: Proc. 12th Int. Conf. on Computer Vision, ICCV'12, Kyoto, Japan, 2009, pp. 2106-2113.
29. K. Li et al., Human-centered attention models for video summarization, in: Proc. 12th Int. Conf. on Multimodal Interfaces and 7th Workshop on Machine Learning for Multimodal Interaction, ICMI-MLMI'10, Beijing, China, 2010.
30. Z. Li, G.M. Schuster, and A.K. Katsaggelos MINMAX optimal video summarization IEEE Trans. Circuits Syst. Video Technol. 15 2005 1245 1256
31. T. Liu, Z. Yuan, J. Sun, J. Wang, N. Zheng, X. Tang, and H.Y. Shum Learning to detect a salient object IEEE Trans. Pattern Anal. Mach. Intell. 33 2011 353 367
32. N.K. Logothetis What we can do and what we cannot do with fMRI Nature 453 2008 869 878
33. G. Lohmann, K. Volz, and M. Ullsperger Using non-negative matrix factorization for single-trial analysis of fMRI data NeuroImage 37 2007 1148 1160
34. J. Luo, C. Papin, and K. Costello Towards extracting semantically meaningful key frames from personal video clips: from humans to computers IEEE Trans. Circuits Syst. Video Technol. 19 2009 289 301
35. Y.F. Ma, X.S. Hua, L. Lu, and H.J. Zhang A generic framework of user attention model and its application in video summarization IEEE Trans. Multimedia 7 2005 907 919
36. Y.F. Ma, H.J. Zhang, Contrast-based image attention analysis by using fuzzy growing, in: Proc. 11th ACM Int. Conf. on Multimedia, MM'03, Berkeley, CA, USA, 2003, pp. 374-381.
37. S. Marat, M. Guironnet, D. Pellerin, Video summarization using a visual attention model, in: Proc. 15th European Signal Processing Conference, EUSIPCO'07, Poznań, Poland, 2007.
38. S. Marat, T.H. Phuoc, L. Granjon, N. Guyader, D. Pellerin, and A. Guérin-Dugué Modelling spatio-temporal saliency to predict gaze direction for short videos Int. J. Comput. Vision 82 2009 231 243
39. J.D. Meier, T.N. Aflalo, S. Kastner, and M.S.A. Graziano Complex organization of human primary motor cortex: a high-resolution fMRI study J. Neurophysiol. 100 2008 1800 1812
40. O.L. Meur, T. Baccino, A. Roumy, Prediction of the Inter-Observer Visual Congruency (IOVC) and application to image ranking, in: Proc. 19th ACM Int. Conf. on Multimedia, MM'11, Scottsdale, AZ, USA, 2011, pp. 373-382.
41. T.M. Mitchell, S.V. Shinkareva, A. Carlson, K.M. Chang, V.L. Malave, R.A. Mason, and M.A. Just Predicting human brain activity associated with the meanings of nouns Science 320 2008 1191 1195
42. M.R. Naphade, L. Kennedy, J.R. Kender, S. Chang, J.R. Smith, P. Over, A. Hauptmann, Light Scale Concept Ontology for Multimedia understanding for TRECVID 2005. < http://www-nlpir.nist.gov/projects/tv2005/LSCOMlite-NKKCSOH.pdf >.
43. C.W. Ngo, Y.F. Ma, and H.J. Zhang Video summarization and scene detection by graph modeling IEEE Trans. Circuits Syst. Video Technol. 15 2005 296 305
44. D. Pedronette, and R. Torres Exploiting pairwise recommendation and clustering strategies for image re-ranking Inform. Sci. 207 2012 19 34
45. J.C. Principe, D. Xu, Q. Zhao, and J.W. Fisher Learning from examples with information theoretic criteria J. VLSI Signal Process. 26 2000 61 77
46. C.E. Rasmussen, and C.K.I. Williams Gaussian Processes for Machine Learning 2006 MIT Press
47. J. Ren, and J. Jiang Hierarchical modeling and adaptive clustering for real-time summarization of rush videos IEEE Trans. Multimedia 11 2009 906 917
48. H.J. Seo, and P. Milanfar Static and space-time visual saliency detection by self-resemblance J. Vision 9 2009 1 27
49. L. Shao, and M. Brady Invariant salient regions based image retrieval under viewpoint and illumination variations J. Vis. Commun. Image Represent. 17 2006 1256 1272
50. L. Shao, and M. Brady Specific object retrieval based on salient regions Pattern Recogn. 39 2006 1932 1948
51. L. Shao, T. Kadir, and M. Brady Geometric and photometric invariant distinctive regions detection Inform. Sci. 177 2007 1088 1122
52. N. Shroff, P. Turaga, and R. Chellappa Video précis: highlighting diverse aspects of videos IEEE Trans. Multimedia 12 2010 853 868
53. A.F. Smeaton, P. Over, W. Kraaij, Evaluation campaigns and TRECVid, in: Proc. 8th ACM Int. Workshop on Multimedia Information Retrieval, MIR'06, Santa Barbara, California, USA, 2006, pp. 321-330.
54. M. Tkalcic, A. Odic, and A. Kosir The impact of weak ground truth and facial expressiveness on affect detection accuracy from time-continuous videos of facial expressions Inform. Sci. 249 2013 13 23
55. B.T. Truong, and S. Venkatesh Video abstraction: a systematic review and classification ACM Trans. Multimedia Comput. Commun. Appl. 3 2007 1 37
56. D.B. Walther, E. Caddigan, L. Fei-Fei, and D.M. Beck Natural scene categories revealed in distributed patterns of activity in the human brain J. Neurosci. 29 2009 10573 10581
57. F. Wang, and C. Ngo Summarizing rushes videos by motion object, and event understanding IEEE Trans. Multimedia 14 2012 76 87
58. J. Wang, E. Pohlmeyer, B. Hanna, Y.G. Jiang, P. Sajda, S.F. Chang, Brain state decoding for rapid image retrieval, in: Proc. 17th ACM Int. Conf. on Multimedia, MM'09, Vancouver, British Columbia, Canada, 2009, pp. 945-954.
59. R. Wang, S.J. McKenna, J. Han, and A.A. Ward Visualizing image collections using high-entropy layout distributions IEEE Trans. Multimedia 12 2010 803 813
60. B. Xuan, D. Zhang, S. He, and X. Chen Larger stimuli are judged to last longer J. Vision 7 2007 1 5
61. J. You, G. Liu, L. Sun, and H. Li A multiple visual models based perceptive analysis framework for multilevel video summarization IEEE Trans. Circuits Syst. Video Technol. 17 2007 273 285
62. Y. Zhai, M. Shah, Visual attention detection in video sequences using spatiotemporal cues, in: Proc. 14th Annual ACM Int. Conf. on Multimedia, MM'06, Santa Barbara, CA, USA, 2006, pp. 815-824.
63. H.J. Zhang, J. Wu, D. Zhong, and S.W. Smoliar An integrated system for content-based video retrieval and browsing Pattern Recogn. 30 1997 643 658