Learning Computational Models of Video Memorability from fMRI Brain Imaging

IEEE Transactions on Cybernetics

Volumn 45, Issue 8, 2015, Pages 1692-1703

  Han, Jungong ^b   Han, Junwei ^a   Chen, Changyuan ^a   Shao, Ling ^c   Hu, Xintao ^a   Liu, Tianming ^d  

^a NORTHWESTERN POLYTECHNICAL UNIVERSITY   (China)

^b Civolution   (Netherlands)

^c NORTHUMBRIA UNIVERSITY   (United Kingdom)

^d University of Georgia ^*  (United States)

Author keywords

Audiovisual features; brain imaging; semantic gap; video memorability (VM)

Indexed keywords

BRAIN; BRAIN MAPPING; COMPUTATION THEORY; COMPUTATIONAL METHODS; DATABASE SYSTEMS; MAGNETIC RESONANCE IMAGING; NEUROIMAGING; NEUROPHYSIOLOGY; SEMANTICS; VIDEO CAMERAS;

AUDIO-VISUAL FEATURES; BENCHMARK DATASETS; BRAIN IMAGING; COMPUTATIONAL FRAMEWORK; COMPUTATIONAL MODEL; FUNCTIONAL MAGNETIC RESONANCE IMAGING; SEMANTIC GAP; VIDEO MEMORABILITY (VM);

FUNCTIONAL NEUROIMAGING;

BIOLOGICAL MODEL; BRAIN; COMPUTER SIMULATION; HUMAN; LEARNING; MEMORY; NEUROIMAGING; NUCLEAR MAGNETIC RESONANCE IMAGING; PHYSIOLOGY; PROCEDURES; SEMANTICS; VIDEORECORDING;

BRAIN; COMPUTER SIMULATION; HUMANS; LEARNING; MAGNETIC RESONANCE IMAGING; MEMORY; MODELS, NEUROLOGICAL; NEUROIMAGING; SEMANTICS; VIDEO RECORDING;

EID: 84926497889     PISSN: 21682267     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCYB.2014.2358647     Document Type: Article

References (40)

1. T. Konkle, T. F. Brady, G. A. Alvarez, and A. Oliva, "Scene memory is more detailed than you think: The role of categories in visual long-term memory," Psychol. Sci., vol. 21, no. 11, pp. 1551-1556, 2010.
2. O. Furman, N. Dorfman, U. Hasson, L. Davachi, and Y. Dudai, "They saw a movie: Long-term memory for an extended audiovisual narrative," Learn. Memory, vol. 14, no. 6, pp. 457-467, 2007.
3. Q. Wang, Y. Yuan, P. Yan, and X. Li, "Saliency detection by multipleinstance learning," IEEE Trans. Cybern., vol. 43, no. 3, pp. 660-672, Apr. 2013.
4. Z. Yücel et al., "Joint attention by gaze interpolation and saliency," IEEE Trans. Cybern., vol. 43, no. 2, pp. 829-842, Jun. 2013.
5. P. Isola, D. Parikh, A. Torralba, and A. Oliva, "Understanding the intrinsic memorability of images," in Proc. Conf. Adv. Neural Inf. Process. Syst., 2011, pp. 2429-2437.
6. P. Isola, J. Xiao, D. Parikh, A. Torralba, and A. Oliva, "What makes a photograph memorable?" IEEE Trans. Pattern Anal. Mach. Intell., vol. 36, no. 7, pp. 1469-1482, Jul. 2014.
7. A. Khosla, J. Xiao, A. Torralba, and A. Oliva, "Memorability of image regions," in Proc. Conf. Adv. Neural Inf. Process. Syst., 2012, pp. 296-304.
8. S. Dhar, V. Ordonez, and T. L. Berg, "High level describable attributes for predicting aesthetics and interestingness," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., Providence, RI, USA, 2011, pp. 1657-1664.
9. T. Barnett and R. Preisendorfer, "Origins and levels of monthly and seasonal forecast skill for United States surface air temperatures determined by canonical correlation analysis," Mon. Weather Rev., vol. 115, no. 9, pp. 1825-1850, 1987.
10. Y. Ke, X. Tang, and F. Jing, "The design of high-level features for photo quality assessment," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., Washington, DC, USA, 2006, pp. 419-426.
11. A. K. Moorthy, P. Obrador, and N. Oliver, "Towards computational models of the visual aesthetic appeal of consumer videos," in Proc. Eur. Conf. Comput. Vis., Heraklion, Greece, 2010, pp. 1-14.
12. L.-J. Li, H. Su, Y. Lim, and L. Fei-Fei, "Objects as attributes for scene classification," in Proc. Eur. Conf. Comput. Vis., Heraklion, Greece, 2010, pp. 57-69.
13. J. Tilke, K. Ehinger, F. Durand, and A. Torralba, "Learning to predict where humans look," in Proc. IEEE Int. Conf. Comput. Vis., Kyoto, Japan, 2009, pp. 2106-2113.
14. P. Dollar, V. Rabaud, G. Cottrell, and S. Belongie, "Behavior recognition via sparse spatio-temporal features," in Proc. IEEE Int. Workshop Vis. Surveillance Perform. Eval. Tracking Surveillance, 2005, pp. 65-72.
15. Q.-S. Sun, S.-G. Zeng, Y. Liu, P.-A. Heng, and D.-S. Xia, "A new method of feature fusion and its application in image recognition," Pattern Recognit., vol. 38, pp. 2437-2448, Dec. 2005.
16. S. B. Davis and P. Mermelstein, "Comparison of parametric representations for nonsyllabic word recognition in continuously spoken sentences," IEEE Trans. Acoust., Speech, Signal Process., vol. 28, no. 4, pp. 357-366, Aug. 1980.
17. G. C. Cawley and N. L. Talbot, "Gene selection in cancer classification using sparse logistic regression with Bayesian regularization," Bioinformatics, vol. 22, no. 19, pp. 2348-2355, 2006.
18. J. Wang et al., "Brain state decoding for rapid image retrieval," in Proc. ACM Int. Conf. Multimedia, New York, NY, USA, 2009, pp. 945-954.
19. A. Kapoor, P. Shenoy, and D. Tan, "Combining brain computer interfaces with vision for object categorization," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., Anchorage, AK, USA, 2008, pp. 1-8.
20. X. Hu et al., "Bridging low-level features and high-level semantics via fMRI brain imaging for video classification," in Proc. ACM Int. Conf. Multimedia, Firenze, Italy, 2010, pp. 451-460.
21. U. Hasson, O. Furman, D. Clark, Y. Dudai, and L. Davachi, "Enhanced intersubject correlations during movie viewing correlate with successful episodic encoding," Neuron, vol. 57, no. 3, pp. 452-462, 2008.
22. U. Hasson, Y. Nir, I. Levy, G. Fuhrmann, and R. Malach, "Intersubject synchronization of cortical activity during natural vision," Science, vol. 303, no. 5664, pp. 1634-1640, 2004.
23. U. Hasson et al., "Neurocinematics: The neuroscience of film," Projections, vol. 2, no. 1, pp. 1-26, 2008.
24. D. Zhu et al., "DICCCOL: Dense individualized and common connectivity-based cortical landmarks," Cereb. Cortex, vol. 23, pp. 786-800, Apr. 2013.
25. M. Mancas and O. Le Meur, "Memorability of natural scenes: The role of attention," in Proc. IEEE Int. Conf. Image Process., Melbourne, VIC, Australia, 2013, pp. 196-200.
26. J. S. Pedro and S. Siersdorfer, "Ranking and classifying attractiveness of photos in folksonomies," in Proc. Int. Conf. World Wide Web, Madrid, Spain, 2009, pp. 771-780.
27. Y. Luo and X. Tang, "Photo and video quality evaluation: Focusing on the subject," in Proc. Eur. Conf. Comput. Vis., Marseille, France, 2008, pp. 386-399.
28. S. S. Cheung and A. Zakhor, "Efficient video similarity measurement and search," in Proc. IEEE Int. Conf. Image Process., 2000, pp. 85-88.
29. O. Lartillot and P. Toiviainen, "A MATLAB toolbox for musical feature extraction from audio," in Proc. Int. Conf. Digit. Audio Effects, Bordeaux, France, 2007, pp. 237-244.
30. Y.-F. Ma, L. Lu, H.-J. Zhang, and M. Li, "A user attention model for video summarization," in Proc. ACM Int. Conf. Multimedia, Juan-les-Pins, France, 2002, pp. 533-542.
31. A. Farhadi, I. Endres, D. Hoiem, and D. Forsyth, "Describing objects by their attributes," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., Miami, FL, USA, 2009, pp. 1778-1785.
32. J. Han et al., "Representing and retrieving video shots in human-centric brain imaging space," IEEE Trans. Image Process., vol. 22, no. 7, pp. 2723-2736, Jul. 2013.
33. O. L. Meur, T. Baccino, and A. Roumy, "Prediction of the inter-observer visual congruency (IOVC) and application to image ranking," in Proc. ACM Int. Conf. Multimedia, El Paso, TX, USA, 2011, pp. 373-382.
34. M. S. Lew, N. Sebe, C. Djeraba, and R. Jain, "Content-based multimedia information retrieval: State of the art and challenges," ACM Trans. Multimedia Comput., Commun. Appl., vol. 2, pp. 1-19, Feb. 2006.
35. B. Celikkale, A. Erdem, and E. Erdem, "Visual attention-driven spatial pooling for image memorability," in Proc. IEEE Workshop Int. Conf. Comput. Vis. Pattern Recognit., Portland, OR, USA, 2013, pp. 976-983.
36. J. Kim, S. Yoon, and V. Pavlovic, "Relative spatial features for image memorability," in Proc. ACM Int. Conf. Multimedia, Barcelona, Spain, 2013, pp. 761-764.
37. (2005). TRECVID Database [Online]. Available: http://www-nlpir. nist.gov/projects/tv2005/
38. C. Chang and C. Lin, "LIBSVM: A library for support vector machines," ACM Trans. Intell. Syst. Technol., vol. 2, no. 27, pp. 1-27, 2011.
39. A. Khosla, W. Bainbridge, A. Torralba, and A. Oliva, "Modifying the memorability of face photographs," in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 3200-3207.
40. A. Khosla, J. Xiao, P. Isola, A. Torralba, and A. Oliva, "Image memorability and visual inception," in Proc. SIGGRAPH Asia, New York, NY, USA, 2012, pp. 1-4.