Using facial expressions and peripheral physiological signals as implicit indicators of topical relevance

MM'09 - Proceedings of the 2009 ACM Multimedia Conference, with Co-located Workshops and Symposiums

Volumn , Issue , 2009, Pages 461-470

  Arapakis, Ioannis ^a   Konstas, Ioannis ^a   Jose, Joemon M ^a  

^a UNIVERSITY OF GLASGOW   (United Kingdom)

Author keywords

Affective feedback; Facial expression analysis; Multimedia retrieval, classification; Pattern recognition; Physiological signal processing; Support vector machines

Indexed keywords

FACIAL EXPRESSION ANALYSIS; MULTIMEDIA RETRIEVAL; MULTIMEDIA RETRIEVAL, CLASSIFICATION; PHYSIOLOGICAL SIGNAL PROCESSING; PHYSIOLOGICAL SIGNALS;

FEATURE EXTRACTION; IMAGE RETRIEVAL; METHOD OF MOMENTS; MULTIMEDIA SIGNAL PROCESSING; SAFETY DEVICES; SEMANTICS; SIGNAL PROCESSING; SUPPORT VECTOR MACHINES; TECHNICAL PRESENTATIONS;

MULTIMEDIA SYSTEMS;

EID: 72449132811     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1631272.1631336     Document Type: Conference Paper

References (55)

1. E. Agichtein, E. Brill, and S. Dumais. Improving web search ranking by incorporating user behavior information. In SIGIR '06: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages 19-26. ACM, 2006.
2. D. Andre, R. Pelletier, J. Farringdon, S. Safier, W. Talbott, R. Stone, N. Vyas, J. Trimble, D. Wolf, S. Vishnubhatla, S. Boehmke, J. Stivoric, and A. Teller. The development of the sensewear® armband, a revolutionary energy assessment device to assess physical activity and lifestyle. Technical report, BodyMedia, Inc., 2006.
3. I. Arapakis, J. M. Jose, and P. D. Gray. Affective feedback: an investigation into the role of emotions in the information seeking process. In SIGIR '08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pages 395-402. ACM, 2008.
4. I. Arapakis, Y. Moshfeghi, H. Joho, R. Ren, D. Hannah, and J. M. Jose. Enriching user profiling with a affective features for the improvement of a multimodal recommender system. In Conference on Image and Video Retrieval, 2009.
5. R. Badi, S. Bae, J. M. Moore, K. Meintanis, A. Zacchi, H. Hsieh, F. Shipman, and C. C. Marshall. Recognizing user interest and document value from reading and organizing activities in document triage. In Proceedings of the 11th international conference on Intelligent User Interfaces, pages 218-225, New York, NY, USA, 2006. ACM.
6. P. Bailey, N. Craswell, and D. Hawking. Engineering a multi-purpose test collection for web retrieval experiments. Inf. Process. Manage., 39(6):853-871, 2003.
7. N. J. Belkin. Anomalous state of knowledge for information retrieval. Canadian Journal of Information Science, 5:133-143, 1980.
8. D. Bilal and J. Kirby. Differences and similarities in information seeking: children and adults as web users. Information Processing and Management: an International Journal, 38(5):649-670, 2002.
9. W. Boucsein. Electrodermal activity. Plenum Press, New York, 1992.
10. C. H. Chan and G. J. F. Jones. Affect-based indexing and retrieval of films. In MULTIMEDIA '05: Proceedings of the 13th annual ACM international conference on Multimedia, pages 427-430, New York, NY, USA, 2005. ACM.
11. G. Chanel, C. Rebetez, M. Betrancourt, and T. Pun. Boredom, engagement and anxiety as indicators for adaptation to difficulty in games. In MindTrek '08: Proceedings of the 12th international conference on Entertainment and media in the ubiquitous era, pages 13-17, New York, NY, USA, 2008. ACM.
12. R. R. Cornelius. Theoretical approaches to emotion. Proc ISCA: workshop on Speech and Emotion, pages 3-11, 2000.
13. W. Daelemans and A. van den Bosch. Memory-Based Language Processing (Studies in Natural Language Processing). Cambridge University Press, New York, NY, USA, 2005.
14. A. R. Damasio. Descartes Error: Emotion, Reason, and the Human Brain. Putnam/Grosset Press, 1994.
15. P. Ekman. Facial Expressions. Handbook of Cognition and Emotion. John Wiley & Sons Ltd., New York, 1999.
16. P. Ekman. Emotions Revealed: Recognizing Faces and Feelings to Improve Communication and Emotional Life. Times Books, New York, 2003.
17. P. Ekman. Unmasking the face. MA: Malor books, Cambridge, 2003.
18. P. Ekman and H. Oster. Facial expressions of emotion. Annual Review of Psychology, 30(1):527-554, 1979.
19. A. Hanjalic and L.-Q. Xu. Affective video content representation and modeling. Multimedia, IEEE Transactions on, 7(1):143-154, 2005.
20. F. Hopfgartner. A news video retrieval framework for the study of implicit relevance feedback. In Proceedings of the Second International Workshop on Semantic Media Adaptation and Personalization, pages 233-236, Washington, DC, USA, 2007. IEEE Computer Society.
21. http://www.polarusa.com.
22. H. Kataoka, H. Kano, H. Yoshida, A. Saijo, M. Yasuda, and M. Osumi. Delelopment of a skin temperature measuring system for non-contact stress evaluation. In Proceedings of the 20th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, volume 2, pages 940-943, 1998.
23. J. Koenemann and N. J. Belkin. A case for interaction: a study of interactive information retrieval behavior and effectiveness. In CHI '96: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 205-212, New York, NY, USA, 1996. ACM.
24. J. Kracker. Research anxiety and students' perceptions of research: an experiment. part i: Effect of teaching kuhlthau's isp model. Journal of the American Society for Information Science and Technology, 53(4):282-294, 2002.
25. I. Lopatovska and B. Mokros, H. Willingness to pay and experienced utility as measures of affective value of information objects: Users' accounts. Information Processing and Management: an International Journal, 44(1):92-104, 2008.
26. M. Lynne, R. C. Sheldrick, and R. Paulette. Affective dimensions of information seeking in the context of reading. Medford, NJ: Information Today, 2007.
27. C. Mooney, M. Scully, G. J. Jones, and A. F. Smeaton. Investigating Biometric Response for Information Retrieval Application, volume 3936/2006 of Lecture Notes in Computer Science. Springer Berlin, 2006.
28. D. Nahl. Learning the internet and the structure of information behavior. Journal of the American Society for Information Science, 49(11):1017-1023, 1998.
29. D. Nahl. Measuring the affective information environment of web searchers. In Proceedings of the American Society for Information Science and Technology, volume 41, pages 191-197, 2004.
30. D. Nahl and C. Tenopir. Affective and cognitive searching behavior of novice end-users of a full-text database. Journal of the American Society for Information Science, 47(4):276-286, 1996.
31. M. Pantic and L. Rothkrantz. Expert system for automatic analysis of facial expression. Image and Vision Computing Journal, 18(11):881-905, July 2000.
32. M. Pantic and L. J. M. Rothkrantz. Toward an affect-sensitive multimodal human-computer interaction. In Proceedings of the IEEE, pages 1370-1390, 2003.
33. M. Pantic, N. Sebe, J. F. Cohn, and T. Huang. Affective multimodal human-computer interaction. In MULTIMEDIA '05: Proceedings of the 13th annual ACM international conference on Multimedia, pages 669-676, New York, NY, USA, 2005. ACM.
34. H.-R. Pfister and G. Bohm. The multiplicity of emotions: A framework of emotional functions in decision making. Judgment and Decision Making, 3:5-17, 2008.
35. R. W. Picard, A. Wexelblat, and C. I. N. I. Clifford I. Nass. Future interfaces: social and emotional. In CHI '02 extended abstracts on Human factors in computing systems, pages 698-699, 2002.
36. K. Puolamaki, J. Salojarvi, E. Savia, J. Simola, and S. Kaski. Combining eye movements and collaborative filtering for proactive information retrieval. In SIGIR '05: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pages 146-153, New York, NY, USA, 2005. ACM.
37. B. Reeves and C. Nass. The media equation: how people treat computers, television, and new media like real people and places. Cambridge University Press, New York, NY, USA, 1996.
38. Y. Rui and S. Huang, T. Optimizing learning in image retrieval. In IEEE Proceedings of Conference on Computer Vision, volume 1, pages 236-243, 2000.
39. A. Russell, J., J. Bachorowski, and J. Fernandez-Dols. Facial and vocal expressions of emotion. Annual Review of Psychology, 2003.
40. J. A. Russell and J. H. Steiger. The Structure in Person's Implicit Taxonomy of Emotions. Journal of Research in Personality, 1982.
41. J. Salojärvi, K. Puolamaki, and S. Kaski. Implicit Relevance Feedback from Eye Movements, volume Volume 3696/2005 of Artificial Neural Networks: Biological Inspirations - ICANN 2005. Springer Berlin, 2005.
42. T. Saracevic. Relevance reconsidered. information science: Integration in perspectives. Proceedings of the Second Conference on Conceptions of Library and Information Science, Copenhagen, Denmark, pages 201-218, 1996.
43. K. R. Scherer. Appraisal considered as a process of multi-level sequential checking. Appraisal processes in emotion: Theory, Methods, Research. Oxford University Press, New York and Oxford, k. r. scherer, a. schorr, & t. johnstone (eds.) edition, 2001.
44. K. R. Scherer. What are emotions? and how can they be measured? Social Science Information, 44(4):695-729, December 2005.
45. A. F. Smeaton, P. Over, and W. Kraaij. Evaluation campaigns and trecvid. In MIR '06: Proceedings of the 8th ACM International Workshop on Multimedia Information Retrieval, pages 321-330, New York, NY, USA, 2007. ACM Press.
46. A. F. Smeaton and S. Rothwell. Biometric responses to music-rich segments in films: The cdvplex. In Seventh International Workshop on Content-Based Multimedia Indexing, pages 162-168, 2009.
47. M. Smeulders, A., M. Worring, S. Santini, A. Gupta, and R. Jain. Content-based image retrieval at the end of the early years. In IEEE Transactions on Pattern Analysis and Machine Intelligence, volume 22, pages 1349-1380. IEEE Computer Society, 2000.
48. M. Soleymani, G. Chanel, J. J. Kierkels, and T. Pun. Affective ranking of movie scenes using physiological signals and content analysis. In MS '08: Proceeding of the 2nd ACM workshop on Multimedia semantics, pages 32-39, New York, NY, USA, 2008. ACM.
49. C. Tenopir, P. Wang, Y. Zhang, B. Simmons, and R. Pollard. Academic users' interactions with sciencedirect in search tasks: Affective and cognitive behaviors. Information Processing and Management: an International Journal, 44(1):105-121, 2008.
50. J. Urban and J. Jose, M. Evaluating a workspace's usefulness for image retrieval. Journal of Multimedia Systems, 12(4-5):355-373, 2007.
51. R. Valenti, N. Sebe, and T. Gevers. Facial expression recognition: A fully integrated approach. Image Analysis and Processing Workshops, 2007. ICIAPW 2007. 14th International Conference on, pages 125-130, Sept. 2007.
52. A. Vinciarelli, N. Suditu, and M. Pantic. Implicit human-centred tagging. In Proceedings of IEEE International Conference on Multimedia and Expo, 2009.
53. S. Vrochidis, P. King, L. Makris, A. Moumtzidou, V. Mezaris, and I. Kompatsiaris. Mklab interactive video retrieval system. In CIVR '08: Proceedings of the 2008 international conference on Content-based image and video retrieval, pages 563-564. ACM, 2008.
54. P. Wang, B. Hawk, W., and C. Tenopir. Users' interaction with world wide web resources: an exploratory study using a holistic approach. Information Processing and Management: an International Journal, 36(2):229-251, 2000.
55. G. M. Wilson and M. A. Sasse. Listen to your heart rate: counting the cost of media quality. pages 9-20, 2000.