Dynamic information for the recognition of conversational expressions

Journal of Vision

Volumn 9, Issue 13, 2009, Pages 1-17

  Cunningham, Douglas W ^a,b   Wallraven, Christian ^a,c  

^a MAX PLANCK INSTITUTE FOR BIOLOGICAL CYBERNETICS   (Germany)

^b BRANDENBURG UNIVERSITY OF TECHNOLOGY   (Germany)

^c Korea University   (South Korea)

Author keywords

Communication; Dynamic information; Facial expressions; Perception

Indexed keywords

ARTICLE; COMPARATIVE STUDY; FACIAL EXPRESSION; FEMALE; HUMAN; INTERPERSONAL COMMUNICATION; MALE; PATTERN RECOGNITION; PHYSIOLOGY; REFERENCE VALUE; VERBAL BEHAVIOR; VIDEORECORDING;

COMMUNICATION; FACIAL EXPRESSION; FEMALE; HUMANS; MALE; PATTERN RECOGNITION, VISUAL; REFERENCE VALUES; VERBAL BEHAVIOR; VIDEO RECORDING;

EID: 77950361833     PISSN: None     EISSN: 15347362     Source Type: Journal    
DOI: 10.1167/9.13.1     Document Type: Article

References (81)

1. Adolphs, R., Tranel, D., & Damasio, A. R. (2003). Dissociable neural systems for recognizing emotions. Brain and Cognition, 52, 61-69.
2. Ambadar, Z., Schooler, J. W., & Cohn, J. (2005). Deciphering the enigmatic face: The importance of facial dynamics in interpreting subtle facial expressions. Psychological Science, 16, 403-410.
3. Anaki, D., Boyd, J., & Moscovitch, M. (2007). Temporal integration in face perception: Evidence of configural processing of temporally separated face parts. Journal of Experimental Psychology: Human Perception and Performance, 22, 1-19.
4. Averbach, E., & Coriell, A. S. (1961). Short-term memory in vision. Bell System Technical Journal, 40, 309-328.
5. Baron-Cohen, S., Wheelwright, S., & Jolliffe, T. (1997). Is there a "language of the eyes"? Evidence from normal adults and adults with autism or Asperger syndrome. Visual Cognition, 4, 311-331.
6. Bassili, J. (1978). Facial motion in the perception of faces and of emotional expression. Journal of Experimental Psychology, 4, 373-379.
7. Bassili, J. (1979). Emotion recognition: The role of facial motion and the relative importance of upper and lower areas of the face. Journal of Personality and Social Psychology, 37, 2049-2059.
8. Bavelas, J. B., Black, A., Lemery, C. R., & Mullett, J. (1986). I show how you feelVMotor mimicry as a communicative act. Journal of Personality and Social Psychology, 59, 322-329.
9. Bavelas, J. B., Coates, L., & Johnson, T. (2000). Listeners as conarrators. Journal of Personality and Social Psychology, 79, 941-952.
10. Becker, M., & Pashler, H. (2002). Volatile visual representations: Failing to detect changes in recently processed information. Psychonomic Bulletin and Review, 9, 744-750.
11. Beintema, J., & Lappe, M. (2002). Perception of biological motion without local image motion. Proceedings of the National Academy of Sciences of the United States of America, 99, 5661-5663.
12. Bloch, A. M. (1885). Experiences sur la vision. C. R. Seances Society of Biology Paris, 37, 493-495.
13. Boyle, E., Anderson, A., & Newlands, A. (1994). The effects of visibility on dialogue in a cooperative problem solving task. Language and Speech, 37, 1-20.
14. Bull, P. (2001). State of the art: Nonverbal communication. The Psychologist, 14, 644-647.
15. Carrera-Levillain, P., & Fernandez-Dols, J. (1994). Neutral faces in context: Their emotional meaning and their function. Journal of Nonverbal Behavior, 18, 281-299.
16. Casile, A., & Giese, M. (2005). Critical features for the recognition of biological motion. Journal of Vision, 5(4):6, 348-360, http://journalofvision.org/5/4/6/, doi:10.1167/5.4.6.
17. Cassell, J., Bickmore, T., Cambell, L., Vilhjalmsson, H., & Yan, H. (2001). More than just a pretty face: Conversational protocols and the affordances of embodiment. Knowledge-Based Systems, 14, 22-64.
18. Cassell, J., & Thorisson, K. R. (1999). The power of a nod and a glance: Envelope vs. emotional feedback in animated conversational agents. Applied Artificial Intelligence, 13, 519-538.
19. Chuang, L., Vuong, Q. C., Thornton, I. M., & Bülthoff, H. H. (2006). Recognising novel deforming objects. Visual Cognition, 14, 85-88.
20. Cohn, J. F., & Kanade, T. (2007). Automated facial image analysis for measurement of emotion expression. In J. A. C. J. B. Allen (Ed.), The handbook of emotionelicitation and assessment (pp. 222-238). New York: Oxford University Press Series in Affective Science.
21. Cunningham, D. W., Breidt, M., Kleiner, M., Wallraven, C., &Bülthoff, H. H. (2003a). How believable are real faces?: Towards a perceptual basis for conversational animation. In D. Metaxas, N. Magnenat-Thalmann, & H.-S. Ko (Eds.), Computer animation and social agents 2003 (pp. 23-29). Washington, DC: IEEE Computer Society.
22. Cunningham, D. W., Breidt, M., Kleiner, M., Wallraven, C., &Bülthoff, H. H. (2003b). The inaccuracy and insincerity of real faces. In M. H. Hamza (Ed.), Proceedings of visualization, imaging, and image processing 2003 (pp. 7-12). Calgary: ACTA Press.
23. Cunningham, D. W., Kleiner, M., Wallraven, C., & Bülthoff, H. H. (2005). Manipulating video sequences to determine the components of conversational facial expressions. ACM Transactions on Applied Perception, 2, 251-269.
24. Cunningham, D. W., Nusseck, M., Wallraven, C., & Bülthoff, H. H. (2004). The role of image size in the recognition of conversational facial expressions. Computer Animation & Virtual Worlds, 15, 305-310.
25. Cunningham, D. W., & Wallraven, C. (2009). The interaction between motion and form in expression recognition. In B. Bodenheimer & C. O'Sullivan (Eds.), Proceedings of the 6th Symposium on Applied Perception in Graphics and Visualization (APGV2009) (pp. 41-44). New York, NY: ACM.
26. Cutting, J., & Kozlowski, L. (1977). Recognizing friends by their walk: Gait perception without familiarity cues. Bulletin of the Psychonomic Society, 9, 353-356.
27. Dittrich, W. (1993). Action categories and the perception of biological motion. Perception, 22, 15-22.
28. Dittrich, W., Troscianko, T., Lea, S., & Morgan, D. (1996). Perception of emotion from dynamic pointlight displays represented in dance. Perception, 25, 727-738.
29. Edwards, K. (1998). The face of time: Temporal cues in facial expressions of emotion. Psychological Science, 9, 270-276.
30. Ehrlich, S. M., Schiano, D. J., & Scheridan, K. (2000). Communicating facial affect: It's not the realism, it's the motion. In G. Szwillus & T. Turner (Eds.), Proceedings of the ACM CHI 2000 Conference on Human Factors in Computing Systems (pp. 252-253). New York, NY: ACM.
31. Ekman, P. (1972). Universal and cultural differences in facial expressions of emotion. In J. R. Cole (Ed.), Nebraska symposium on motivation 1971 (pp. 207-283). Lincoln, NE: University of Nebraska Press.
32. Fernandez-Dols, J., Wallbott, H., & Sanchez, F. (1991). Emotion category accessibility and the decoding of emotion from facial expression and context. Journal of Nonverbal Behavior, 15, 107-124.
33. Frank, M. G., & Stennett, J. (2001). The forced-choice paradigm and the perception of facial expressions of emotion. Journal of Personality and Social Psychology, 80, 75-85.
34. Gibson, J. J. (1979). The ecological approach to visual perception. Hillsdale, NJ: Lawrence Erlbaum.
35. Giese, M., & Poggio, T. (2003). Neural mechanisms for the recognition of biological movements and action. Nature Reviews Neuroscience, 4, 179-192.
36. Harwood, N., Hall, L., & Shinkfield, A. (1999). Recognition of facial emotional expressions from moving and static displays by individuals with mental retardation. American Journal on Mental Retardation, 104, 270-278.
37. Hill, H., Troje, N., & Johnston, A. (2005). Range-and domain-specific exaggeration of facial speech. Journal of Vision, 5(10):4, 793-807, http://journalofvision.org/5/10/4/, doi:10.1167/5.10.4.
38. Humphreys, G., Donnelly, N., & Riddoch, M. (1993). Expression is computed separately from facial identity, and is computed separately for moving and static faces: Neuropsychological evidence. Neuropsychologia, 31, 173-181.
39. Irwin, D. E. (1991). Information integration across saccadic eye movements. Cognitive Psychology, 23, 420-456.
40. Isaacs, E., & Tang, J. (1993). What video can and can't do for collaboration: A case study. In J. J. Garcia-Luna & P. Venkat Rangan (Eds.), MULTIMEDIA '93: Proceedings of the first ACM international conference on Multimedia (pp. 496-503). New York: ACM.
41. Johansson, G. (1973). Visual perception of biological motion and a model for its analysis. Perception & Psychophysics, 14, 201-211.
42. Kaetsyri, J., Klucharev, V., Frydrych, M., & Sams, M. (2003). Identification of synthetic and natural emotional facial expressions. In J.-L. Schwartz, F. Berthommier, M.-A. Cathiard, & D. Sodoyer (Eds.), International Conference on Audio-Visual Speech Processing (AVSP 2003) (pp. 239-243).
43. Kahneman, D. (1968). Method, findings, and theory in studies of visual masking. Psychological Bulletin, 70, 404-425.
44. Kleiner, M., Wallraven, C., & Bülthoff,H.H.(2004).The MPI VideolabVA system for high quality synchronous recording of video and audio from multiple viewpoints (Tech. Rep. No. 123). Tübingen, Germany: Max Planck Institute for Biological Cybernetics.
45. Knappmeyer, B., Thornton, I., & Bülthoff, H. (2003). The use of facial motion and facial form during the processing of identity. Vision Research, 43, 1921-1936.
46. Kozlowski, L., & Cutting, J. (1977). Recognizing the sex of a walker from a dynamic point light display. Perception and Psychophysics, 21, 575-580.
47. LaBar, K., Crupain, M. J., Voyvodic, J. T., & McCarthy, G. (2003). Dynamic perception of facial affect and identity in the human brain. Cerebral Cortex, 13, 1023-1033.
48. Lander, K. (2005). Why are moving faces easier to recognize? Visual Cognition, 3, 429-442.
49. Lange, J., & Lappe, M. (2006). A model of biological motion perception from configural form cues. Journal of Neuroscience, 26, 2894-2906.
50. Marr, D., & Vaina, L. M. V. (1982). Representation and recognition of the movements of shapes. Proceedings of the Royal Society of London B, 214, 501-524.
51. Mather, G., & Murdoch, L. (1994). Gender discrimination in biological motion displays based on dynamic cues. Proceedings of the Royal Society of London B, 258, 273-279.
52. Mather, G., Radford, K., & West, S. (1992). Low-level visual processing of biological motion. Proceedings of the Royal Society of London B, 249, 149-155.
53. Maurer, D., Grand, R. L., & Mondloch, C. (2002). The many faces of configural processing. Trends in Cognitive Sciences, 6, 255-260.
54. Mehrabian, A., & Ferris, S. (1967). Inference of attitudes from nonverbal communication in two channels. Journal of Consulting Psychology, 31, 248-252.
55. Mondloch, C., & Maurer, D. (2008). The effect of face orientation on holistic processing. Perception, 37, 1175-1186.
56. Nusseck, M., Cunningham, D. W., De Ruiter, J. P., & Wallraven, C. (under review). Perception of emphasis intensity in audiovisual speech. Speech Communication, 11.
57. Nusseck, M., Cunningham, D. W., Wallraven, C., & Bülthoff, H. H. (2008). The contribution of different facial regions to the recognition of conversational expressions. Journal of Vision, 8(8):1, 1-23, http:// journalofvision.org/8/8/1/, doi:10.1167/8.8.1.
58. O'Rourke, J., & Badler, N. (1980). Model-based image analysis of human motion using constraint propagation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2, 522-536.
59. Pelachaud, C., & Poggi, I. (2002). Subtleties of facial expressions in embodied agents. Journal of Visualization and Computer Animation, 13, 301-312.
60. Poggi, I., & Pelachaud, C. (2000). Perfomative facial expressions in animated faces. In J. Cassell, J.Sullivan,S.Prevost,&E.Churchill(Eds.), Embodied conversational agents (pp. 115-188). Cambridge, MA: MIT Press.
61. Pollick, F., Paterson, H., Bruderlin, A., & Sanford, A. (2001). Perceiving affect from arm movement. Cognition, 82, B51-B61.
62. Rensink, R. A., O'Regan, J. K., & Clark, J. J. (1997). To see or not to see: The need for attention to perceive changes in scenes. Psychological Science, 8, 368-373.
63. Rohr, K. (1994). Towards model-based recognition of human movements in image sequences. Computer Vision, Graphics, and Image Processing: Image Understanding, 59, 94-115.
64. Schultz, J., & Pilz, K. S. (2009). Natural facial motion enhances cortical responses to faces. Experimental Brain Research, 194, 465-475.
65. Schwaninger, A., Wallraven, C., Cunningham, D. W., & Chiller-Glaus, S. D. (2006). Processing of facial identity and expression: A psychophysical, physiological and computational perspective. Progress in Brain Research, 156, 325-348.
66. Shipley, T. F., & Cunningham, D. W. (2001). Perception of occluding and occluded objects over time: Spatiotemporal segmentation and unit formation. In T. F. Shipley & P. J. Kellman (Eds.), From fragments to objects: Segmentation and grouping in vision (pp. 557-585). Oxford, UK: Elsevier Science.
67. Shipley, T. F., & Kellman, P. J. (1994). Spatiotemporal boundary formation: Boundary, form, and motion perception from transformations of surface elements. Journal of Experimental Psychology: General, 123, 3-20.
68. Simons, D. J., & Levin, D. T. (1998). Failure to detect changes to people during a real-world interaction. Psychonomic Bulletin and Review, 5, 644-649.
69. Stephenson, G., Ayling, K., & Rutter, D. (1976). The role of visual communication in social exchange. British Journal of Social and Clinical Psychology, 15, 113-120.
70. Stigler, R. (1910). Chronophotische Studien über den Umgebungskontrast [Effects of exposure duration and luminance on the contrast of the surround]. Pflägers Archiv für die Gesamte Physiologie des Menschen und der Tiere, 134, 365-435.
71. Stone, J. (1998). Object recognition using spatio-temporal signatures. Vision Research, 38, 947-951.
72. Tanaka, J., & Farah, M. (1993). Parts and wholes in face recognition. Quarterly Journal of Experimental Psychology A, 46, 225-245.
73. Troje, N. (2002). Decomposing biological motion: A framework for analysis and synthesis of human gait patterns. Journal of Vision, 2(5):2, 371-387, http:// journalofvision.org/2/5/2/, doi:10.1167/2.5.2.
74. Vertegaal, R. (1997). Conversational awareness in multiparty vmc. In S. Pemberton (Ed.), Extended abstracts of CHI'97 (pp. 496-503). Atlanta, GA: ACM.
75. Wallraven, C., Breidt, M., Cunningham, D. W., & Bülthoff, H. H. (2008). Evaluating the perceptual realism of animated facial expressions. ACM Transactions on Applied Perception, 4, 1-20.
76. Webb, J., & Aggarwal, J. (1982). Structure from motion of rigid and jointed objects. Artificial Intelligence, 19, 107-130.
77. Wehrle, T., Kaiser, S., Schmidt, S., & Schere, K. R. (2000). Studying the dynamics of emotional expressions using synthesized facial muscle movements. Journal of Personality and Social Psychology, 78, 105-119.
78. Weyers, P., Mühlberger, A., Hefele, C., & Pauli, P. (2006). Electromyographic responses to static and dynamic avatar emotional facial expressions. Psychophysiology, 43, 450-453.
79. Williams, M., Breitmeyer, B., Lovegrove, W., & Gutierrez, L. (1991). Metacontrast with masks varying in spatial frequency and wavelength. Vision Research, 31, 2017-2023.
80. Yngve, V. H. (1970). On getting a word in edgewise. In S. Pemberton (Ed.), Papers from the Sixth Regional Meeting of the Chicago Linguistic Society (pp. 567-578). Chicago: Chicago Linguistic Society.
81. Zeng, Z., Pantic, M., Roisman, G., & Huang, T. (2009). A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 59, 39-58.