A dynamic appearance descriptor approach to facial actions temporal modeling

IEEE Transactions on Cybernetics

Volumn 44, Issue 2, 2014, Pages 161-174

  Jiang, Bihan ^a   Valstar, Michel ^b   Martinez, Brais ^c   Pantic, Maja ^a  

^a IMPERIAL COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY OF NOTTINGHAM   (United Kingdom)

^c UNIVERSITY OF TWENTE   (Netherlands)

Author keywords

Action unit detection; dynamic appearance descriptors; facial dynamics; LPQ TOP; temporal segment detection

Indexed keywords

ACTION UNIT; DISCRIMINATIVE CLASSIFIERS; DYNAMIC APPEARANCE; FACIAL ACTION CODING SYSTEM; LOCAL PHASE QUANTIZATIONS; LPQ-TOP; STATE-OF-THE-ART METHODS; TEMPORAL SEGMENTS;

BEHAVIORAL RESEARCH; DATABASE SYSTEMS; DYNAMICS; EXPERIMENTS; FACE RECOGNITION; GESTURE RECOGNITION; MARKOV PROCESSES;

COMPUTER KEYBOARDS;

ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; AUTOMATED PATTERN RECOGNITION; BIOLOGICAL MODEL; BIOMETRY; COMPUTER ASSISTED DIAGNOSIS; COMPUTER SIMULATION; FACE; FACIAL EXPRESSION; HISTOLOGY; HUMAN; METHODOLOGY; MULTIMODAL IMAGING; PHOTOGRAPHY; PHYSIOLOGY; REPRODUCIBILITY; SENSITIVITY AND SPECIFICITY; STATISTICAL MODEL;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; BIOMETRY; COMPUTER SIMULATION; FACE; FACIAL EXPRESSION; HUMANS; IMAGE INTERPRETATION, COMPUTER-ASSISTED; MODELS, BIOLOGICAL; MODELS, STATISTICAL; MULTIMODAL IMAGING; PATTERN RECOGNITION, AUTOMATED; PHOTOGRAPHY; REPRODUCIBILITY OF RESULTS; SENSITIVITY AND SPECIFICITY;

EID: 84892815469     PISSN: 21682267     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCYB.2013.2249063     Document Type: Article

References (54)

1. P. Ekman and E. L. Ronsenberg, What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System. Oxford, U.K.: Oxford Univ. Press, 2005.
2. J. F. Cohn and P. Ekman, Measuring Facial Actions. Oxford, U.K.: Oxford Univ. Press, 2005.
3. M. Pantic, "Machine analysis of facial behaviour: Naturalistic and dynamic behaviour," Philosoph. Trans. Roy. Soc. B: Biol. Sci., vol. 365, no. 1535, pp. 3505-3513, 2009.
4. P. Ekman, W. Friesen, and J. C. Hager, "Facial action coding system," in A Human Face. Salt Lake City, UT, USA, 2002.
5. P. Ekman, "Darwin, deception, and facial expression," Ann. New York Acad. Sci., vol. 1000, pp. 205-221, Dec. 2003. (Pubitemid 38264836)
6. J. F. Cohn and K. L. Schmidt, "The timing of facial motion in posed and spontaneous smiles," Int. J. Wavelets, Multiresolution Inform. Process., vol. 2, pp. 1-12, Jun. 2004.
7. M. Valstar, M. Pantic, Z. Ambadar, and J. Cohn, "Spontaneous vs. posed facial behavior: Automatic analysis of brow actions," in Proc. ACM Int. Conf. Multimodal Interfaces, 2006, pp. 162-170. (Pubitemid 47129473)
8. M. Valstar, H. Gunes, and M. Pantic, "How to distinguish posed from spontaneous smiles using geometric features," in Proc. ACM Int. Conf. Multimodal Interfaces, 2007, pp. 38-45.
9. M. Hoque, L.-P. Morency, and R. W. Picard, "Are you friendly or just polite?-analysis of smiles in spontaneous face-to-face interactions," in Proc. 4th Int. Conf. Affective Comput. Intell. Interaction, part I. 2011, pp. 135-144.
10. A. C. Williams, "Facial expression of pain: An evolutionary account," Behav. Brain Sci., vol. 25, no. 4, pp. 439-488, 2002.
11. Z. Ambadar, J. Schooler, and J. F. Cohn, "Deciphering the enigmatic face: The importance of facial dynamics to interpreting subtle facial expressions," Psychol. Sci., vol. 16, no. 5, pp. 403-410, 2005. (Pubitemid 40763688)
12. H. Gunes and M. Piccardi, "Automatic temporal segment detection and affect recognition from face and body display," IEEE Trans. Syst., Man, Cybern., B, Cybern., vol. 39, no. 1, pp. 64-84, Feb. 2009.
13. M. Kim and V. Pavlovic, "Structured output ordinal regression for dynamic facial emotion intensity prediction," in Proc. Eur. Conf. Comput. Vision, 2010, pp. 649-662.
14. G. Sandbach, S. Zafeiriou, M. Pantic, and D. Rueckert, "Recognition of 3-D facial expression dynamics," Image Vision Comput., vol. 30, no. 10, pp. 762-773, Oct. 2012.
15. O. Rudovic, V. Pavlovic, and M. Pantic, "Multi-output Laplacian dynamic ordinal regression for facial expression recognition and intensity estimation," in Proc. IEEE Conf. Comput. Vision Pattern Recognit., Jun. 2012, pp. 2634-2641.
16. C. Shan, S. Gong, and P. W. Mcowan, "Dynamic facial expression recognition using a Bayesian temporal manifold model," in Proc. Brit. Mach. Vision Conf., 2006, pp. 297-306.
17. J. Delannoy and J. McDonald, "Rankboost with L1 regularization for facial expression recognition and intensity estimation," in Proc. IEEE Int. Conf. Comput. Vision, Sep. 2009, pp. 1018-1025.
18. S. Kaltwang, O. Rudovic, and M. Pantic, "Continuous pain intensity estimation from facial expressions," in Proc. Int. Symp. Visual Comput., Jul. 2012, pp. 368-377.
19. P. Lucey, J. F. Cohn, K. M. Prkachin, P. E. Solomon, S. W. Chew, and I. Matthews, "Painful monitoring: Automatic pain monitoring using the UNBC-McMaster shoulder pain expression archive database," Image Vision Comput., vol. 30, no. 3, pp. 197-205, 2012.
20. P. Lucey, J. F. Cohn, I. Matthews, S. Lucey, S. Sridharan, J. Howlett, and K. M. Prkachin, "Automatically detecting pain in video through facial action units," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 41, no. 3, pp. 664-674, 2010.
21. M. Pantic and I. Patras, "Dynamics of facial expression: Recognition of facial actions and their temporal segments from face profile image sequences," IEEE Trans. Syst., Man, Cybern., B, vol. 36, no. 2, pp. 433-449, Apr. 2006.
22. M. Valstar and M. Pantic, "Fully automatic recognition of the temporal phases of facial actions," IEEE Trans. Syst., Man, Cybern., B, vol. 42, no. 1, pp. 28-43, Feb. 2012.
23. M. Bartlett, G. Littlewort-Ford, M. Frank, C. Lainscsek, I. Fasel, and J. Movellan, "Fully automatic facial action recognition in spontaneous behaviour," in Proc. IEEE Int. Conf. Automatic Face Gesture Recognit., 2006, pp. 223-230. (Pubitemid 44711354)
24. B. Jiang, M. Valstar, and M. Pantic, "Action unit detection using sparse appearance descriptors in space-time video volumes," in Proc. IEEE Int. Conf. Automatic Face Gesture Recognit., Mar. 2011, pp. 314-321.
25. G. Zhao and M. Pietikainen, "Dynamic texture recognition using local binary pattern with an application to facial expressions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 2, no. 6, pp. 915-928, Jun. 2007. (Pubitemid 46667407)
26. V. Ojansivu and J. Heikkila, "Blur insensitive texture classification using local phase quantization," in Proc. Int. Conf. on Image and Signal Processing, vol. 5099. 2008, pp. 236-243.
27. B. Martinez, M. F. Valstar, X. Binefa, and M. Pantic, "Local evidence aggregation for regression based facial point detection," IEEE Trans. Pattern Anal. Mach. Intell., to be published.
28. M. S. Bartlett, J. Larsen, J. C. Hager, and P. Ekman, "Classifying facial actions," in Proc. Advances Neural Inform. Process. Syst., 1996, pp. 823-829.
29. J. J. Lien, T. Kanade, J. F. Cohn, and C.-C. Li, "Automated facial expression recognition based on FACS action units," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., Apr. 1998, pp. 390-395.
30. M. Pantic, L. Rothkrantz, and H. Koppelaar, "Automation of non-verbal communication of facial expressions," in Proc. Eur. Conf. Multimedia, 1998, pp. 86-93.
31. Y. Tian, T. Kanade, and J. Cohn, "Recognizing action units for facial expression analysis," IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 2, pp. 97-115, Feb. 2001. (Pubitemid 32254704)
32. M. Valstar and M. Pantic, "Fully automatic facial action unit detection and temporal analysis," in Proc. IEEE Conf. Comput. Vision Pattern Recognit., Jun. 2006, p. 149.
33. Z. Zeng, M. Pantic, G. I. Roisman, and T. S. Huang, "A survey of affect recognition methods: Audio, visual, and spontaneous expressions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 1, pp. 39-58, Jan. 2009.
34. H. Gunes and M. Pantic, "Automatic, dimensional and continuous emotion recognition," Int. J. Synthetic Emotion, vol. 1, no. 1, pp. 68-99, 2010.
35. M. Valstar, M. Mehu, B. Jiang, M. Pantic, and K. Scherer, "Metaanalysis of the first facial expression recognition challenge," Syst., Man, Cybern., B, vol. 42, no. 4, pp. 966-979, Aug. 2012.
36. S. Koelstra, M. Pantic, and I. Patras, "A dynamic texture based approach to recognition of facial actions and their temporal models," IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 11, pp. 1940-1954, Nov. 2010.
37. M. Pantic and I. Patras, "Detecting facial actions and their temporal segments in nearly frontal-view face image sequences," in Proc. IEEE Int. Conf. Syst., Man, Cybern., Oct. 2005, pp. 3358-3363. (Pubitemid 41667331)
38. O. Rudovic, V. Pavlovic, and M. Pantic, "Kernel conditional ordinal random fields for temporal segmentation of facial action units," in Proc. 12th Eur. Conf. Comput. Vision, 2012, pp. 260-269.
39. M. H. Mahoor, S. Cadavid, D. S. Messinger, and J. F. Cohn, "A framework for automated measurement of the intensity of non-posed facial action units," in Proc. IEEE Workshop CVPR Human Communicative Behavior Anal. (CVPR4HB), Jun. 2009, pp. 74-80.
40. B. S. Arman Savran and M. T. Bilge, "Regression-based intensity estimation of facial action units," Image Vison Comput., vol. 30, no. 10, pp. 774-784, Oct. 2011.
41. Y. Tong, W. Liao, and Q. Ji, "Facial action unit recognition by exploiting their dynamic and semantic relationships," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 10, pp. 1683-1699, 2007.
42. Y. Zhang and Q. Ji, "Active and dynamic information fusion for facial expression understanding from image sequences," IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 5, pp. 699-714, 2005. (Pubitemid 40608403)
43. M. F. Valstar, B. Martinez, X. Binefa, and M. Pantic, "Facial point detection using boosted regression and graph models," in Proc. IEEE Int. Conf. Comput. Vision Pattern Recognit., Jun. 2010, pp. 2729-2736.
44. T. Ahonen, A. Hadid, and M. Pietikainen, "Face recognition with local binary patterns," in Proc. Eur. Conf. Comput. Vision, 2004, pp. 469-481.
45. C. Shan, S. Gong, and P. McOwan, "Facial expression recognition based on local binary patterns: A comprehensive study," Image Vision Comput., vol. 27, no. 6, pp. 803-816, 2008.
46. S. Maji, A. Berg, and J. Malik, "Classification using intersection kernel support vector machines is efficient," in Proc. IEEE Conf. Comput. Vision Pattern Recognit., Jun. 2008, pp. 1-8.
47. C. M. Bishop, Pattern Recognition and Machine Learning (Information Science and Statistics). New York, USA: Springer-Verlag, 2006.
48. T. Kanade, J. F. Cohn, and Y. Tian, "Comprehensive database for facial expression analysis," in Proc. IEEE Int. Conf. Automatic Face Gesture Recognit., Mar. 2000, pp. 46-53.
49. P. Lucey, J. F. Cohn, K. M. Prkachin, P. E. Solomon, and I. Matthews, "Painful data: The UNBC-McMaster shoulder pain expression archive database," in Proc. IEEE Int. Conf. Automatic Face Gesture Recognit., 2011, pp. 57-64.
50. E. Douglas-Cowie, R. Cowie, C. Cox, N. Amier, and D. Heylen, "The sensitive artificial listener: An induction technique for generating emotionally coloured conversation," in Proc. LREC Workshop Corpora Res. Emotion Affect, 2008, pp. 1-4.
51. G. McKeown, M. Valstar, R. Cowie, M. Pantic, and M. Schroder, "The SEMAINE database: Annotated multimodal records of emotionally colored conversations between a person and a limited agent," IEEE Trans. Affective Comput., vol. 3, no. 1, pp. 5-17, Mar. 2012.
52. K. Sung and T. Poggio, "Example-based learning for view-based human face detection," IEEE Trans. Pattern Anal. Mach. Intell., vol. 20, no. 1, pp. 39-51, Jan. 1998. (Pubitemid 128741306)
53. D. M. Green and J. A. Swets, Signal Detection Theory and Psychophysics. New York, USA: Wiley, 1966.
54. M. Valstar and M. Pantic, "Combined support vector machines and hidden Markov models for modeling facial action temporal dynamics," in Proc. ICCV-HCI, 2007, pp. 118-127.