Towards reading hidden emotions: A comparative study of spontaneous micro-expression spotting and recognition methods

IEEE Transactions on Affective Computing

Volumn 9, Issue 4, 2018, Pages 563-577

  Li, Xiaobai ^a   Hong, Xiaopeng ^a   Moilanen, Antti ^a   Huang, Xiaohua ^a   Pfister, Tomas ^b   Zhao, Guoying ^a   Pietikainen, Matti ^a  

^a UNIVERSITY OF OULU   (Finland)

^b UNIVERSITY OF OXFORD   (United Kingdom)

Author keywords

affective computing; facial expression recognition; HOG; LBP; Micro expression

Indexed keywords

COMPUTER PROGRAMMING; COMPUTER SCIENCE;

AFFECTIVE COMPUTING; COMPARATIVE STUDIES; FACIAL EXPRESSION RECOGNITION; FACIAL EXPRESSIONS; FEATURE DIFFERENCES; MICRO-EXPRESSIONS; RECOGNITION METHODS; SHORT DURATIONS;

HUMAN COMPUTER INTERACTION;

EID: 85057763182     PISSN: None     EISSN: 19493045     Source Type: Journal    
DOI: 10.1109/TAFFC.2017.2667642     Document Type: Article

References (60)

1. P. Ekman, "Darwin, deception, and facial expression," Ann. New York Academy Sci., vol. 1000, no. 1, pp. 205-221, 2003.
2. W.-J. Yan, Q. Wu, J. Liang, Y.-H. Chen, and X. Fu, "How fast are the leaked facial expressions: The duration of micro-expressions," J. Nonverbal Behavior, vol. 37, no. 4, pp. 217-230, 2013.
3. D. Matsumoto and H. Hwang, "Evidence for training the ability to read microexpressions of emotion," Motivation Emotion, vol. 35, pp. 181-191, 2011.
4. S. Porter and L. ten Brinke, "Reading between the lies identifying concealed and falsified emotions in universal facial expressions," Psychological Sci., vol. 19, no. 5, pp. 508-514, 2008.
5. E. Haggard and K. Isaacs, "Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy," in Methods of Research in Psychotherapy. New York, NY, USA: Appleton-Century-Crofts, 1966, pp. 154-165.
6. P. Ekman and W. V. Friesen, "Nonverbal leakage and clues to deception," Psychiatry, vol. 32, no. 1, pp. 88-106, 1969.
7. M. Frank and P. Ekman, "The ability to detect deceit generalizes across different types of high-stake lies." J. Personality Soc. Psychology, vol. 72, no. 6, 1997, Art. no. 1429.
8. P. Ekman, Microexpression Training Tool (METT). San Francisco, CA, USA: University California, 2002.
9. P. Ekman, "Lie catching and microexpressions," in Philosophy of Deception, C. Martin, Ed. Oxford University Press, ch. 7, pp. 118-133, 2009.
10. P. Ekman, Emotions Revealed: Recognizing Faces and Feelings to Improve Communication and Emotional Life. London, U.K.: Macmillan, 2007.
11. P. Ekman, M. Sullivan, and M. Frank, "A few can catch a liar," Psychological Sci., vol. 10, no. 3, pp. 263-266, 1999.
12. Z. Zeng,M. Pantic,G. Roisman, and T.Huang, "A survey of affect recognition methods:Audio, visual, and spontaneous expressions," IEEE Trans. PatternAnal. Mach. Intell., vol. 31, no. 1, pp. 39-58, Jan. 2009.
13. R. Picard, Affective Computing. Cambridge, MA, USA: MIT Press, 1997.
14. A. Moilanen, G. Zhao, and M. Pietikainen, "Spotting rapid facial movements from videos using appearance-based feature difference analysis," in Proc. 22nd Int. Conf. Pattern Recognit., 2014, pp. 1722-1727.
15. M. Kamachi, M. Lyons, and J. Gyoba, The Japanese female facial expression (JAFFE) database, 1998. [Online]. Available: http:// www. kasrl. org/jaffe.html
16. T. Kanade, J. Cohn, and Y. Tian, "Comprehensive database for facial expression analysis," in Proc. 4th IEEE Int. Conf. Autom. Face Gesture Recognit., 2000, pp. 46-53.
17. M. Pantic, M. Valstar, R. Rademaker, and L. Maat, "Web-based database for facial expression analysis," in Proc. IEEE Int. Conf. Multimedia Expo, 2005, Art. no. 5.
18. X. Li, T. Pfister, X. Huang, G. Zhao, and M. Pietikainen, "A spontaneous micro-expression database: Inducement, collection and baseline," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., 2013, pp. 1-6.
19. W.-J. Yan, Q. Wu, Y.-J. Liu, S.-J. Wang, and X. Fu, "CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., 2013, pp. 1-7.
20. W.-J. Yan, X. Li, S.-J. Wang, G. Zhao, Y.-J. Liu, Y.-H. Chen, and X. Fu, "CASME II: An improved spontaneous micro-expression database and the baseline evaluation," PloS One, vol. 9, no. 1, 2014, Art. no. e86041.
21. M. Shreve, S. Godavarthy, V. Manohar, D. Goldgof, and S. Sarkar, "Towards macro-and micro-expression spotting in video using strain patterns," in Proc.WorkshopAppl. Comput. Vis., 2009, pp. 1-6.
22. M. Shreve, S. Godavarthy, D. Goldgof, and S. Sarkar, "Macro-and micro-expression spotting in long videos using spatio-temporal strain," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., 2011, pp. 51-56.
23. S. Polikovsky, Y. Kameda, and Y. Ohta, "Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor," in Proc. 3rd Int. Conf. Crime Detection Prevention, 2009, pp. 1-6.
24. P. Ekman and W. Friesen, Facial Action Coding System: A Technique for the Measurement of Facial Movement. Palo Alto, CA, USA: Consulting Psychologists Press, 1978.
25. T. Pfister, X. Li, G. Zhao, and M. Pietikainen, "Recognising spontaneous facial micro-expressions," in Proc. IEEE Int. Conf. Comput. Vis., 2011.
26. J. Coan and J. Allen, Handbook of Emotion Elicitation and Assessment. New York, NY, USA: Oxford University Press, 2007.
27. J. Gross and R. Levenson, "Emotion elicitation using films," Cognition Emotion, vol. 9, no. 1, pp. 87-108, 1995.
28. Z. Zeng, Y. Fu, G. I. Roisman, Z. Wen, Y. Hu, and T. S. Huang, "Spontaneous emotional facial expression detection." J. Multimedia, vol. 1, no. 5, pp. 1-8, 2006.
29. A. Krolak and P. Strumi»»o, "Eye-blink detection system for human-computer interaction," Universal Access the Inform. Soc., vol. 11, no. 4, pp. 409-419, 2012.
30. S. Liwicki, S. Zafeiriou, and M. Pantic, "Incremental slow feature analysis with indefinite kernel for online temporal video segmentation," in Proc. 11th Asian Conf. Comput. Vis., 2013, pp. 162-176.
31. S. Polikovsky and Y. Kameda, "Facial micro-expression detection in high-speed video based on facial action coding system (FACS)," IEICE Trans. Inform. Syst., vol. 96, no. 1, pp. 81-92, 2013.
32. Q. Wu, X. Shen, and X. Fu, "The machine knows what you are hiding: An automatic micro-expression recognition system," in Proc. 4th Int. Conf. Affective Comput. Intell. Interaction, 2011, pp. 152-162.
33. G. Warren, E. Schertler, and P. Bull, "Detecting deception from emotional and unemotional cues," J. Nonverbal Behavior, vol. 33, no. 1, pp. 59-69, 2009.
34. J. A. Ruiz-Hernandez and M. Pietikainen, "Encoding local binary patterns using the re-parametrization of the second order gaussian jet," in Proc. IEEE Int. Conf. Autom. Face Gesture Recognit., 2013, pp. 1-6.
35. A. K. Davison, M. H. Yap, N. Costen, K. Tan, C. Lansley, and D. Leightley, "Micro-facial movements: An investigation on spatiotemporal descriptors," in Proc. ECCVWorkshop, 2014, pp. 111-123.
36. S. Yao, N. He, H. Zhang, and O. Yoshie, "Micro-expression recognition by feature points tracking," in Proc. 10th Int. Conf. Commun., 2014, pp. 1-4.
37. Y. Song, L.-P. Morency, and R. Davis, "Learning a sparse codebook of facial and body microexpressions for emotion recognition," in Proc. 15th ACM Int. Conf. Multimodal Interaction, 2013, pp. 237-244.
38. S.-J. Wang, W.-J. Yan, X. Li, G. Zhao, and X. Fu, "Micro-expression recognition using dynamic textures on tensor independent color space," in Proc. 2014 22nd Int. Conf. Pattern Recognit., 2014, pp. 4678-4683.
39. S.-J. Wang, W.-J. Yan, G. Zhao, X. Fu, and C.-G. Zhou, "Microexpression recognition using robust principal component analysis and local spatiotemporal directional features," in Proc. ECCV Workshop, 2014, pp. 325-338.
40. C. Tomasi and T. Kanade, Detection and Tracking of Point Features. School Comput. Sci., Carnegie Mellon Univ., Pittsburgh, PA, USA, 1991.
41. T. Ojala, M. Pietikainen, and T. Maenpaa, "Multiresolution grayscale and rotation invariant texture classification with local binary patterns," IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 7, pp. 971-987, Jul. 2002.
42. C. Liu, "Beyond pixels: Exploring new representations and applications for motion analysis," Ph.D. dissertation, Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA, USA, 2009.
43. T. Ahonen, A. Hadid, and M. Pietikainen, "Face description with local binary patterns: Application to face recognition," IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 12, pp. 2037-2041, Dec. 2006.
44. T. Cootes, et al., "Active shape models-their training and application," Comput. Vis. Image Understanding, vol. 61, no. 1, pp. 38-59, 1995.
45. A. Goshtasby, "Image registration by local approximation methods," Image Vis. Comput., vol. 6, no. 4, pp. 255-261, 1988.
46. H.-Y. Wu, M. Rubinstein, E. Shih, J. V. Guttag, F. Durand, and W. T. Freeman, "Eulerian video magnification for revealing subtle changes in the world," ACMTrans. Graph., vol. 31, no. 4, p. 65, 2012.
47. Z. Zhou, G. Zhao, and M. Pietikainen, "Towards a practical lipreading system," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2011, pp. 137-144.
48. G. Zhao and M. Pietikainen, "Dynamic texture recognition using local binary patterns with an application to facial expressions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 6, pp. 915-928, Jun. 2007.
49. A. Dhall, R. Goecke, J. Joshi, M. Wagner, and T. Gedeon, "Emotion recognition in the wild challenge 2013," in Proc. ACM Int. Conf. Multimodal Interaction, 2013, pp. 509-516.
50. S. E. Kahou, P. Froumenty, and C. Pal, "Facial expression analysis based on high dimensional binary features," in Proc. Comput. Vision-ECCV Workshops, 2014, pp. 135-147.
51. N. Dalal and B. Triggs, "Histograms of oriented gradients for human detection," in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 2005, pp. 886-893.
52. O. Deniz, G. Bueno, J. Salido, and F. De la Torre, "Face recognition using histograms of oriented gradients," PR Lett., vol. 32, no. 12, pp. 1598-1603, 2011.
53. Z. Li, J.-I. Imai, and M. Kaneko, "Facial-component-based bag of words and PHOG descriptor for facial expression recognition," in Proc. IEEE Int. Conf. Syst. Man Cybern., 2009, pp. 1353-1358.
54. T. Zhang, Y. Y. Tang, B. Fang, Z. Shang, and X. Liu, "Face recognition under varying illumination using gradientfaces," IEEE Trans. Image Process., vol. 18, no. 11, pp. 2599-2606, Nov. 2009.
55. C.-C. Chang and C.-J. Lin, ''LIBSVM: A library for support vector machines," ACM Trans. Intell. Syst. Technol., vol. 2, no. 3, 2011, Art. no. 27.
56. G. Zhao, X. Huang, M. Taini, S. Li, and M. Pietikainen, "Facial expression recognition from near-infrared videos," Image Vis. Comput., vol. 29, pp. 607-619, 2011.
57. Y. Wang, J. See, R. C.-W. Phan, and Y.-H. Oh, "LBP with six intersection points: Reducing redundant information in lbp-top for micro-expression recognition," in Proc. 12th Asian Conf. Comput. Vis., 2014, pp. 525-537.
58. S. J. Wang, et al. "Micro-expression recognition using color spaces," IEEE Trans. Image Process., vol. 24, no. 12, pp. 6034-6047, Dec. 2015.
59. S. Liong, R. Phan, J. See, Y. Oh, and K. Wong, "Optical strain based recognition of subtle emotions," in Proc. Int. Symp. Intell. Signal Process. Commun. Syst., 2014, pp. 180-184.
60. S. Liong, J. See, R. Phan, A. Le Ngo , Y. Oh, and K. Wong, "Subtle expression recognition using optical strain weighted features," in Proc. Asian Conf. Comput. Vis. Workshop, 2014, pp. 644-654