(Not) Hearing Happiness: Predicting Fluctuations in Happy Mood From Acoustic Cues Using Machine Learning

Emotion

Volumn 20, Issue 4, 2020, Pages 642-658

  Weidman, Aaron C ^a   Sun, Jessie ^b   Vazire, Simine ^b   Quoidbach, Jordi ^c   Ungar, Lyle H ^d   Dunn, Elizabeth W ^e  

^a UNIVERSITY OF MICHIGAN   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c ESADE BUSINESS SCHOOL   (Spain)

^d UNIVERSITY OF PENNSYLVANIA   (United States)

^e UNIVERSITY OF BRITISH COLUMBIA   (Canada)

Author keywords

acoustic analysis; experience sampling; happiness; happy mood; machine learning

Indexed keywords

ADULT; ASSOCIATION; EMOTION; FEMALE; HAPPINESS; HUMAN; MACHINE LEARNING; MALE; PHYSIOLOGY; YOUNG ADULT;

ADULT; CUES; EMOTIONS; FEMALE; HAPPINESS; HUMANS; MACHINE LEARNING; MALE; YOUNG ADULT;

EID: 85061440982     PISSN: 15283542     EISSN: 19311516     Source Type: Journal    
DOI: 10.1037/emo0000571     Document Type: Article

References (63)

1. Affectiva. (2018). Affectiva automotive AI. Retrieved from https://www.affectiva.com/product/affectiva-automotive-ai/
2. American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders: DSM–5 (5th ed.). Arlington, VA: Author.
3. Baird, B. M., & Lucas, R. E. (2011). “… And how about now?”: Effects of item redundancy on contextualized self-reports of personality. Journal of Personality, 79, 1081–1112. http://dx.doi.org/10.1111/j.1467-6494.2011.00716.x
4. Banse, R., & Scherer, K. R. (1996). Acoustic profiles in vocal emotion expression. Journal of Personality and Social Psychology, 70, 614–636. http://dx.doi.org/10.1037/0022-3514.70.3.614
5. Beck, E. D., & Jackson, J. J. (2018). Consistency and change in idiographic personality: A longitudinal ESM network study. Manuscript submitted for publication. Retrieved from https://psyarxiv.com/pb92q/
6. Beyond Verbal. (2018). Virtual private assistant. Retrieved from https://craft.co/beyond-verbal
7. Breil, S. M., Geukes, K., Wilson, R. E., Nestler, S., Vazire, S., & Back, M. (2018). Zooming into real-life extraversion–How personality and situation shape sociability in social interactions. Manuscript submitted for publication.
8. Colman, D. E., Vineyard, J., & Letzring, T. D. (2018). Exploring beyond simple demographic variables: Differences between traditional laboratory samples and crowdsourced online samples on the Big Five personality traits. Personality and Individual Differences, 133, 41–46. http://dx.doi.org/10.1016/j.paid.2017.06.023
9. Conner, T. S., Tennen, H., Fleeson, W., & Barrett, L. F. (2009). Experience sampling methods: A modern idiographic approach to personality research. Social and Personality Psychology Compass, 3, 292–313. http://dx.doi.org/10.1111/j.1751-9004.2009.00170.x
10. Cordaro, D. T., Keltner, D., Tshering, S., Wangchuk, D., & Flynn, L. M. (2016). The voice conveys emotion in ten globalized cultures and one remote village in Bhutan. Emotion, 16, 117–128. http://dx.doi.org/10.1037/emo0000100
11. Diener, E., Kanazawa, S., Suh, E. M., & Oishi, S. (2015). Why people are in a generally good mood. Personality and Social Psychology Review, 19, 235–256. http://dx.doi.org/10.1177/1088868314544467
12. Dodds, P. S., Harris, K. D., Kloumann, I. M., Bliss, C. A., & Danforth, C. M. (2011). Temporal patterns of happiness and information in a global social network: Hedonometrics and Twitter. PLoS ONE, 6, e26752. http://dx.doi.org/10.1371/journal.pone.0026752
13. Edwards, T., & Holtzman, N. S. (2017). A meta-analysis of correlations between depression and first person singular pronoun use. Journal of Research in Personality, 68, 63–68. http://dx.doi.org/10.1016/j.jrp.2017.02.005
14. Eyben, F., Scherer, K. R., Schuller, B. W., Sundberg, J., Andre, E., Busso, C.,… Truong, K. P. (2016). The Geneva Minimalist Acoustic Parameter Set (GeMAPS) for voice research and affective computing. IEEE Transactions on Affective Computing, 7, 190–202. http://dx.doi.org/10.1109/TAFFC.2015.2457417
15. Eyben, F., Weninger, F., Gross, F., & Schuller, B. (2013). Recent developments in openSMILE, the Munich Open-Source Multimedia Feature Extractor. MM ’13 Proceedings of the 21st ACM international conference on Multimedia (pp. 835–838). New York, NY: ACM. http://dx.doi.org/10.1145/2502081.2502224
16. Feldman Barrett, L., & Russell, J. A. (1998). Independence and bipolarity in the structure of current affect. Journal of Personality and Social Psychology, 74, 967–984. http://dx.doi.org/10.1037/0022-3514.74.4.967
17. Finnigan, K. M., & Vazire, S. (2018). The incremental validity of average state self-reports over global self-reports of personality. Journal of Personality and Social Psychology, 115, 321–337. http://dx.doi.org/10.1037/pspp0000136
18. Geldhof, G. J., Preacher, K. J., & Zyphur, M. J. (2014). Reliability estimation in a multilevel confirmatory factor analysis framework. Psychological Methods, 19, 72–91. http://dx.doi.org/10.1037/a0032138
19. Harari, G. M., Lane, N. D., Wang, R., Crosier, B. S., Campbell, A. T., & Gosling, S. D. (2016). Using smartphones to collect behavioral data in psychological science: Opportunities, practical considerations, and challenges. Perspectives on Psychological Science, 11, 838–854. http://dx.doi.org/10.1177/1745691616650285
20. Joel, S., Eastwick, P. W., & Finkel, E. J. (2017). Is romantic desire predictable? Machine learning applied to initial romantic attraction. Psychological Science, 28, 1478–1489. http://dx.doi.org/10.1177/0956797617714580
21. Kahneman, D., Krueger, A. B., Schkade, D. A., Schwarz, N., & Stone, A. A. (2004). A survey method for characterizing daily life experience: The day reconstruction method. Science, 306, 1776–1780. http://dx.doi.org/10.1126/science.1103572
22. Karam, Z. N., Provost, E. M., Singh, S., Montgomery, J., Archer, C., Harrington, G., & Mcinnis, M. G. (2014, May). Ecologically valid long-term mood monitoring of individuals with bipolar disorder using speech. In International Conference on Acoustics, Speech and Signal Processing (ICASSP). Florence, Italy.
23. Kassam, K. S., & Mendes, W. B. (2013). The effects of measuring emotion: Physiological reactions to emotional situations depend on whether someone is asking. PLoS ONE, 8, e64959. http://dx.doi.org/10.1371/journal.pone.0064959
24. Knight, W. (2016, June). Amazon working on making Alexa recognize your emotions. MIT Technology Review. Retrieved from https://www.technologyreview.com/s/601654/amazon-working-on-making-alexarecognize-your-emotions/
25. Kosinski, M., Stillwell, D., & Graepel, T. (2013). Private traits and attributes are predictable from digital records of human behavior. Proceedings of the National Academy of Sciences of the United States of America, 110, 5802–5805. http://dx.doi.org/10.1073/pnas.1218772110
26. Kross, E., Verduyn, P., Boyer, M., Drake, B., Gainsburg, I., Vickers, B.,… Jonides, J. (2018). Does counting emotion words on online social networks provide a window into people’s subjective experience of emotion? A case study on Facebook. Emotion. Advance online publication. http://dx.doi.org/10.1037/emo0000416
27. Kuppens, P., Oravecz, Z., & Tuerlinckx, F. (2010). Feelings change: Accounting for individual differences in the temporal dynamics of affect. Journal of Personality and Social Psychology, 99, 1042–1060. http://dx.doi.org/10.1037/a0020962
28. LaHuis, D. M., Hartman, M. J., Hakoyama, S., & Clark, P. C. (2014). Explained variance measures for multilevel models. Organizational Research Methods, 17, 433–451. http://dx.doi.org/10.1177/1094428114541701
29. Laukka, P., Juslin, P., & Bresin, R. (2005). A dimensional approach to vocal expression of emotion. Cognition and Emotion, 19, 633–653. http://dx.doi.org/10.1080/02699930441000445
30. Laukka, P., Neiberg, D., & Elfenbein, H. A. (2014). Evidence for cultural dialects in vocal emotion expression: Acoustic classification within and across five nations. Emotion, 14, 445–449. http://dx.doi.org/10.1037/a0036048
31. Lieberman, M. D., Inagaki, T. K., Tabibnia, G., & Crockett, M. J. (2011). Subjective responses to emotional stimuli during labeling, reappraisal, and distraction. Emotion, 11, 468–480. http://dx.doi.org/10.1037/a0023503
32. Lyubomirsky, S., & Lepper, H. S. (1999). A measure of subjective happiness: Preliminary reliability and construct validation. Social Indicators Research, 46, 137–155. http://dx.doi.org/10.1023/A:1006824100041
33. Mehl, M. R. (2017). The electronically activated recorder (EAR): A method for the naturalistic observation of daily social behavior. Current Directions in Psychological Science, 26, 184–190. http://dx.doi.org/10.1177/0963721416680611
34. Mehl, M. R., & Conner, T. S. (Eds.). (2013). Handbook of research methods for studying daily life. New York, NY: Guilford Press.
35. Murray, H. A. (1943). Thematic apperception test. Cambridge, MA: Harvard University Press.
36. Nelson, B. W., & Allen, N. B. (2018). Extending the passive-sensing toolbox: Using smart-home technology in psychological science. Perspectives on Psychological Science, 13, 718–733. http://dx.doi.org/10.1177/1745691618776008
37. Pereira, C. (2000). Dimensions of emotional meaning in speech. ISCA Tutorial and Research Workshop (ITRW) on Speech and Emotion (pp. 25–28). Retrieved from https://www.isca-speech.org/archive_open/speech_emotion/spem_025.html
38. Pew Research Center. (2018, February 5). Mobile fact sheet. Retrieved from http://www.pewinternet.org/fact-sheet/mobile/
39. Richard, F. D., Bond, C. F., & Stokes-Zoota, J. J. (2003). One hundred years of social psychology quantitatively described. Review of General Psychology, 7, 331–363. http://dx.doi.org/10.1037/1089-2680.7.4.331
40. Robinson, M. D., & Clore, G. L. (2002). Episodic and semantic knowledge in emotional self-report: Evidence for two judgment processes. Journal of Personality and Social Psychology, 83, 198–215. http://dx.doi.org/10.1037/0022-3514.83.1.198
41. Russell, J. A., & Barrett, L. F. (1999). Core affect, prototypical emotional episodes, and other things called emotion: Dissecting the elephant. Journal of Personality and Social Psychology, 76, 805–819. http://dx.doi.org/10.1037/0022-3514.76.5.805
42. Sauter, D. A., Eisner, F., Ekman, P., & Scott, S. K. (2010). Cross-cultural recognition of basic emotions through nonverbal emotional vocalizations. Proceedings of the National Academy of Sciences of the United States of America, 107, 2408–2412. http://dx.doi.org/10.1073/pnas.0908239106
43. Sauter, D. A., & Scott, S. K. (2007). More than one kind of happiness: Can we recognize vocal expressions of different positive states? Motivation and Emotion, 31, 192–199. http://dx.doi.org/10.1007/s11031-007-9065-x
44. Scherer, K. R. (1986). Vocal affect expression: A review and a model for future research. Psychological Bulletin, 99, 143–165. http://dx.doi.org/10.1037/0033-2909.99.2.143
45. Schwartz, H. A., Eichstaedt, J. C., Kern, M. L., Dziurzynski, L., Ramones, S. M., Agrawal, M.,… Ungar, L. H. (2013). Personality, gender, and age in the language of social media: The open-vocabulary approach. PLoS ONE, 8, e73791. http://dx.doi.org/10.1371/journal.pone.0073791
46. Schwartz, H. A., Eichstaedt, J., Kern, M. L., Park, G., Sap, M., Stillwell, D.,… Ungar, L. (2014). Towards assessing changes in degree of depression through Facebook. Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality (pp. 118–125). Baltimore, MD: Association for Computational Linguistics. Retrieved from http://www.aclweb.org/anthology/W/W14/W14-3214
47. Schwarz, N. (1999). Self-reports: How the questions shape the answers. American Psychologist, 54, 93–105. http://dx.doi.org/10.1037/0003-066X.54.2.93
48. Scikit Learn Developers. (2018a). Neural network models. Retrieved from http://scikit-learn.org/stable/modules/neural_networks_supervised.html
49. Scikit Learn Developers. (2018b). Random forest classifier. Retrieved from http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html
50. Scikit Learn Developers. (2018c). Support vector machines. Retrieved from http://scikit-learn.org/stable/modules/svm.html
51. Shrout, P. E., Stadler, G., Lane, S. P., McClure, M. J., Jackson, G. L., Clavél, F. D.,… Bolger, N. (2018). Initial elevation bias in subjective reports. Proceedings of the National Academy of Sciences of the United States of America, 115, E15–E23. http://dx.doi.org/10.1073/pnas.1712277115
52. Solomon, B. C., & Vazire, S. (2016). Knowledge of identity and reputation: Do people have knowledge of others’ perceptions? Journal of Personality and Social Psychology, 111, 341–366. http://dx.doi.org/10.1037/pspi0000061
53. Strobl, C., Malley, J., & Tutz, G. (2009). An introduction to recursive partitioning: Rationale, application, and characteristics of classification and regression trees, bagging, and random forests. Psychological Methods, 14, 323–348. http://dx.doi.org/10.1037/a0016973
54. Sun, J., Schwartz, H. A., Son, Y., Kern, M. L., & Vazire, S. (2018). The language of well-being: Tracking within-person emotion fluctuations through everyday speech. Manuscript submitted for publication.
55. Sun, J., & Vazire, S. (in press). Do people know what they’re like in the moment? Psychological Science. Retrieved from https://psyarxiv.com/sg5aw/
56. Tackman, A. M., Sbarra, D. A., Carey, A. L., Donnellan, M. B., Horn, A. B., Holtzman, N. S.,… Mehl, M. R. (2018). Depression, negative emotionality, and self-referential language: A multi-lab, multi-measure, and multi-language-task research synthesis. Journal of Personality and Social Psychology. Advance online publication. http://dx.doi.org/10.1037/pspp0000187
57. Vazire, S., Wilson, R. E., Solomon, B. C., Bollich, K. L., Harris, K., Weston, S. J.,… Jackson, J. J. (2017). Personality and Interpersonal Roles (PAIRS). Study in progress. Retrieved from https://osf.io/akbfj/
58. Vogels, W. (2018, January). Voice is about to fix our love-hate relationship with machines. WIRED Magazine. Retrieved from https://www.wired.co.uk/article/google-home-amazon-alexa-apple-homepod-voicerecognition
59. Wallbott, H. G., & Scherer, K. R. (1986). How universal and specific is emotional experience? Evidence from 27 countries on five continents. Social Sciences Information, 25, 763–795. http://dx.doi.org/10.1177/053901886025004001
60. Wang, Y., & Kosinski, M. (2018). Deep neural networks are more accurate than humans at detecting sexual orientation from facial images. Journal of Personality and Social Psychology, 114, 246–257. http://dx.doi.org/10.1037/pspa0000098
61. Watson, D., & Tellegen, A. (1985). Toward a consensual structure of mood. Psychological Bulletin, 98, 219–235. http://dx.doi.org/10.1037/0033-2909.98.2.219
62. Wilson, R. E., Harris, K., & Vazire, S. (2015). Personality and friendship satisfaction in daily life: Do everyday social interactions account for individual differences in friendship satisfaction? European Journal of Personality, 29, 173–186. http://dx.doi.org/10.1002/per.1996
63. Wilson, R. E., Thompson, R. J., & Vazire, S. (2017). Are fluctuations in personality states more than fluctuations in affect? Journal of Research in Personality, 69, 110–123. http://dx.doi.org/10.1016/j.jrp.2016.06.006