Crowdsourcing a word-emotion association lexicon

Computational Intelligence

Volumn 29, Issue 3, 2013, Pages 436-465

  Mohammad, Saif M ^a   Turney, Peter D ^a  

^a NATIONAL RESEARCH COUNCIL CANADA   (Canada)

Author keywords

affect; crowdsourcing; emotion lexicon; emotions; Mechanical Turk; polarity; polarity lexicon; semantic orientation; sentiment analysis; word emotion associations

Indexed keywords

AFFECT; CROWDSOURCING; EMOTION LEXICON; EMOTIONS; MECHANICAL TURKS; POLARITY; POLARITY LEXICON; SEMANTIC ORIENTATION; SENTIMENT ANALYSIS;

ARTIFICIAL INTELLIGENCE; COMPUTATIONAL METHODS;

SEMANTICS;

EID: 84881408290     PISSN: 08247935     EISSN: 14678640     Source Type: Journal    
DOI: 10.1111/j.1467-8640.2012.00460.x     Document Type: Conference Paper

References (103)

1. Adamic, L. A., J. Zhang, E. Bakshy, and, M. S. Ackerman,. 2008. Knowledge sharing and Yahoo answers: Everyone knows something. In Proceedings of the 17th International Conference on World Wide Web, WWW '08. ACM, New York, NY, pp. 665-674.
2. Alm, C. O., D. Roth, and, R. Sproat,. 2005. Emotions from text: Machine learning for text-based emotion prediction. In Proceedings of the Joint Conference on Human Language Technology/Empirical Methods in Natural Language Processing (HLT/EMNLP-2005), Vancouver, Canada, pp. 579-586.
3. Aman, S., and, S. Szpakowicz,. 2007. Identifying expressions of emotion in text. In Text, Speech and Dialogue. Edited by, V. Matoušek, and, P. Mautner, Vol. 4629 of Lecture Notes in Computer Science. Springer: Berlin/Heidelberg, pp. 196-205.
4. Aristotle. 1913. Physiognomonica. In The Works of Aristotle. Edited by, W. D. Ross,. Clarendon: Oxford, England, pp. 805-813. Translated by T. Loveday and E. S. Forster.
5. Bellegarda, J., 2010. Emotion analysis using latent affective folding and embedding. In Proceedings of the NAACL-HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, Los Angeles, CA.
6. Bernard, J., editor, 1986. The Macquarie Thesaurus. Macquarie Library: Sydney, Australia.
7. Boucouvalas, A. C., 2002. Real time text-to-emotion engine for expressive Internet communication. Emerging Communication: Studies on New Technologies and Practices in Communication, 5: 305-318.
8. Bougie, J. R. G., R. Pieters, and, M. Zeelenberg,. 2003. Angry customers don't come back, they get back: The experience and behavioral implications of anger and dissatisfaction in services. Open access publications from Tilburg University, Tilburg, the Netherlands.
9. Bradley, M., and, P. Lang,. 1999. Affective norms for English words ANEW: Instruction manual and affective ratings. Technical Report, C-1, The Center for Research in Psychophysiology, University of Florida, Gainesville, FL.
10. Brants, T., and, A. Franz,. 2006. Web 1T 5-Gram Version 1. Linguistic Data Consortium.
11. Breazeal, C., and, R. Brooks,. 2004. Robot emotions: A functional perspective. In Who Needs Emotions. Oxford University Press: Oxford.
12. Brennan, R. L., and, D. J. Prediger,. 1981. Coefficient Kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement, 41 (3): 687-699.
13. Callison-Burch, C., 2009. Fast, cheap and creative: Evaluating translation quality using Amazon's mechanical turk. In Proceedings of the Conference on Empirical Methods in Natural Language Processing EMNLP-2009, Singapore, pp. 286-295.
14. Cohen, J., 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20 (1): 37-46.
15. Cohen, S. H,. 2003. Maximum difference scaling: Improved measures of importance and preference for segmentation. In Sawtooth Software Conference Proceedings, Sawtooth Software, Inc., pp. 61-74.
16. Dabholkar, P. A., C. D. Shepherd, and, D. I. Thorpe,. 2000. A comprehensive framework for service quality: An investigation of critical conceptual and measurement issues through a longitudinal study. Journal of Retailing, 76 (2): 139-173.
17. Darwin, C., 1872. The Expressions of the Emotions in Man and Animals. John Murray: London, UK.
18. Davenport, T. H., J. G. Harris, and, A. K. Kohli,. 2001. How do they know their customers so well? MIT Sloan Management Review, 42 (2): 63-73.
19. Díaz, A. B. C., and, F. J. M. RuÃz,. 2002. The consumers reaction to delays in service. International Journal of Service Industry Management, 13 (2): 118-140.
20. Dubé, L., and, M. Maute,. 1996. The antecedents of brand switching, brand loyalty and verbal responses to service failure. Advances in Services Marketing and Management, 5: 127-151.
21. Ekman, P., 1992. An argument for basic emotions. Cognition and Emotion, 6 (3): 169-200.
22. Ekman, P., 2005. Emotion in the Human Face. Oxford University Press: Oxford, UK.
23. Ekman, P., and, W. V. Friesen,. 2003. Unmasking the Face: A Guide to Recognizing Emotions from Facial Expressions. Malor Books: Cambridge, MA.
24. Elfenbein, H. A., and, N. Ambady,. 1994. Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. Psychological Bulletin, 115: 102-141.
25. Elliott, C., 1992. The affective reasoner: A process model of emotions in a multi-agent system. Ph.D. Thesis, Institute for the Learning Sciences, Northwestern University, Evanston, IL.
26. Fleiss, J. L,. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76 (5): 378-382.
27. Folkes, V. S., S. Koletsky, and, J. L. Graham,. 1987. A field study of causal inferences and consumer reaction: The view from the airport. Journal of Consumer Research: An Interdisciplinary Quarterly, 13 (4): 534-539.
28. Francisco, V., and, P. Gervás,. 2006. Automated mark up of affective information in English texts. In Text, Speech and Dialogue. Edited by, P. Sojka, I. Kopecek, and, K. Pala,. Vol. 4188 of Lecture Notes in Computer Science. Springer: Berlin/Heidelberg, pp. 375-382.
29. Genereux, M., and, R. P. Evans,. 2006. Distinguishing affective states in weblogs. In AAAI-2006 Spring Symposium on Computational Approaches to Analysing Weblogs, Stanford, CA, pp. 27-29.
30. Goyal, A., E. Riloff, H. Daume III, and, N. Gilbert,. 2010. Toward plot units: Automatic affect state analysis. In Proceedings of the NAACL-HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, Los Angeles, CA.
31. Gross, J. J., 1998. The emerging field of emotion regulation: An integrative review. Review of General Psychology, 2 (3): 271-299.
32. Guo, K., C. Hall, S. K. M. Hall, and, D. Mills,. 2007. Left gaze bias in human infants, rhesus monkeys, and domestic dogs. Animal Cognition, 12: 409-418.
33. Harris, L. C., and, M. M. H. Goode,. 2004. The four levels of loyalty and the pivotal role of trust: A study of online service dynamics. Journal of Retailing, 80 (2): 139-158.
34. Hollinger, G., Y. Georgiev, A. Manfredi, B. A. Maxwell, Z. A. Pezzementi, and, B. Mitchell,. 2006. Design of a social mobile robot using emotion-based decision mechanisms. In 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3093-3098.
35. Holsti, O. R., 1969. Content analysis for the social sciences and humanities. Addison-Wesley: Reading, MA.
36. Holzman, L. E., and, W. M. Pottenger,. 2003. Classification of emotions in internet chat: An application of machine learning using speech phonemes. Technical report, Lehigh University, Bethlehem, PA.
37. Howe, J., and, M. Robinson,. 2006. Crowdsourcing: A definition. In Crowdsourcing: Tracking the Rise of the Amateur. Weblog.
38. James, W., 1884. What is an emotion? Mind, 9: 188-205.
39. John, D., A. C. Boucouvalas, and, Z. Xu,. 2006. Representing emotional momentum within expressive Internet communication. In Proceedings of the 24th IASTED International Conference on Internet and Multimedia Systems and Applications. ACTA Press: Anaheim, CA, pp. 183-188.
40. Kalra, A., and, K. Karahalios,. 2005. Texttone: Expressing emotion through text. In Human-Computer Interaction-INTERACT 2005. Edited by, M. F. Costabile, and, F. Patern,. Vol. 3585 of Lecture Notes in Computer Science. Springer: Berlin/Heidelberg, pp. 966-969.
41. Knautz, K., T. Siebenlist, and, W. G. Stock,. 2010. Memose: Search engine for emotions in multimedia documents. In Proceeding of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval,SIGIR '10, ACM, New York, NY, pp. 791-792.
42. Kǒvecses, Z,. 2003. Metaphor and Emotion: Language, Culture, and Body in Human Feeling Studies in Emotion and Social Interaction. Cambridge University Press: Cambridge, UK.
43. Landis, J. R., and, G. G. Koch,. 1977. The measurement of observer agreement for categorical data. Biometrics, 33 (1): 159-174.
44. Lazarus, R. S., 1984. On the primacy of cognition. American Psychologist, 39 (2): 124-129.
45. Lazarus, R. S,. 2000. The cognition-emotion debate: A bit of history. In Handbook of Cognition and Emotion. Edited by, M. Lewis, and, J. Haviland-Jones,. Guilford Press: New York, pp. 1-20.
46. Lehrer, A., 1974. Semantic Fields and Lexical Structure. North-Holland: Amsterdam, the Netherlands.
47. Litman, D. J., and Forbes-Riley. 2004. Predicting student emotions in computer-human tutoring dialogues. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, ACL '04, USA. Association for Computational Linguistics, Morristown, NJ.
48. Liu, H., H. Lieberman, and, T. Selker,. 2003. A model of textual affect sensing using real-world knowledge. In Proceedings of the 8th International Conference on Intelligent User Interfaces, IUI '03. ACM, New York, NY, pp. 125-132.
49. Louviere, J. J., 1991. Best-worst scaling: A model for the largest difference judgments. Technical report, University of Alberta, Edmonton, Canada.
50. Louviere, J. J., and, A. Finn,. 1992. Determining the appropriate response to evidence of public concern: The case of food safety. Journal of Public Policy and Marketing, 11 (2): 12-25.
51. Ma, C., H. Prendinger, and, M. Ishizuka,. 2005. Emotion estimation and reasoning based on affective textual interaction. In First International Conference on Affective Computing and Intelligent Interaction ACII-2005, Beijing, China, pp. 622-628.
52. Masson, J. M., 1996. When Elephants Weep: The Emotional Lives of Animals. Delta: New York.
53. Matykiewicz, P., W. Duch, and, J. P. Pestian,. 2009. Clustering semantic spaces of suicide notes and newsgroups articles. In Proceedings of the Workshop on Current Trends in Biomedical Natural Language Processing, BioNLP '09, USA. Association for Computational Linguistics: Stroudsburg, PA, pp. 179-184.
54. Maute, M. F., and, W. J. Forrester,. 1993. The structure and determinants of consumer complaint intentions and behavior. Journal of Economic Psychology, 14 (2): 219-247.
55. Mihalcea, R., and, H. Liu,. 2006. A corpus-based approach to finding happiness. In AAAI-2006 Spring Symposium on Computational Approaches to Analysing Weblogs, pp. 139-144.
56. Mohammad, S. M., 2011a. Even the abstract have colour: Consensus in wordâ colour associations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, OR.
57. Mohammad, S. M., 2011b. From once upon a time to happily ever after: Tracking emotions in novels and fairy tales. In Proceedings of the ACL 2011 Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities LaTeCH, Portland, OR.
58. Mohammad, S. M., and, P. D. Turney,. 2010. Emotions evoked by common words and phrases: Using Mechanical Turk to create an emotion lexicon. In Proceedings of the NAACL-HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, Los Angeles, CA.
59. Mohammad, S. M., and, T. W. Yang,. 2011. Tracking sentiment in mail: How genders differ on emotional axes. In Proceedings of the ACL 2011 Workshop on Computational Approaches to Subjectivity and Sentiment Analysis WASSA, Portland, OR.
60. Mohammad, S. M., C. Dunne, and, B. Dorr,. 2009. Generating high-coverage semantic orientation lexicons from overtly marked words and a thesaurus. In Proceedings of Empirical Methods in Natural Language Processing EMNLP-2009, Singapore, pp. 599-608.
61. Neviarouskaya, A., H. Prendinger, and, M. Ishizuka,. 2009. Compositionality principle in recognition of fine-grained emotions from text. In Proceedings of the Third International Conference on Weblogs and Social Media ICWSM-09, San Jose, CA, pp. 278-281.
62. Neviarouskaya, A., H. Prendinger, and, M. Ishizuka,. 2010. Affect analysis model: Novel rule-based approach to affect sensing from text. Natural Language Engineering, 17 (Part 1): 95-135.
63. Nowlis, V., and, H. H. Nowlis,. 2001. The description and analysis of mood. Annals of the New York Academy of Sciences, 65 (4): 345-355.
64. Oliver, R. L,. 1997. Satisfaction a behavioral perspective on the consumer. McGraw-Hill: New York.
65. Ortony, A., and, T. J. Turner,. 1990. What's basic about basic emotions? Psychological Review, 97: 315-331.
66. Ortony, A., G. L. Clore, and, A. Collins,. 1988. The Cognitive Structure of Emotions. Cambridge University Press: Cambridge, UK.
67. Osgood, C. E., and, E. G. Walker,. 1959. Motivation and language behavior: A content analysis of suicide notes. Journal of Abnormal and Social Psychology, 59 (1): 58-67.
68. Pang, B., and, L. Lee,. 2008. Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2 (1-2): 1-135.
69. Parrot, W., 2001. Emotions in Social Psychology. Psychology Press: Philadelphia, PA.
70. Pearl, L., and, M. Steyvers,. 2010. Identifying emotions, intentions, and attitudes in text using a game with a purpose. In Proceedings of the NAACL-HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, Los Angeles, CA.
71. Perreault, W. D., and, L. E. Leigh,. 1989. Reliability of nominal data based on qualitative judgments. Journal of Marketing Research, 26: 135-148.
72. Pestian, J. P., P. Matykiewicz, and, J. Grupp-Phelan,. 2008. Using natural language processing to classify suicide notes. In Proceedings of the Workshop on Current Trends in Biomedical Natural Language Processing, BioNLP '08, USA. Association for Computational Linguistics: Stroudsburg, PA, pp. 96-97.
73. Plutchik, R., 1962. The Emotions. Random House: New York.
74. Plutchik, R., 1980. A general psychoevolutionary theory of emotion. Emotion: Theory, Research, and Experience, 1 (3): 3-33.
75. Plutchik, R., 1985. On emotion: The chicken-and-egg problem revisited. Motivation and Emotion, 9 (2): 197-200.
76. Plutchik, R., 1994. The Psychology and Biology of Emotion. Harper Collins: New York.
77. Rashid, R., J. Aitken, and, D. Fels,. 2006. Expressing emotions using animated text captions. In Computers Helping People with Special Needs. Edited by, K. Miesenberger, J. Klaus, W. Zagler, and, A. Karshmer,. Vol. 4061 of Lecture Notes in Computer Science. Springer: Berlin/Heidelberg, pp. 24-31.
78. Ravaja, N., T. Saari, M. Turpeinen, J. Laarni, M. Salminen, and, M. Kivikangas,. 2006. Spatial presence and emotions during video game playing: Does it matter with whom you play? Presence: Teleoperators and Virtual Environments, 15 (4): 381-392.
79. Read, J., 2004. Recognising affect in text using pointwise-mutual information, Ph.D. Thesis, Department of Informatics, University of Sussex, Brighton, UK.
80. Reichheld, F. F., and, P. Schefter,. 2000. E-loyalty: Your secret weapon on the web. Harvard Business Review, 78 (2): 105-113.
81. Rentoumi, V., G. Giannakopoulos, V. Karkaletsis, and, G. A. Vouros,. 2009. Sentiment analysis of figurative language using a word sense disambiguation approach. In Proceedings of the International Conference RANLP-2009, Borovets, Bulgaria, pp. 370-375.
82. Richins, M., 1987. A multivariate analysis of responses to dissatisfaction. Journal of the Academy of Marketing Science, 15: 24-31.
83. Richins, M. L., 1984. Word of mouth communication as negative information. Advances in Consumer Research, 11: 697-702.
84. Russell, J. A., 1994. Is there universal recognition of emotion from facial expression?A review of the cross-cultural studies. Psychological Bulletin, 115: 102 -141.
85. Sautera, D. A., F. Eisner, P. Ekman, and, S. K. Scott,. 2010. Cross-cultural recognition of basic emotions through nonverbal emotional vocalizations. Proceedings of the National Academy of Sciences, 107 (6): 2408-2412.
86. Scherer, K. R., 1982. Emotion as a process: Function, origin and regulation. Social Science Information, 21 (4-5): 555-570.
87. Scherer, K. R., 1984. Emotion as a multicomponent process: A model and some cross-cultural data. Review of Personality and Social Psychology, 5: 37-63.
88. Scott, W. A., 1955. Reliability of content analysis. Public Opinion Quarterly, 19 (3): 321-325.
89. Shankar, V., G. L. Urban, and, F. Sultan,. 2002. Online trust: A stakeholder perspective, concepts, implications, and future directions. The Journal of Strategic Information Systems, 11 (3-4): 325-344.
90. Shaver, P., J. Schwartz, D. Kirson, and, G. O'Connor,. 1987. Emotion knowledge: Further exploration of a prototype approach. Journal of Personality and Social Psychology, 52: 1061-1086.
91. Singh, J., 1988. Consumer complaint intentions and behavior: Definitional and taxonomical issues. The Journal of Marketing, 52 (1): 93-107.
92. Snow, R., B. O'Connor, D. Jurafsky, and, A. Ng,. 2008. Cheap and fast-but is it good? Evaluating nonexpert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing EMNLP-2008, Waikiki, HI, pp. 254-263.
93. Steunebrink, B. R,. 2010. The logical structure of emotions, Ph.D. Thesis, Dutch Research School for Information and Knowledge Systems.
94. Stewart, K. J., 2003. Trust transfer on the World Wide Web. Organization Science, 14: 5-17.
95. Stone, P., D. C. Dunphy, M. S. Smith, and, D. M. Ogilvie,. 1966. The General Inquirer: A Computer Approach to Content Analysis. The MIT Press: Cambridge, MA.
96. Strapparava, C., and, A. Valitutti,. 2004. Wordnet-Affect: An affective extension of WordNet. In Proceedings of the 4th International Conference on Language Resources and Evaluation LREC-2004, Lisbon, Portugal, pp. 1083-1086.
97. Subasic, P., and, A. Huettner,. 2001. Affect analysis of text using fuzzy semantic typing. IEEE Transactions on Fuzzy Systems, 9 (4): 483-496.
98. Turney, P., and, M. Littman,. 2003. Measuring praise and criticism: Inference of semantic orientation from association. ACM Transactions on Information Systems, 21 (4): 315-346.
99. Velásquez, J. D., 1997. Modeling emotions and other motivations in synthetic agents. In Proceedings of the Fourteenth National Conference on Artificial Intelligence and Ninth Conference on Innovative Applications of Artificial Intelligence, AAAI'97/IAAI'97. AAAI Press: Cambridge, MA, pp. 10-15.
100. Weiner, B., 1985. An attributional theory of achievement motivation and emotion. Psychological Review, 92 (4): 548-573.
101. Wiebe, J. M., 1994. Tracking point of view in narrative. Computational Linguistics, 20 (2): 233-287.
102. Zajonc, R. B., 1984. On the primacy of affect. American Psychologist, 39 (2): 117-123.
103. Zhe, X., and, A. Boucouvalas,. 2002. Text-to-emotion engine for real time Internet communication. In International Symposium on CSNDSP, pp. 164-168.