Multilingual twitter sentiment classification: The role of human annotators

PLoS ONE

Volumn 11, Issue 5, 2016, Pages

  Mozetič, Igor ^a   Grčar, Miha ^a   Smailović, Jasmina ^a  

^a JO EF STEFAN INSTITUTE   (Slovenia)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; HUMAN; LANGUAGE; QUANTITATIVE ANALYSIS; INTERNET; MULTILINGUALISM;

HUMANS; INTERNET; MULTILINGUALISM;

EID: 84994503001     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0155036     Document Type: Article

References (44)

1. Liu B. Sentiment analysis and opinion mining. Synthesis Lectures on Human Language Technologies. 2012; 5(1):1–167. doi: 10.2200/S00416ED1V01Y201204HLT016
2. Liu B. Sentiment Analysis: Mining Opinions, Sentiments, and Emotions. Cambridge University Press; 2015.
3. Dodds PS, Clark EM, Desu S, Frank MR, Reagan AJ, Williams JR, et al. Human language reveals a universal positivity bias. Proceedings of the National Academy of Sciences. 2015; 112(8):2389–2394. Available from: http://dx.doi.org/10.1073/pnas.1411678112. doi: 10.1073/pnas.1411678112
4. Baccianella S, Esuli A, Sebastiani F. SentiWordNet 3.0: An enhanced lexical resource for sentiment analysis and opinion mining. In: LREC. vol. 10; 2010. p. 2200–2204.
5. Martínez-Cámara E, Martín-Valdivia MT, Urena-López LA, Montejo-Ráez AR. Sentiment analysis in Twitter. Natural Language Engineering. 2014; 20(1):1–28.
6. Kolchyna, O, Souza, TTP, Treleaven, PC, Aste, T. Twitter Sentiment Analysis: Lexicon Method, Machine Learning Method and Their Combination. arXiv preprint arXiv:150700955. 2015;.
7. Krippendorff K. Content Analysis, An Introduction to Its Methodology. 3rd ed. Thousand Oaks, CA: Sage Publications; 2012.
8. Zollo F, Kralj Novak P, Del Vicario M, Bessi A, Mozetič I, Scala A, et al. Emotional dynamics in the age of misinformation. PLoS ONE. 2015; 10(9):e0138740. Available from: http://dx.doi.org/10.1371/journal.pone.0138740. doi: 10.1371/journal.pone.0138740 PMID: 26422473
9. Ranco G, Aleksovski D, Caldarelli G, Grčar M, Mozetič I. The effects of Twitter sentiment on stock price returns. PLoS ONE. 2015; 10(9):e0138441. Available from: http://dx.doi.org/10.1371/journal.pone. 0138441. doi: 10.1371/journal.pone.0138441 PMID: 26390434
10. Sluban B, Smailović J, Battiston S, Mozetič I. Sentiment leaning of influential communities in social networks. Computational Social Networks. 2015; 2(9):1–21. Available from: http://dx.doi.org/10.1186/ s40649-015-0016-5.
11. Kralj Novak P, Smailović J, Sluban B, Mozetič I. Sentiment of emojis. PLoS ONE. 2015; 10(12): e0144296. Available from: http://dx.doi.org/10.1186/s40649-015-0016-5. doi: 10.1371/journal.pone. 0144296 PMID: 26641093
12. Mooney CZ, Duval RD. Bootstrapping: A Nonparametric Approach to Statistical Inference. Thousand Oaks, CA: Sage Publications; 1993.
13. Vapnik VN. The Nature of Statistical Learning Theory. Springer; 1995.
14. Russell S, Norvig P. Artificial Intelligence: A Modern Approach. 2nd ed. Prentice Hall; 2003.
15. Friedman M. A comparison of alternative tests of significance for the problem of m rankings. Annals of Mathematical Statistics. 1940; 11:86–92.
16. Nemenyi PB. Distribution-free multiple comparisons. Princeton University; 1963.
17. Saif H, Fernández M, He Y, Alani H. Evaluation datasets for Twitter sentiment analysis: A survey and a new dataset, the STS-Gold. In: 1st Intl. Workshop on Emotion and Sentiment in Social and Expressive Media: Approaches and Perspectives from AI (ESSEM); 2013.
18. dos Santos CN, Gatti M. Deep convolutional neural networks for sentiment analysis of short texts. In: Proc. 25th Intl. Conf. on Computational Linguistics (COLING), Dublin, Ireland; 2014.
19. dos Santos CN. Think positive: Towards Twitter sentiment analysis from scratch. In: Proc. 8th Intl. Workshop on Semantic Evaluation (SemEval); 2014. p. 647–651.
20. Tang D, Wei F, Qin B, Liu T, Zhou M. Coooolll: A deep learning system for Twitter sentiment classification. In: Proc. 8th Intl. Workshop on Semantic Evaluation (SemEval); 2014. p. 208–212.
21. Ghosh A, Li G, Veale T, Rosso P, Shutova E, Barnden J, et al. SemEval-2015 task 11: Sentiment analysis of figurative language in Twitter. In: Proc. 9th Intl. Workshop on Semantic Evaluation (SemEval); 2015. p. 470–478.
22. Mohammad SM, Kiritchenko S. Using hashtags to capture fine emotion categories from tweets. Computational Intelligence. 2015; 31(2):301–326. doi: 10.1111/coin.12024
23. Van Rijsbergen CJ. Information Retrieval. Butterworth; 1979.
24. Kiritchenko S, Zhu X, Mohammad SM. Sentiment analysis of short informal texts. Journal of Artificial Intelligence Research. 2014;p. 723–762.
25. Gaudette L, Japkowicz N. Evaluation methods for ordinal classification. In: Advances in Artificial Intelligence. Springer; 2009. p. 207–210.
26. Go A, Bhayani R, Huang L. Twitter sentiment classification using distant supervision. CS224N Project Report, Stanford. 2009;p. 1–12.
27. Rosenthal S, Nakov P, Kiritchenko S, Mohammad SM, Ritter A, Stoyanov V. SemEval-2015 task 10: Sentiment Analysis in Twitter. In: Proc. 9th Intl. Workshop on Semantic Evaluation (SemEval); 2015. p. 451–463.
28. Haldenwang N, Vornberger O. Sentiment Uncertainty and Spam in Twitter Streams and Its Implications for General Purpose Realtime Sentiment Analysis. In: Proc. Intl. Conf. of the German Society for Computational Linguistics and Language Technology (GSCL); 2015. p. 157–159.
29. Saif H, He Y, Fernandez M, Alani H. Semantic patterns for sentiment analysis of Twitter. In: The Semantic Web (ISWC). Springer; 2014. p. 324–340.
30. Parikh R, Movassate M. Sentiment analysis of user-generated Twitter updates using various classification techniques. CS224N Final Report. 2009;p. 1–18.
31. Pak A, Paroubek P. Twitter as a Corpus for Sentiment Analysis and Opinion Mining. In: LREC. vol. 10; 2010. p. 1320–1326.
32. Asiaee T A, Tepper M, Banerjee A, Sapiro G. If you are happy and you know it. . . tweet. In: Proc. 21st ACM Intl. Conf. on Information and Knowledge Management. ACM; 2012. p. 1602–1606.
33. Saif H, He Y, Alani H. Semantic sentiment analysis of Twitter. In: The Semantic Web (ISWC). vol. 7649 of Lecture Notes in Computer Science. Springer; 2012. p. 508–524.
34. Barbosa L, Feng J. Robust sentiment detection on Twitter from biased and noisy data. In: Proc. 23rd Intl. Conf. on Computational Linguistics: Posters. ACL; 2010. p. 36–44.
35. Jiang L, Yu M, Zhou M, Liu X, Zhao T. Target-dependent Twitter sentiment classification. In: Proc. 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. ACL; 2011. p. 151–160.
36. Davidov D, Tsur O, Rappoport A. Enhanced sentiment learning using Twitter hashtags and smileys. In: Proc. 23rd Intl. Conf. Computational Linguistics: Posters. ACL; 2010. p. 241–249.
37. Kouloumpis E, Wilson T, Moore J. Twitter sentiment analysis: The good the bad and the OMG! In: Proc. Intl. Conf. on Weblogs and Social Media (ICWSM). vol. 11; 2011. p. 538–541.
38. Hsu C, Lin C. A comparison of methods for multiclass Support Vector Machines. IEEE Transactions on Neural Networks. 2002; 13:415–425.
39. Grčar M. Mining text-enriched heterogeneous information networks. PhD thesis, Jožef Stefan International Postgraduate School, Ljubljana, Slovenia; 2015.
40. Smailović J. Sentiment analysis in streams of microblogging posts. PhD thesis, Jožef Stefan International Postgraduate School, Ljubljana, Slovenia; 2014.
41. Smailović J, Kranjc J, Grčar M, Žnidaršič M, Mozetič I. Monitoring the Twitter sentiment during the Bulgarian elections. In: Proc. IEEE Intl. Conf. on Data Science and Advanced Analytics. IEEE; 2015. p. 1–10.
42. Bergmeir C, Benítez JM. On the Use of Cross-validation for Time Series Predictor Evaluation. Information Sciences. 2012; 191:192–213. Available from: http://dx.doi.org/10.1016/j.ins.2011.12.028. doi: 10. 1016/j.ins.2011.12.028
43. Martineau J, Finin T. Delta TFIDF: An improved feature space for sentiment analysis. In: Proc. 3rd AAAI Intl. Conf. on Weblogs and Social Media (ICWSM); 2009. p. 258–261.
44. Demšar J. Statistical comparison of classifiers over multiple data sets. Journal of Machine Learning Research. 2006; 7:1–30.