TopicCheck: Interactive alignment for assessing topic model stability

NAACL HLT 2015 - 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference

Volumn , Issue , 2015, Pages 175-184

  Chuang, Jason ^c,d   Roberts, Margaret E ^a   Stewart, Brandon M ^b   Weiss, Rebecca ^c   Tingley, Dustin ^b   Grimmer, Justin ^c   Heer, Jeffrey ^d  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b HARVARD UNIVERSITY   (United States)

^c STANFORD UNIVERSITY   (United States)

^d UNIVERSITY OF WASHINGTON   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BEHAVIORAL RESEARCH; COMPUTATIONAL LINGUISTICS;

ALIGNMENT ALGORITHMS; AUTOMATED CODING; COMPUTER SCIENTISTS; REPRODUCIBILITIES; RESEARCH QUESTIONS; SENSITIVITY EVALUATION; SOCIAL SCIENCE RESEARCH; SOCIAL SCIENTISTS;

RELIABILITY ANALYSIS;

EID: 84956624433     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.3115/v1/n15-1018     Document Type: Conference Paper

References (43)

1. Anima Anandkumar, Yikai Liu, Daniel J. Hsu, Dean P Foster, and Sham M Kakade. 2012. A spectral algorithm for latent dirichlet allocation. In Neural Information Processing Systems (NIPS), pages 917-925.
2. Richard A. Becker and William S. Cleveland. 1987. Brushing scatterplots. Technometrics, 29(2):127-142.
3. Bernard Berelson. 1952. Content analysis in communication research. Free Press.
4. David M. Blei and John D. Lafferty. 2006. Dynamic topic models. In International Conference on Machine Learning (ICML), pages 113-120.
5. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet allocation. Journal of Machine Learning Research, 3(1):993-1022.
6. David M. Blei, Thomas L. Griffiths, Michael I. Jordan, and Joshua B. Tenenbaum. 2004. Hierarchical topic models and the nested chinese restaurant process. In Neural Information Processing Systems (NIPS).
7. Allison June-Barlow Chaney and David M. Blei. 2014. Visualizing topic models. In International Conference on Weblogs and Social Media (ICWSM), pages 419-422.
8. Jonathan Chang, Jordan Boyd-Graber, Chong Wang, Sean Gerrish, and David M. Blei. 2009. Reading tea leaves: How humans interpret topic models. In Neural Information Processing Systems (NIPS), pages 288-296.
9. Jason Chuang, Christopher D. Manning, and Jeffrey Heer. 2012a. Interpretation and trust: Designing model-driven visualizations for text analysis. In Conference on Human Factors in Computing Systems (CHI), pages 443-452.
10. Jason Chuang, Christopher D. Manning, and Jeffrey Heer. 2012b. Termite: Visualization techniques for assessing textual topic models. In Advanced Visual Interfaces (AVI).
11. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20:37-46.
12. Jacob Eisenstein and Eric Xing. 2010. The CMU 2008 Political Blog Corpus. Carnegie Mellon University.
13. Antske Fokkens, Marieke van Erp, Marten Postma, Ted Pedersen, Piek Vossen, and Nuno Freire. 2013. Offspring from reproduction problems: What replication failure teaches us. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 1691-1701.
14. Justin Grimmer and Brandon M. Stewart. 2011. Text as data: The promise and pitfalls of automatic content analysis methods for political texts. Political Analysis, 21(3):267-297.
15. Justin Grimmer. 2013. Appropriators not position takers: The distorting effects of electoral incentives on congressional representation. American Journal of Political Science, 57(3):624-642.
16. Ole R. Holsti. 1969. Content analysis for the social sciences and humanities. Addison-Wesley Publishing Company.
17. Yuening Hu, Jordan Boyd-Graber, Brianna Satinoff, and Alison Smith. 2014. Interactive topic modeling. Machine Learning, 95(3):423-469.
18. Shanto Iyengar and Adam Simon. 1993. News coverage of the gulf crisis and public opinion: A study of agenda-setting, priming, and framing. Communication Research, 20(3):365-383.
19. Klaus Krippendorff. 1970. Bivariate agreement coefficients for reliability of data. In E. R. Borgatta and G. W. Bohrnstedt, editors, Sociological methodology, pages 139-150. John Wiley & Sons.
20. Klaus Krippendorff. 1989. Content analysis. In E. Barnouw, G. Gerbner, W. Schramm, T. L. Worth, and L. Gross, editors, International encyclopedia of communication. Oxford University Press.
21. Klaus Krippendorff. 2004a. Content analysis: An introduction to its methodology. Sage, 2nd edition.
22. Klaus Krippendorff. 2004b. Reliability in content analysis: Some common misconceptions and recommendations. Human Communication Research, 30(3):411-433.
23. Jey Han Lau, David Newman, and Timothy Baldwin. 2014. Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. In Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 530-539.
24. Matthew Lombard, Jennifer Snyder-Duch, and Cheryl Campanella Bracken. 2002. Content analysis in mass communication: Assessment and reporting of intercoder reliability. Human Communication Research, 28(4):587-604.
25. Andrew McCallum. 2013. MALLET: A machine learning for language toolkit. http://mallet.cs.umass.edu.
26. Maxwell E. McCombs and Donald L. Shaw. 1972. The agenda-setting function of mass media. Public Opinion Quarterly, 36(5):176-187.
27. Daniel A. McFarland, Daniel Ramage, Jason Chuang, Jeffrey Heer, and Christopher D. Manning. 2013. Differentiating language usage through topic models. Poetics: Special Issue on Topic Models and the Cultural Sciences, 41(6):607-625.
28. C. E. Osgood. 1959. The representational model and relevant research. In I. de Sola Pool, editor, Trends in content analysis, pages 33-88. University of Illinois Press.
29. Ted Pedersen. 2008. Empiricism is not a matter of faith. Computational Linguistics, 34(3):465-470.
30. Pew Research Journalism Project. 2014. News coverage index methodology. http://www.journalism.org/news index methodology/99/.
31. Kevin M. Quinn, Burt L. Monroe, Michael Colaresi, Michael H. Crespin, and Dragomir R. Radev. 2010. How to analyze political attention with minimal assumptions and costs. American Journal of Political Science, 54(1):209-228.
32. Radim Řehůrěk and Petr Sojka. 2010. Software framework for topic modelling with large corpora. In LREC Workshop on New Challenges for NLP Frameworks, pages 45-50.
33. Margaret E. Roberts, Brandon M. Stewart, Dustin Tingley, and Edoardo M. Airoldi. 2013. The structural topic model and applied social science. In NIPS Workshop on Topic Models.
34. Margaret E. Roberts, Brandon Stewart, Dustin Tingley, Chris Lucas, Jetson Leder-Luis, Bethany Albertson, Shana Gadarian, and David Rand. 2014a. Topic models for open-ended survey responses with applications to experiments. American Journal of Political Science. Forthcoming.
35. Margaret E. Roberts, Brandon M. Stewart, and Dustin Tingley. 2014b. Navigating the local modes of big data: The case of topic models. In R. Michael Alvarez, editor, Data Science for Politics, Policy and Government. In Press.
36. Michal Rosen-Zvi, Thomas Griffiths, Mark Steyvers, and Padhraic Smyth. 2004. The author-topic model for authors and documents. In Conference on Uncertainty in Artificial Intelligence (UAI), pages 487-494.
37. Benjamin M. Schmidt. 2012. Words alone: Dismantling topic models in the humanities. Journal of Digital Humanities, 2(1).
38. William A. Scott. 1995. Reliability of content analysis:: The case of nominal scale coding. Public Opinion Quarterly, 19(3):321-325.
39. Burr Settles. 2011. Closing the loop: Fast, interactive semi-supervised annotation with queries on features and instances. In Empirical Methods in Natural Language Processing (EMNLP), pages 1467-1478.
40. Robin Sibson. 1973. SLINK: an optimally efficient algorithm for the single-link cluster method. The Computer Journal, 16:30-34.
41. Keith Stevens, Philip Kegelmeyer, David Andrzejewski, and David Buttler. 2012. Exploring topic coherence over many models and many topics. In Conference on Empirical Methods on Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 952-961.
42. Hanna M. Wallach, Iain Murray, Ruslan Salakhutdinov, and David Mimno. 2009. Evaluation methods for topic models. In International Conference on Machine Learning (ICML), pages 1105-1112.
43. Roger Wimmer and Joseph Dominick. 2010. Mass Media Research: An Introduction. Cengage Learning.