Crowd-sourced Text Analysis: Reproducible and Agile Production of Political Data

American Political Science Review

Volumn 110, Issue 2, 2016, Pages 278-295

  Benoit, Kenneth ^a   Conway, Drew ^b   Lauderdale, Benjamin E ^a   Laver, Michael ^b   Mikhaylov, Slava ^c  

^a LONDON SCHOOL OF ECONOMICS   (United Kingdom)

^b NEW YORK UNIVERSITY   (United States)

^c UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84982918683     PISSN: 00030554     EISSN: 15375943     Source Type: Journal    
DOI: 10.1017/S0003055416000058     Document Type: Conference Paper

References (61)

1. Alonso, O., and R. Baeza-Yates. 2011. "Design and Implementation of Relevance Assessments Using Crowdsourcing." In Advances in Information Retrieval, eds. P. Clough, C. Foley, C. Gurrin, G. Jones, W. Kraaij, H. Lee, and V. Mudoch. Berlin: Springer.
2. Alonso, O., and S. Mizzaro. 2009. Can we get rid of TREC assessors? Using Mechanical Turk for relevance assessment. Paper read at Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation.
3. Ariely, D., W. T.Au, R. H.Bender, D. V. Budescu, C. B. Dietz, H. Gu, and G. Zauberman. 2000. "The effects of averaging subjective probability estimates between and within judges." Journal of Experimental Psychology: Applied 6 (2): 130-47.
4. Armstrong, J. S., ed. 2001. Principles of Forecasting: AHandbook for Researchers and Practitioners. New York: Springer.
5. Baker, Frank B, and Seock-Ho Kim. 2004. Item Response Theory: Parameter Estimation Techniques. Boca Raton: CRC Press.
6. Bakker, Ryan, Catherine de Vries, Erica Edwards, Liesbet Hooghe, Seth Jolly, Gary Marks, Jonathan Polk, Jan Rovny, Marco Steenbergen and Milada Vachudova. 2015. "Measuring Party Positions in Europe: The Chapel Hill Expert Survey Trend File, 1999-2010." Party Politics 21 (1): 143-52.
7. Benoit, Kenneth. 2005. "Policy Positions in Britain 2005: Results from an Expert Survey." London School of Economics.
8. Benoit, Kenneth. 2010. "Expert Survey of British Political Parties." Trinity College Dublin.
9. Benoit, Kenneth, and Michael Laver. 2006. Party Policy in Modern Democracies. London: Routledge.
10. Berinsky, A., G. Huber, andG.Lenz. 2012. "Evaluating OnlineLabor Markets for Experimental Research: Amazon.com's Mechanical Turk." Political Analysis 20 (3): 351-68.
11. Berinsky, A., M. Margolis, and M. Sances. 2014. "Separating the Shirkers from theWorkers? Making SureRespondents Pay Attention on Self-Administered Surveys." American Journal of Political Science.
12. Bohannon, J. 2011. "Social Science for Pennies." Science 334: 307.
13. Budge, Ian, Hans-DieterKlingemann, AndreaVolkens, Judith Bara, Eric Tannenbaum, Richard Fording, Derek Hearl, Hee Min Kim, Michael McDonald, and Silvia Mendes. 2001. Mapping Policy Preferences:Estimates for Parties, Electors andGovernments 1945-1998. Oxford: Oxford University Press.
14. Budge, Ian, David Robertson, and Derek Hearl. 1987. Ideology, Strategy and Party Change: Spatial Analyses of Post-War Election Programmes in 19 Democracies. Cambridge, UK: Cambridge University Press.
15. Cao, J, S. Stokes, and S. Zhang. 2010. "A Bayesian Approach to Ranking and Rater Evaluation: An Application to Grant Reviews." Journal of Educational and Behavioral Statistics 35 (2): 194-214.
16. Carpenter, B. 2008. "Multilevel Bayesian Models of CategoricalData Annotation." Unpublished manuscript.
17. Chandler, Jesse, Pam Mueller, and Gabriel Paolacci. 2014. "Nonnaïvet é among Amazon Mechanical Turk Workers: Consequences and Solutions for Behavioral Researchers." Behavior Research Methods 46 (1): 112-30.
18. Clemen, R., and R. Winkler. 1999. "Combining Probability Distributions From Experts in Risk Analysis." Risk Analysis 19 (2): 187-203.
19. Conway, Drew. 2013. "Applications of Computational Methods in Political Science." Department of Politics, New York University. Däubler, Thomas, Kenneth Benoit, Slava Mikhaylov, and
20. Michael Laver. 2012. "Natural Sentences as Valid Units for Coded Political Text." British Journal of Political Science 42 (4): 937-51.
21. Eickhoff, C., and A. de Vries. 2012. "Increasing Cheat Robustness of Crowdsourcing Tasks." Information Retrieval 15: 1-17.
22. Fox, Jean-Paul. 2010. Bayesian Item ResponseModeling: Theory and Applications: New York: Springer.
23. Galton, F. 1907. "Vox Populi." Nature (London) 75: 450-1.
24. Goodman, Joseph, Cynthia Cryder, and Amar Cheema. 2013. "Data Collection in a FlatWorld: Strengths andWeaknesses ofMechanical Turk Samples." Journal of Behavioral DecisionMaking 26 (3): 213-24.
25. Grimmer, Justin, and Brandon M Stewart. 2013. "Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts." Political Analysis 21 (3): 267-97.
26. Hambleton, Ronald K., Hariharan Swaminathan, and H. Jane Rogers. 1991. Fundamentals of Item Response Theory. Thousand Oaks, CA: Sage.
27. Hooghe, Liesbet, Ryan Bakker, Anna Brigevich, Catherine de Vries, Erica Edwards, Gary Marks, Jan Rovny, Marco Steenbergen, and Milada Vachudova. 2010. "Reliability and Validity of Measuring Party Positions: The Chapel Hill Expert Surveys of 2002 and 2006." European Journal of Political Research. 49 (5): 687-703.
28. Horton, J., D. Rand, and R. Zeckhauser. 2011. "The Online Laboratory: Conducting Experiments in a Real Labor Market." Experimental Economics 14: 399-425.
29. Hsueh, P., P. Melville, and V. Sindhwani. 2009. "Data Quality from Crowdsourcing:AStudy of Annotation Selection Criteria." Paper read at Proceedings of theNAACLHLT2009Workshop on Active Learning for Natural Language Processing.
30. Ipeirotis, Panagiotis G., Foster Provost, Victor S. Sheng, and Jing Wang. 2014. "Repeated Labeling using Multiple Noisy Labelers." Data Mining and Knowledge Discovery 28 (2): 402-41.
31. Jones, Frank R., and Bryan D. Baumgartner. 2013. "Policy Agendas Project."
32. Kapelner, A., and D. Chandler. 2010. "Preventing Satisficing in Online Surveys: A 'Kapcha' to Ensure Higher Quality Data." Paper read at TheWorld's First Conference on the Future ofDistributed Work (CrowdConf 2010).
33. King, Gary. 1995. "Replication, Replication." PS: Political Science & Politics 28 (03): 444-52.
34. Klingemann, Hans-Dieter, Richard I. Hofferbert, and Ian Budge. 1994. Parties, Policies, and Democracy. Boulder: Westview Press.
35. Klingemann, Hans-Dieter, AndreaVolkens, Judith Bara, Ian Budge, and Michael McDonald. 2006. Mapping Policy Preferences II: Estimates for Parties, Electors, and Governments in Eastern Europe, European Union and OECD 1990-2003. Oxford: Oxford University Press.
36. Krippendorff, Klaus. 2013. Content Analysis: An Introduction to Its Methodology. 3rd ed. Thousand Oaks, CA: Sage.
37. Laver, M. 1998. "Party Policy in Britain 1997: Results from an Expert Survey." Political Studies 46 (2): 336-47.
38. Laver, Michael, and Ian Budge. 1992. Party Policy and Government Coalitions. New York: St. Martin's Press.
39. Laver, Michael, and W. Ben Hunt. 1992. Policy and Party Competition. New York: Routledge.
40. Lawson, C., G. Lenz, A. Baker, and M. Myers. 2010. "Looking Like a Winner: Candidate Appearance and Electoral Success in New Democracies." World Politics 62 (4): 561-93.
41. Lin, L. 1989. "A Concordance Correlation Coefficient to Evaluate Reproducibility." Biometrics 45: 255-68.
42. Lin, L. 2000. "A Note on the Concordance Correlation Coefficient." Biometrics 56: 324-5.
43. Lord, Frederic. 1980. Applications of Item Response Theory to Practical Testing Problems. New York: Routledge.
44. Lyon, Aidan, and Eric Pacuit. 2013. "The Wisdom of Crowds: Methods of Human Judgement Aggregation." In Handbook of Human Computation, ed. P. Michelucci. New York: Springer.
45. Mason, W., and S. Suri. 2012. "Conducting Behavioral Research on Amazon's Mechanical Turk." Behavior Research Methods 44 (1): 1-23.
46. Mikhaylov, Slava, Michael Laver, andKenneth Benoit. 2012. "Coder Reliability and Misclassification in Comparative Manifesto Project Codings." Political Analysis 20 (1): 78-91.
47. Nowak, S., and S. Rger. 2010. "How Reliable are Annotations via Crowdsourcing? A Study about Inter-Annotator Agreement for Multi-Label Image Annotation." Paper read at The 11th ACM International Conference on Multimedia Information Retrieval, 29-31 March 2010, Philadelphia.
48. Paolacci, Gabriel, Jesse Chandler, and Panagiotis Ipeirotis. 2010. "Running Experiments on Amazon Mechanical Turk." Judgement and Decision Making 5: 411-9.
49. Quoc Viet Hung, Nguyen, Nguyen Thanh Tam, Lam Ngoc Tran, and Karl Aberer. 2013. "An Evaluation of Aggregation Techniques in Crowdsourcing." InWeb Information Systems Engineering-WISE 2013, eds. X. Lin, Y. Manolopoulos, D. Srivastava, and G. Huang: Berlin: Springer.
50. Raykar, V. C., S.Yu, L. H. Zhao, G. H.Valadez, C. Florin, L. Bo-goni, and L. Moy. 2010. "Learning from Crowds." Journal of Machine Learning Research 11: 1297-322.
51. Ruedin, Didier. 2013. "Obtaining Party Positions on Immigration in Switzerland: Comparing Different Methods." Swiss Political Science Review 19 (1): 84-105.
52. Ruedin, Didier, andLaura Morales. 2012. "Obtaining Party Positions on Immigration from Party Manifestos." Paper presented at the Elections, Public Opinion and Parties (EPOP)conference, Oxford, 7 Sept 2012.
53. Schwarz, Daniel, Denise Traber, and Kenneth Benoit. Forthcoming. "Estimating Intra-Party Preferences: Comparing Speeches to Votes." Political Science Research and Methods.
54. Sheng, V., F. Provost, and Panagiotis Ipeirotis. 2008. "Get Another Label? Improving Data Quality and Data Mining using Multiple, Noisy Labelers." Paper read at Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.
55. Snow, R., B. O'Connor, D. Jurafsky, and A. Ng. 2008. "Cheap and Fast - But is it Good?: Evaluating Non-expert Annotations for Natural Language Tasks." Paper read at Proceedings of the Conference on Empirical Methods in Natural Language Processing.
56. Surowiecki, J. 2004. The Wisdom of Crowds.NewYork:W.W.Norton & Company, Inc.
57. Turner, Brandon M., Mark Steyvers, Edgar C. Merkle, David V. Budescu, and Thomas S. Wallsten. 2014. "Forecast Aggregation via Recalibration." Machine Learning 95 (3): 261-89.
58. Volkens, Andrea. 2001. "Manifesto Coding Instructions, 2nd revised ed." In Discussion Paper (2001), ed. W. Berlin, p. 96, Andrea Volkens Berlin:Wissenschaftszentrum Berlin f ür Sozialforschung gGmbH (WZB).
59. Welinder, P., S. Branson, S. Belongie, and P. Perona. 2010. "The MultidimensionalWisdom of Crowds." Paper read at Advances in Neural Information Processing Systems 23 (NIPS 2010).
60. Welinder, P., and P. Perona. 2010. "Online Crowdsourcing: Rating Annotators and Obtaining Cost-effective Labels." Paper read at IEEE Conference on Computer Vision and Pattern Recognition Workshops (ACVHL).
61. Whitehill, J., P. Ruvolo, T. Wu, J. Bergsma, and J. Movellan. 2009. "Whose Vote should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise." Paper read at Advances in Neural Information Processing Systems 22 (NIPS 2009).