Language understanding in the wild: Combining crowdsourcing and machine learning

WWW 2015 - Proceedings of the 24th International Conference on World Wide Web

Volumn , Issue , 2015, Pages 992-1002

  Simpson, Edwin ^a   Venanzi, Matteo ^b   Reece, Steven ^a   Kohli, Pushmeet ^c   Guiver, John ^c   Roberts, Stephen J ^a   Jennings, Nicholas R ^a  

^a UNIVERSITY OF OXFORD   (United Kingdom)

^b UNIVERSITY OF SOUTHAMPTON   (United Kingdom)

^c MICROSOFT RESEARCH   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

AGGREGATES; ARTIFICIAL INTELLIGENCE; BAYESIAN NETWORKS; CLASSIFICATION (OF INFORMATION); COMPUTATIONAL LINGUISTICS; LEARNING SYSTEMS; SEMANTICS; SOCIAL NETWORKING (ONLINE); WORLD WIDE WEB;

BAYESIAN APPROACHES; HUMAN COMPUTATION; LANGUAGE MODEL; LANGUAGE UNDERSTANDING; PUBLIC OPINIONS; SOCIAL MEDIA; STATE-OF-THE-ART METHODS; WEALTH OF INFORMATION;

CROWDSOURCING;

EID: 84968863325     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2736277.2741689     Document Type: Conference Paper

References (33)

1. Y. Bachrach, T. Graepel, T. Minka, and J. Guiver. How To Grade a Test Without Knowing the Answers-A Bayesian Graphical Model for Adaptive Crowdsourcing and Aptitude Testing. In Proc. of the 29th Int. Conf. on Machine Learning, pages 1183-1190. ACM, 2012.
2. R. Baeza-Yates and B. Ribeiro-Neto. Modern Information Retrieval. Addison Wesley, 1999.
3. J. Bergstra and Y. Bengio. Random Search for Hyper-Parameter Optimization. The Journal of Machine Learning Research, 13:281-305, 2012.
4. C. Bishop. Pattern Recognition and Machine Learning. Springer, 4th edition, 2006.
5. D. Butler. Crowdsourcing Goes Mainstream in Typhoon Haiyan Response. Nature News, doi:10.1038/nature.2013.14186, 2013.
6. C. Chew and G. Eysenbach. Pandemics in the Age of Twitter: Content Analysis of Tweets during the 2009 H1N1 Outbreak. PloS One, 5(11):e14118, 2010.
7. A. P. Dawid and A. M. Skene. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics), 28(1):20-28, Jan. 1979.
8. P. Donmez, J. Carbonell, and J. Schneider. A Probabilistic Framework to Learn from Multiple Annotators with Time-Varying Accuracy. In Proc. of the Int. Conf. on Data Mining, pages 826-837, 2010.
9. A. Gelfand and A. Smith. Sampling-Based Approaches to Calculating Marginal Densities. Journal of the American Statistical Association, 85(410):398-409, 1990.
10. D. J. Hand and R. J. Till. A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems. Machine learning, 45(2):171-186, 2001.
11. Z. S. Harris. Distributional Structure. Word, pages 146-162, 1954.
12. N. R. Jennings, L. Moreau, D. Nicholson, S. D. Ramchurn, S. Roberts, T. Rodden, and A. Rogers. On Human-Agent Collectives. Communications of the ACM, 2014.
13. E. Kamar, S. Hacker, and E. Horvitz. Combining Human and Machine Intelligence in Large-Scale Crowdsourcing. In Proc. of the 11th Int. Conf. on Autonomous Agents and Multiagent Systems, pages 467-474, 2012.
14. H. Kim and Z. Ghahramani. Bayesian Classifier Combination. In Proc. of the 15th Int. Conf. on Artificial Intelligence and Statistics, page 619, 2012.
15. F. Kivran-Swaine, S. Brody, and M. Naaman. Effects of Gender and Tie Strength on Twitter Interactions. First Monday, 18(9), 2013.
16. S. Kullback and R. A. Leibler. On Information and Suficiency. The Annals of Mathematical Statistics, 22(1):79-86, 1951.
17. A. Levenberg, S. Pulman, K. Moilanen, E. Simpson, and S. Roberts. Predicting Economic Indicators from Web Text Using Sentiment Composition. In Int. Journal of Computer and Communication Engineering, 2014.
18. N. Littlestone and M. Warmuth. The Weighted Majority Algorithm. In 30th Annual Symposium on Foundations of Computer Science, pages 256-261. IEEE, 1989.
19. T. Minka. Expectation Propagation for Approximate Bayesian Inference. In Proc. of the 17th Conf. on Uncertainty in Artificial Intelligence, page 362, 2001.
20. T. Minka, J. Winn, J. Guiver, and D. Knowles. Infer.NET 2.5. Microsoft Research Cambridge. See http://research. microsoft. com/infernet, 2013.
21. K. Moilanen and S. Pulman. Sentiment composition. In Proc. of the Recent Advances in Natural Language Processing Int. Conf., pages 378-382, 2007.
22. N. Morrow, N. Mock, A. Papendieck, and N. Kocmich. Independent Evaluation of the Ushahidi Haiti Project. Development Information Systems., 8:2011, 2011.
23. B. Pang and L. Lee. A Sentimental Education: Sentiment Analysis using Subjectivity Summarization Based on Minimum Cuts. In Proc. of the 42nd annual meeting on Association for Computational Linguistics, page 271, 2004.
24. M. F. Porter. An algorithm for suffix stripping. Program: Electronic library and Information Systems, 14(3):130-137, 1980.
25. V. C. Raykar and S. Yu. Eliminating Spammers and Ranking Annotators for Crowdsourced Labeling Tasks. Journal of Machine Learning Research, 13:491-518, 2012.
26. V. C. Raykar, S. Yu, L. H. Zhao, G. H. Valadez, C. Florin, L. Bogoni, and L. Moy. Learning From Crowds. Journal of Machine Learning Research, 11:1297-1322, 2010.
27. F. Rodrigues, F. Pereira, and B. Ribeiro. Learning from Multiple Annotators: Distinguishing Good from Random Labelers. Pattern Recognition Letters, 34(12):1428-1436, 2013.
28. E. Simpson, S. Roberts, I. Psorakis, and A. Smith. Dynamic Bayesian Combination of Multiple Imperfect Classifiers. In Intelligent Systems Reference Library series: Decision Making and Imperfection, pages 1-35. Springer, 2013.
29. L. Tran-Thanh, M. Venanzi, A. Rogers, and N. R. Jennings. Efficient Budget Allocation with Accuracy Guarantees for Crowdsourcing Classification Tasks. In Proc. of the 12th Int. Conf. on Autonomous Agents and Multiagent Systems, pages 901-908, 2013.
30. M. Venanzi, J. Guiver, G. Kazai, P. Kohli, and M. Shokouhi. Community-based Bayesian Aggregation Models for Crowdsourcing. In Proc. of the 23rd Int. Conf. on World Wide Web, pages 155-164, 2014.
31. P. Welinder, S. Branson, P. Perona, and S. J. Belongie. The Multidimensional Wisdom of Crowds. In Advances in Neural Information Processing Systems, pages 2424-2432, 2010.
32. J. Whitehill, T.-f. Wu, J. Bergsma, J. R. Movellan, and P. L. Ruvolo. Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise. In Advances in Neural Information Processing Systems, pages 2035-2043, 2009.
33. K. W. Willett, C. J. Lintott, S. P. Bamford, K. L. Masters, B. D. Simmons, K. R. Casteels, E. M. Edmondson, L. F. Fortson, S. Kaviraj, W. C. Keel, et al. Galaxy Zoo 2: Detailed Morphological Classifications for 304, 122 Galaxies from the Sloan Digital Sky Survey. Monthly Notices of the Royal Astronomical Society, 435(4):2835-2860, 2013.