Active content-based crowdsourcing task selection

International Conference on Information and Knowledge Management, Proceedings

Volumn 24-28-October-2016, Issue , 2016, Pages 529-537

  Bansal, Piyush ^a   Eickhoff, Carsten ^b   Hofmann, Thomas ^b  

^a INTERNATIONAL INSTITUTE OF INFORMATION TECHNOLOGY   (India)

^b ETH ZURICH   (Switzerland)

Author keywords

Active learning; Crowdsourcing; Relevance assessment

Indexed keywords

ARTIFICIAL INTELLIGENCE; BUDGET CONTROL; KNOWLEDGE MANAGEMENT; REDUNDANCY;

ACTIVE LEARNING; ACTIVE LEARNING SCHEME; CORPUS ANNOTATIONS; DOCUMENT SELECTION; MUTUAL INFORMATIONS; RELEVANCE ASSESSMENTS; RELEVANCE JUDGEMENT; STATE-OF-THE-ART PERFORMANCE;

CROWDSOURCING;

EID: 84996520882     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2983323.2983716     Document Type: Conference Paper

References (36)

1. O. Alonso, D. E. Rose, and B. Stewart. Crowdsourcing for relevance evaluation. SIGIR Forum, 42(2):9-15, Nov. 2008.
2. J. A. Aslam, V. Pavlu, and E. Yilmaz. A statistical method for system evaluation using incomplete judgments. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '06, pages 541-548, New York, NY, USA, 2006. ACM.
3. C. Buckley and E. M. Voorhees. Retrieval evaluation with incomplete information. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '04, pages 25-32, New York, NY, USA, 2004. ACM.
4. J. Callan, M. Hoy, C. Yoo, and L. Zhao. Clueweb09 data set, 2009.
5. B. Carterette, J. Allan, and R. Sitaraman. Minimal test collections for retrieval evaluation. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '06, pages 268-275, New York, NY, USA, 2006. ACM.
6. C. Cleverdon. Readings in information retrieval. In K. Sparck Jones and P. Willett, editors, Readings in Information Retrieval, chapter The Cranfield Tests on Index Language Devices, pages 47-59. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1997.
7. M. Davtyan, C. Eickhoff, and T. Hofmann. Exploiting document content for efficient aggregation of crowdsourcing votes. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pages 783-790. ACM, 2015.
8. D. E. Difallah, M. Catasta, G. Demartini, and P. Cudré-Mauroux. Scaling-up the crowd: Micro-task pricing schemes for worker retention and latency improvement. In Second AAAI Conference on Human Computation and Crowdsourcing, 2014.
9. D. E. Difallah, G. Demartini, and P. Cudré-Mauroux. Pick-a-crowd: tell me what you like, and i'll tell you what to do. In Proceedings of the 22nd international conference on World Wide Web, pages 367-374. International World Wide Web Conferences Steering Committee, 2013.
10. C. Eickhoff and A. P. de Vries. Increasing cheat robustness of crowdsourcing tasks. Information retrieval, 16(2):121-137, 2013.
11. C. Eickhoff, C. G. Harris, A. P. de Vries, and P. Srinivasan. Quality through flow and immersion: gamifying crowdsourced relevance assessments. In Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval, pages 871-880. ACM, 2012.
12. C. Grady and M. Lease. Crowdsourcing document relevance assessment with mechanical turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, CSLDAMT '10, pages 172-179, Stroudsburg, PA, USA, 2010. Association for Computational Linguistics.
13. M. Hirth, T. Hoßfeld, and P. Tran-Gia. Cheat-detection mechanisms for crowdsourcing. University of Würzburg, Tech. Rep, 474, 2010.
14. J. Howe. Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business. Crown Publishing Group, New York, NY, USA, 1 edition, 2008.
15. T. Joachims, L. Granka, B. Pan, H. Hembrooke, and G. Gay. Accurately interpreting clickthrough data as implicit feedback. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '05, pages 154-161, New York, NY, USA, 2005. ACM.
16. G. Kazai, J. Kamps, M. Koolen, and N. Milic-Frayling. Crowdsourcing for book search evaluation: impact of hit design on comparative system ranking. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, pages 205-214. ACM, 2011.
17. G. Kazai, J. Kamps, and N. Milic-Frayling. Worker types and personality traits in crowdsourcing relevance labels. In Proceedings of the 20th ACM international conference on Information and knowledge management, pages 1941-1944. ACM, 2011.
18. G. Kazai, J. Kamps, and N. Milic-Frayling. An analysis of human factors and label accuracy in crowdsourcing relevance judgments. Information retrieval, 16(2):138-178, 2013.
19. G. Kazai and N. Milic-Frayling. On the evaluation of the quality of relevance assessments collected through crowdsourcing. In SIGIR Workshop on Future of IR Evaluation. Association for Computing Machinery, Inc., July 2009.
20. A. Kittur, E. H. Chi, and B. Suh. Crowdsourcing user studies with mechanical turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '08, pages 453-456, New York, NY, USA, 2008. ACM.
21. A. Krause, A. Singh, and C. Guestrin. Near-optimal sensor placements in gaussian processes: Theory, efficient algorithms and empirical studies. The Journal of Machine Learning Research, 9:235-284, 2008.
22. Q. V. Le and T. Mikolov. Distributed representations of sentences and documents. arXiv preprint arXiv:1405.4053, 2014.
23. M. Lease. On quality control and machine learning in crowdsourcing. Human Computation, 11:11 2011.
24. M. Lease and G. Kazai. Overview of the trec 2011 crowdsourcing track. In Proceedings of the text retrieval conference (TREC), 2011.
25. D. J. MacKay. Information-based objective functions for active data selection. Neural computation, 4(4):590-604, 1992.
26. C. C. Marshall and F. M. Shipman. The ownership and reuse of visual media. In Proceedings of the 11th annual international ACM/IEEE joint conference on Digital libraries, pages 157-166. ACM, 2011.
27. G. L. Nemhauser, L. A. Wolsey, and M. L. Fisher. An analysis of approximations for maximizing submodular set functionsâǍŤi. Mathematical Programming, 14(1):265-294, 1978.
28. J. Pennington, R. Socher, and C. D. Manning. Glove: Global vectors for word representation. In EMNLP, Volume 14, pages 1532-1543, 2014.
29. N. Ramakrishnan, C. Bailey-Kellogg, S. Tadepalli, V. Pandey, et al. Gaussian processes for active data mining of spatial aggregates. In SDM, pages 427-438. SIAM, 2005.
30. M. Sabou, K. Bontcheva, L. Derczynski, and A. Scharl. Corpus annotation through crowdsourcing: Towards best practice guidelines. In LREC, pages 859-866, 2014.
31. N. Srinivas, A. Krause, S. M. Kakade, and M. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. arXiv preprint arXiv:0912.3995, 2009.
32. C. J. van Rijsbergen and K. SPARCK JONES. A test for the separation of relevant and non-relevant documents in experimental retrieval collections. Journal of Documentation, 29(3):251-257, 1973.
33. E. M. Voorhees and D. K. Harman. TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing). The MIT Press, 2005.
34. J. Wang, S. Faridani, and P. G. Ipeirotis. Estimating the completion time of crowdsourced tasks using survival analysis models. Crowdsourcing for Search and Data Mining (CSDM 2011), page 31, 2011.
35. F. Wilcoxon. Individual comparisons by ranking methods. Biometrics bulletin, 1(6):80-83, 1945.
36. Y. Yan, G. M. Fung, R. Rosales, and J. G. Dy. Active learning from crowds. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 1161-1168, 2011.