Optimal crowd-powered rating and filtering algorithms

Proceedings of the VLDB Endowment

Volumn 7, Issue 9, 2014, Pages 685-696

  Parameswaran, Aditya ^a   Boyd, Stephen ^a   Garcia Molina, Hector ^a   Gupta, Ashish ^a   Polyzotis, Neoklis ^b   Widom, Jennifer ^a  

^a STANFORD UNIVERSITY   (United States)

^b GOOGLE INC   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

SIGNAL FILTERING AND PREDICTION;

BUILDING BLOCKES; FILTERING ALGORITHM; FIXED COST; ONLINE COURSE; PEER EVALUATIONS; REAL-WORLD;

CROWDSOURCING;

EID: 84901773447     PISSN: None     EISSN: 21508097     Source Type: Conference Proceeding    
DOI: 10.14778/2732939.2732942     Document Type: Conference Paper

References (48)

1. Coursera Inc. (Retrieved 14 August 2013). http://www.coursera.com.
2. CrowdFlower Content Moderation Platform (Retrieved 14 August 2013). http://crowdflower.com/type-content-moderation.
3. EdX (Retrieved 14 August 2013). http://edx.edu.
4. Enlightening Statistics about MOOCS (Retrieved 14 August 2013). http://www.edtechmagazine.com/higher/article/2013/02/11-enlighteningstatistics-about-massive-open-online-courses.
5. ODesk (Retrieved 22 July 2013). http://odesk.com.
6. Udacity Inc. (Retrieved 14 August 2013) . http://www.udacity.edu.
7. O. Alonso, D. E. Rose, and B. Stewart. Crowdsourcing for relevance evaluation. SIGIR Forum, 42(2):9-15, 2008.
8. Y. Amsterdamer, Y. Grossman, T. Milo, and P. Senellart. Crowd mining. In SIGMOD Conference, pages 241-252, 2013.
9. E. Bakshy, J. M. Hofman, W. A. Mason, and D. J. Watts. Everyone's an influencer: quantifying influence on twitter. In WSDM, pages 65-74, 2011.
10. M. Barber, K. Donnelly, and S. Rizvi. An Avalanche is Coming: Higher Education and the Revolution Ahead. Institute for Public Policy Research, March 2013.
11. M. S. Bernstein, J. Brandt, R. C. Miller, and D. R. Karger. Crowds in two seconds: enabling realtime crowd-powered interfaces. In UIST, pages 33-42, 2011.
12. M. S. Bernstein, G. Little, R. C. Miller, B. Hartmann, M. S. Ackerman, D. R. Karger, D. Crowell, and K. Panovich. Soylent: a word processor with a crowd inside. In UIST, pages 313-322, 2010.
13. A. Bozzon, M. Brambilla, and S. Ceri. Answering search queries with crowdsearcher. In WWW, pages 1009-1018, 2012.
14. N. Bruno. Minimizing database repros using language grammars. In EDBT, pages 382-393, 2010.
15. K.-T. Chen, C.-C. Wu, Y.-C. Chang, and C.-L. Lei. A crowdsourceable qoe evaluation framework for multimedia content. In ACM Multimedia, pages 491-500, 2009.
16. P. Dai, Mausam, and D. S. Weld. Decision-theoretic control of crowd-sourced workflows. In AAAI, 2010.
17. S. B. Davidson, S. Khanna, T. Milo, and S. Roy. Using the crowd for top-k and group-by queries. In ICDT, pages 225-236, 2013.
18. A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Applied Statistics, 28(1):20-28, 1979.
19. M. J. Franklin, D. Kossmann, T. Kraska, S. Ramesh, and R. Xin. Crowddb:answering queries with crowdsourcing. In SIGMOD, pages 61-72, 2011.
20. R. Gomes, P. Welinder, A. Krause, and P. Perona. Crowdclustering. In NIPS, pages 558-566, 2011.
21. S. Guo, A. Parameswaran, and H. Garcia-Molina. So Who Won? Dynamic Max Discovery with the Crowd. In SIGMOD, pages 385-396, 2012.
22. J. Whitehill et al. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In NIPS. 2009.
23. M. Joglekar, H. Garcia-Molina, and A. Parameswaran. Evaluating the crowd with confidence. In KDD, pages 686-694, 2013.
24. E. Kamar, S. Hacker, and E. Horvitz. Combining human and machine intelligence in large-scale crowdsourcing. In AAMAS, pages 467-474, 2012.
25. D. R. Karger, S. Oh, and D. Shah. Budget-optimal task allocation for reliable crowdsourcing systems. CoRR, abs/1110.3564, 2011.
26. C. H. Lin, Mausam, and D. S. Weld. Crowdsourcing control: Moving beyond multiple choice. In UAI, pages 491-500, 2012.
27. C. H. Lin, Mausam, and D. S. Weld. Dynamically switching between synergistic workflows for crowdsourcing. In AAAI, 2012.
28. G. Little, L. B. Chilton, M. Goldman, and R. C. Miller. Turkit: tools for iterative tasks on mechanical turk. In HCOMP, pages 29-30, 2009.
29. I. Lotosh, T. Milo, and S. Novgorodov. Crowdplanr: Planning made easy with crowd. In ICDE, pages 1344-1347, 2013.
30. A. Marcus, E. Wu, D. Karger, S. Madden, and R. Miller. Human-powered sorts and joins. In VLDB, pages 13-24, 2012.
31. A. Marcus, E. Wu, S. Madden, and R. Miller. Crowdsourced databases:Query processing with people. In CIDR, 2011.
32. M. Motoyama, K. Levchenko, C. Kanich, D. McCoy, G. M. Voelker, and S. Savage. Re: Captchas-understanding captcha-solving services in an economic context. In USENIX Security Symposium, pages 435-462, 2010.
33. J. Noronha, E. Hysen, H. Zhang, and K. Z. Gajos. Platemate:crowdsourcing nutritional analysis from food photographs. In UIST, pages 1-12, 2011.
34. P. Donmez et al. Efficiently learning the accuracy of labeling sources for selective sampling. In KDD, 2009.
35. P. Welinder and P. Perona. Online crowdsourcing: rating annotators and obtaining cost-effective labels. In CVPR, 2010.
36. A. Parameswaran, S. Boyd, H. Garcia-Molina, A. Gupta, N. Polyzotis, and J. Widom. Optimal crowd-powered rating and filtering algorithms. Infolab technical report, Stanford University, 2013. http://ilpubs.stanford.edu:8090/1078/.
37. A. Parameswaran, H. Garcia-Molina, H. Park, N. Polyzotis, A. Ramesh, and J. Widom. Crowdscreen: Algorithms for filtering data with humans. In SIGMOD, pages 361-372, 2012.
38. A. Parameswaran, M. H. Teh, H. Garcia-Molina, and J. Widom. Datasift:An expressive and accurate crowd-powered search toolkit. In HCOMP, 2013.
39. H. Park, R. Pang, A. Parameswaran, H. Garcia-Molina, N. Polyzotis, and J. Widom. An overview of the deco system: Data model and query language; query processing and optimization. ACM SIGMOD Record, 41(4), December 2012.
40. A. Ramesh, A. Parameswaran, H. Garcia-Molina, and N. Polyzotis. Identifying reliable workers swiftly. Infolab technical report, Stanford University, 2012.
41. A. D. Sarma, A. Parameswaran, H. Garcia-Molina, and A. Halevy. Finding with the crowd. In ICDE, 2014 (To Appear).
42. R. Snow, B. O'Connor, D. Jurafsky, and A. Y. Ng. Cheap and fast - but is it good? evaluating non-expert annotations for natural language tasks. In EMNLP, pages 254-263, 2008.
43. R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction (Adaptive Computation and Machine Learning). A Bradford Book, 1998.
44. B. Trushkowsky, T. Kraska, M. J. Franklin, and P. Sarkar. Getting it all from the crowd. CoRR, abs/1202.2335, 2012.
45. V. Raykar et al. Supervised learning from multiple experts: whom to trust when everyone lies a bit. In ICML, 2009.
46. J. Vermorel and M. Mohri. Multi-armed bandit algorithms and empirical evaluation. In ECML, pages 437-448, 2005.
47. J. Wang, T. Kraska, M. Franklin, and J. Feng. Crowder: Crowdsourcing entity resolution. In VLDB, 2012.
48. H. Zhang, E. Law, R. Miller, K. Gajos, D. C. Parkes, and E. Horvitz. Human computation tasks with global constraints. In CHI, pages 217-226, 2012.