Finish them!: Pricing algorithms for human computation

Proceedings of the VLDB Endowment

Volumn 7, Issue 14, 2014, Pages 1965-1976

  Gao, Yihan ^a   Parameswaran, Aditya ^a  

^a UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BUDGET CONTROL; COMPUTATION THEORY; DECISION THEORY; ECONOMICS; MARKOV PROCESSES;

BUDGET CONSTRAINT; COST CONSTRAINTS; HUMAN COMPUTATION; MARKOV DECISION PROCESSES; MONETARY COSTS; PRICING ALGORITHM; PRICING MODELS; TOTAL ELAPSED TIME;

COSTS;

EID: 84936873985     PISSN: None     EISSN: 21508097     Source Type: Journal    
DOI: 10.14778/2733085.2733101     Document Type: Conference Paper

References (50)

1. Crowdsourced data analysis with Clockwork Raven. https://blog.twitter.com/2012/crowdsourced-data-analysis-with-clockworkraven.
2. Crowdsourcing Insights from eBay (Retrieved 10 January 2014). http://crowdopolis.info/james-rubinstein.pdf.
3. Mechanical Turk (Retrieved 22 July 2013). http://www.mturk.com.
4. Mechanical Turk Tracker (Retrieved 20 February 2014). http://mturk-tracker.com.
5. Microsoft Wants to Turn Crowdsourcing from an Art to a Science (Retrieved 10 January 2014). http://www.crowdsourcing.org/editorial/microsoft-wants-toturn-crowdsourcing-from-an-art-into-a-science/21233.
6. Samasource Annual Report (Retrieved 10 January 2014). http://www.slideshare.net/leila_c/samasource-2011-annual-report.
7. Translators Wanted at LinkedIn. The Pay? 0 an Hour. The New York Times (Retrieved 10 January 2014). http://www.nytimes.com/2009/06/29/technology/start-ups/29linkedin.html.
8. O. Alonso, D. E. Rose, and B. Stewart. Crowdsourcing for relevance evaluation. SIGIR Forum, 42(2):9-15, 2008.
9. P. Bohannon, S. Merugu, C. Yu, V. Agarwal, P. DeRose, A. S. Iyer, A. Jain, V. Kakade, M. Muralidharan, R. Ramakrishnan, and W. Shen. Purple sox extraction management system. SIGMOD Record, 37(4):21-27, 2008.
10. S. P. Boyd and L. Vandenberghe. Convex optimization. Cambridge university press, 2004.
11. N. Bruno. Minimizing database repros using language grammars. In EDBT, pages 382-393, 2010.
12. P. Dai, Mausam, and D. S. Weld. Decision-theoretic control of crowd-sourced workflows. In AAAI, 2010.
13. N. Dalvi, A. Dasgupta, R. Kumar, and V. Rastogi. Aggregating crowdsourced binary ratings. In WWW, pages 285-294, 2013.
14. S. B. Davidson, S. Khanna, T. Milo, and S. Roy. Using the crowd for top-k and group-by queries. ICDT '13, pages 225-236, 2013.
15. S. Faridani, B. Hartmann, and P. G. Ipeirotis. What's the right price? pricing tasks for finishing on time. In Proceedings of HCOMP11: The 3rd Workshop on Human Computation, 2011.
16. M. J. Franklin, D. Kossmann, T. Kraska, S. Ramesh, and R. Xin. Crowddb: answering queries with crowdsourcing. In SIGMOD, 2011.
17. Y. Gao and A. Parameswaran. Finish them!: Pricing Algorithms for Human Computation. Technical report, U. Illinois (UIUC), http://arxiv.org/pdf/1408.6292.pdf, 2014.
18. A. Ghosh, S. Kale, and P. McAfee. Who moderates the moderators? crowdsourcing abuse detection in user-generated content. In EC, pages 167-176, 2011.
19. R. Gomes, P. Welinder, A. Krause, and P. Perona. Crowdclustering. In NIPS, pages 558-566, 2011.
20. S. Guo, A. Parameswaran, and H. Garcia-Molina. So who won?: dynamic max discovery with the crowd. In SIGMOD Conference, pages 385-396, 2012.
21. M. Joglekar, H. Garcia-Molina, and A. Parameswaran. Evaluating the crowd with confidence. In SIGKDD, 2013.
22. M. Joglekar, H. Garcia-Molina, and A. Parameswaran. Evaluating the Crowd with Confidence. In KDD, 2013.
23. E. Kamar, S. Hacker, and E. Horvitz. Combining human and machine intelligence in large-scale crowdsourcing. In AAMAS, pages 467-474, 2012.
24. D. Karger, S. Oh, and D. Shah. Effcient crowdsourcing for multi-class labeling. In SIGMETRICS, pages 81-92, 2013.
25. D. R. Karger, S. Oh, and D. Shah. Budget-optimal task allocation for reliable crowdsourcing systems. CoRR, abs/1110.3564, 2011.
26. C. H. Lin, Mausam, and D. S. Weld. Crowdsourcing control: Moving beyond multiple choice. In UAI, pages 491-500, 2012.
27. C. H. Lin, Mausam, and D. S. Weld. Dynamically switching between synergistic workflows for crowdsourcing. In AAAI, 2012.
28. Q. Liu, J. Peng, and A. Ihler. Variational inference for crowdsourcing. In NIPS, pages 701-709, 2012.
29. X. Liu, M. Lu, B. C. Ooi, Y. Shen, S. Wu, and M. Zhang. Cdas: a crowdsourcing data analytics system. Proc. VLDB Endow., 5(10):1040-1051, June 2012.
30. A. Marcus, E. Wu, D. Karger, S. Madden, and R. Miller. Human-powered sorts and joins. In VLDB, 2012.
31. W. A. Massey, G. A. Parker, and W. Whitt. Estimating the parameters of a nonhomogeneous poisson process with linear rate. Telecommunication Systems, 5(2):361-388, 1996.
32. D. McFadden. Conditional logit analysis of qualitative choice behavior. 1973.
33. M. Motoyama, K. Levchenko, C. Kanich, D. McCoy, G. M. Voelker, and S. Savage. Re: Captchas-understanding captcha-solving services in an economic context. In USENIX Security Symposium, pages 435-462, 2010.
34. P. Donmez et al. Efficiently learning the accuracy of labeling sources for selective sampling. In KDD, pages 259-268, 2009.
35. A. Parameswaran, S. Boyd, H. Garcia-Molina, A. Gupta, N. Polyzotis, and J. Widom. Optimal Crowd-Powered Rating and Filtering Algorithms. Technical report, Stanford University, 4.
36. A. Parameswaran, H. Garcia-Molina, H. Park, N. Polyzotis, A. Ramesh, and J. Widom. CrowdScreen: algorithms for filtering data with humans. In SIGMOD Conference, pages 361-372, 2012.
37. A. Parameswaran, H. Park, H. Garcia-Molina, N. Polyzotis, and J. Widom. Deco: declarative crowdsourcing. In CIKM, pages 1203-1212, 2012.
38. A. Parameswaran and N. Polyzotis. Answering Queries using Humans, Algorithms and Databases. In CIDR, pages 160-166, 2011.
39. A. Parameswaran, A. D. Sarma, H. Garcia-Molina, N. Polyzotis, and J. Widom. Human-assisted graph search: it's okay to ask questions. PVLDB, 4(5):267-278, 2011.
40. R. Snow et al. Cheap and fast-but is it good? evaluating non-expert annotations for natural language tasks. In EMNLP, pages 254-263, 2008.
41. A. Ramesh, A. Parameswaran, H. Garcia-Molina, and N. Polyzotis. Identifying reliable workers swiftly. Technical report, Stanford University, September 2012.
42. V. C. Raykar and S. Yu. Eliminating spammers and ranking annotators for crowdsourced labeling tasks. Journal of Machine Learning Research, 13:491-518, 2012.
43. S. M. Ross. Stochastic processes, volume 2. John Wiley New York, 1996.
44. V. C. Raykar et al. Supervised learning from multiple experts: whom to trust when everyone lies a bit. In ICML, page 112, 2009.
45. V. S. Sheng et al. Get another label? improving data quality and data mining using multiple, noisy labelers. In KDD, pages 614-622, 2008.
46. P. Wais, S. Lingamneni, D. Cook, J. Fennell, B. Goldenberg, D. Lubarov, D. Marin, and H. Simons. Towards building a high-quality workforce with mechanical turk. In Computational Social Science and the Wisdom of Crowds, NIPS Workshop, 2010.
47. J. Wang, T. Kraska, M. J. Franklin, and J. Feng. Crowder: Crowdsourcing entity resolution. PVLDB, 5(11):1483-1494, 2012.
48. P. Welinder and P. Perona. Online crowdsourcing: rating annotators and obtaining cost-effective labels. In CVPR, 2010.
49. J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. R. Movellan. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In NIPS, pages 2035-2043. 2009.
50. O. Zaidan and C. Callison-Burch. Feasibility of human-in-the-loop minimum error rate training. In EMNLP, pages 52-61, 2009.