Argonaut: Macrotask crowdsourcing for complex data processing

Proceedings of the VLDB Endowment

Volumn 8, Issue 12 12, 2015, Pages 1642-1653

  Haas, Daniel ^a   Ansel, Jason ^b   Gu, Lydia ^a   Marcus, Adam ^c  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b GoDaddy Inc ^*  (United States)

^c Unlimited Labs   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CROWDSOURCING; DATA HANDLING; DATA MINING; ECONOMIC AND SOCIAL EFFECTS; EXTRACTION; INVESTMENTS; QUALITY CONTROL; RANDOM ERRORS;

COMPLEX DATA; DATA EXTRACTION; DATABASE COMMUNITY; ENTITY RESOLUTIONS; MICROTASKS; OPTIMISATIONS; PREDICTIVE MODELS; QUERY OPERATORS; STRUCTURED DATA; WORKERS';

BUDGET CONTROL;

EID: 84953877202     PISSN: None     EISSN: 21508097     Source Type: Journal    
DOI: 10.14778/2824032.2824062     Document Type: Chapter

References (33)

1. S. Ahmad et al. The jabberwocky programming environment for structured social computing. In UIST, 2011.
2. A. Arasu, H. Garcia-Molina, and S. University. Extracting structured data from Web pages. In the 2003 ACM SIGMOD international conference on, pages 337-348, New York, New York, USA, 2003. ACM Press.
3. A. Arasu, M. Götz, and R. Kaushik. On active learning of record matching packages. In SIGMOD '10: Proceedings of the 2010 ACM SIGMOD International Conference on Management of data. ACM Request Permissions, June 2010.
4. M. Banko, M. J. Cafarella, S. Soderland, M. Broadhead, and O. Etzioni. Open information extraction from the web. IJCAI '07, pages 2670-2676, Jan. 2007.
5. K. Bellare, S. Iyengar, A. G. Parameswaran, and V. Rastogi. Active sampling for entity matching. In KDD '12: Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM Request Permissions, Aug. 2012.
6. M. S. Bernstein, G. Little, R. C. Miller, B. Hartmann, M. S. Ackerman, D. R. Karger, D. Crowell, and K. Panovich. Soylent: a word processor with a crowd inside. In UIST '10: Proceedings of the 23nd annual ACM symposium on User interface software and technology. ACM Request Permissions, Oct. 2010.
7. C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender. Learning to rank using gradient descent. In ICML '05:Proceedings of the 22nd international conference on Machine learning. ACM, Aug. 2005.
8. M. J. Cafarella, A. Halevy, D. Z. Wang, E. Wu, and Y. Zhang. WebTables. Proceedings of the VLDB Endowment, 1(1):538-549, Aug. 2008.
9. A. Das Sarma, A. Jain, A. Machanavajjhala, and P. Bohannon. An automatic blocking mechanism for large-scale de-duplication tasks. In CIKM '12:Proceedings of the 21st ACM international conference on Information and knowledge management. ACM Request Permissions, Oct. 2012.
10. G. Demartini, D. E. Difallah, and P. Cudré-Mauroux. ZenCrowd: leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. In WWW '12: Proceedings of the 21st international conference on World Wide Web. ACM, Apr. 2012.
11. M. J. Franklin, D. Kossmann, T. Kraska, S. Ramesh, and R. Xin. CrowdDB: answering queries with crowdsourcing. SIGMOD, pages 61-72, 2011.
12. C. Gokhale, S. Das, A. Doan, J. F. Naughton, N. Rampalli, J. Shavlik, and X. Zhu. Corleone:Hands-Ofi Crowdsourcing for Entity Matching. SIGMOD 2014, Mar. 2014.
13. P. G. Ipeirotis, F. Provost, and J. Wang. Quality management on Amazon Mechanical Turk. In HCOMP '10: Proceedings of the ACM SIGKDD Workshop on Human Computation. ACM Request Permissions, July 2010.
14. S. Kandel, A. Paepcke, J. M. Hellerstein, and J. Heer. Wrangler, Jan. 2011.
15. D. R. Karger, S. Oh, and D. Shah. Iterative Learning for Reliable Crowdsourcing Systems. Advances in neural information..., pages 1953-1961, 2011.
16. A. Kittur, B. Smus, S. Khamkar, and R. E. Kraut. Crowdforge: crowdsourcing complex work. In UIST, pages 43-52, 2011.
17. A. Kulkarni, P. Gutheim, P. Narula, D. Rolnitzky, T. S. Parikh, and B. Hartmann. MobileWorks:Designing for Quality in a Managed Crowdsourcing Architecture. IEEE Internet Computing, 16(5):28-35, 2012.
18. A. P. Kulkarni, M. Can, and B. Hartmann. Collaboratively crowdsourcing workows with turkomatic. CSCW, pages 1003-1012, 2012.
19. W. S. Lasecki, K. I. Murray, S. White, R. C. Miller, and J. P. Bigham. Real-time crowd control of existing interfaces. In Proceedings of the 24th annual ACM symposium on User interface software and technology, pages 23-32. ACM, 2011.
20. G. Little, L. B. Chilton, M. Goldman, and R. C. Miller. TurKit: tools for iterative tasks on mechanical Turk. KDD Workshop on Human Computation, pages 29-30, 2009.
21. A. Marcus, E. Wu, D. Karger, S. Madden, and R. Miller. Human-powered sorts and joins. In Proceedings of the VLDB Endowment. VLDB Endowment, Sept. 2011.
22. B. Mozafari, P. Sarkar, M. J. Franklin, M. I. Jordan, and S. Madden. Active Learning for Crowd-Sourced Databases. arXiv.org, Sept. 2012.
23. J. Noronha, E. Hysen, H. Zhang, and K. Z. Gajos. Platemate: crowdsourcing nutritional analysis from food photographs. In UIST '11: Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM Request Permissions, Oct. 2011.
24. A. G. Parameswaran, H. Park, H. Garcia-Molina, N. Polyzotis, and J. Widom. Deco: declarative crowdsourcing. In CIKM '12: Proceedings of the 21st ACM international conference on Information and knowledge management. ACM Request Permissions, Oct. 2012.
25. H. Park and J. Widom. CrowdFill: collecting structured data from the crowd. SIGMOD, pages 577-588, 2014.
26. D. Retelny, S. Robaszkiewicz, A. To, W. S. Lasecki, J. Patel, N. Rahmati, T. Doshi, M. Valentine, and M. S. Bernstein. Expert crowdsourcing with ash teams. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, UIST '14, pages 75-85, New York, NY, USA, 2014. ACM.
27. J. M. Rzeszotarski and A. Kittur. Instrumenting the crowd: using implicit behavioral measures to predict task performance. In UIST '11: Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM Request Permissions, Oct. 2011.
28. S. Soderland. Learning Information Extraction Rules for Semi-Structured and Free Text. Machine Learning, 34(1/3):233-272, 1999.
29. M. Stonebraker, D. Bruckner, I. F. Ilyas, G. Beskales, M. Cherniack, S. B. Zdonik, A. Pagan, and S. Xu. Data Curation at Scale: The Data Tamer System. CIDR 2013, 2013.
30. B. Trushkowsky, T. Kraska, M. J. Franklin, and P. Sarkar. Crowdsourced enumeration queries. In Data Engineering (ICDE), 2013 IEEE 29th International Conference on, pages 673-684, 2013.
31. J. Wang, T. Kraska, M. J. Franklin, and J. Feng. CrowdER: Crowdsourcing Entity Resolution. PVLDB, 2012.
32. S. E. Whang, P. Lofgren, and H. Garcia-Molina. Question selection for crowd entity resolution. In Proceedings of the VLDB Endowment. VLDB Endowment, Apr. 2013.
33. H. Zhang, E. Law, R. Miller, K. Gajos, D. Parkes, and E. Horvitz. Human computation tasks with global constraints. In CHI '12: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM Request Permissions, May 2012.