Challenges in Data Crowdsourcing

IEEE Transactions on Knowledge and Data Engineering

Volumn 28, Issue 4, 2016, Pages 901-911

  Garcia Molina, Hector ^a   Joglekar, Manas ^a   Marcus, Adam ^b   Parameswaran, Aditya ^c   Verroios, Vasilis ^a  

^a Stanford University   (United States)

^b Unlimited Labs   (United States)

^c UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

Author keywords

crowdsourcing design; crowdsourcing space; crowdsourcing workflow; data augmenting; Data crowdsourcing; data curation; data processing; worker management

Indexed keywords

CROWDSOURCING; DATA ACQUISITION; DATA HANDLING; DATA PROCESSING; HUMAN RESOURCE MANAGEMENT;

DATA AUGMENTING; DATA CURATION; DESIGN STEPS; LARGE PROBLEMS; SUB-PROBLEMS;

PROBLEM SOLVING;

EID: 84963777578     PISSN: 10414347     EISSN: None     Source Type: Journal    
DOI: 10.1109/TKDE.2016.2518669     Document Type: Review

References (66)

1. CrowdFlower (Retrieved 22 July 2013) [Online]. Available: http://crowdflower.com
2. Duolingo Inc. (Retrieved 14 August 2013) [Online]. Available: http://www.duolingo.com
3. Elance-oDesk (Retrieved 16 March 2015) [Online]. Available: http://elance-odesk.com/
4. Microsoft Wants to Turn Crowdsourcing from an Art to a Science (Retrieved 10 January 2014) [Online]. Available: http://www. crowdsourcing.org/editorial/microsoft-wants-to-turn-crowdsourcing-from-an-art-into-a-science/21233
5. Samasource (Retrieved 22 July 2013) [Online]. Available: http://samasource.com
6. StitchFix (Retrieved 5 August 2015) [Online]. Available: https://www.stitchfix.com//
7. S. Ahmad, A. Battle, Z. Malkani, and S. D. Kamvar, "The jabberwocky programming environment for structured social computing, " in Proc. 24th Annu. ACM Symp. User Interface Softw. Technol., 2011, pp. 53-64.
8. M. Ajtai, V. Feldman, A. Hassidim, and J. Nelson, "Sorting and selection with imprecise comparisons, " in Proc. 36th Int. Colloq. Automata, Languages Program.: Part I, 2009, no. 1, pp. 37-48.
9. Y. Amsterdamer, Y. Grossman, T. Milo, and P. Senellart, "Crowd mining, " in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2013, pp. 241-252.
10. Y. Amsterdamer, A. Kukliansky, and T. Milo, "NL2CM: A natural language interface to crowd mining, " in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2015, pp. 1433-1438.
11. A. Anagnostopoulos, L. Becchetti, A. Fazzone, I. Mele, and M. Riondato, "The importance of being expert: Efficient max-finding in crowdsourcing, " in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2015, pp. 983-998.
12. D. W. Barowy, C. Curtsinger, E. D. Berger, and A. McGregor, "AutoMan: A platform for integrating human-based and digital computation, " in Proc. ACM Int. Conf. Object Oriented Programm. Syst. Languages Appl., 2012, pp. 639-654.
13. K. Bellare, S. Iyengar, A. G. Parameswaran, and V. Rastogi, "Active sampling for entity matching, " in Proc. 18th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2012, pp. 1131-1139.
14. M. S. Bernstein, G. Little, R. C. Miller, B. Hartmann, M. S. Ackerman, D. R. Karger, D. Crowell, and K. Panovich, "Soylent:Aword processor with a crowd inside, " in Proc. 23rd Annu. ACM Symp. User Interface Softw. Technol., 2010, pp. 313-322.
15. C. C. Cao, J. She, Y. Tong, and L. Chen, "Whom to ask? jury selection for decision making tasks on micro-blog services, " in Proc. VLDB Endowment, vol. 5, no. 11, pp. 1495-1506, 2012.
16. C. C. Cao, Y. Tong, L. Chen, and H. V. Jagadish, "Wisemarket: A new paradigm for managing wisdom of online social users, " in Proc. 19th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2013, pp. 455-463.
17. X. Chu, J. Morcos, I. F. Ilyas, M. Ouzzani, P. Papotti, N. Tang, and Y. Ye, "Katara: A data cleaning system powered by knowledge bases and crowdsourcing, " in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2015, pp. 1247-1261
18. N. N. Dalvi, R. Kumar, B. Pang, R. Ramakrishnan, A. Tomkins, P. Bohannon, S. Keerthi, and S. Merugu, "A web of concepts, " in Proc. 28th ACM SIGMOD-SIGACT-SIGART Symp. Principles Database Syst., 2009, pp. 1-12.
19. S. B. Davidson, S. Khanna, T. Milo, and S. Roy, "Using the crowd for top-k and group-by queries, " in Proc. 16th Int. Conf. Database Theory, 2013, pp. 225-236.
20. J. Fan, G. Li, B. C. Ooi, K.-l. Tan, and J. Feng, "iCrowd: An adaptive crowdsourcing framework, " in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2015, pp. 1015-1030.
21. M. J. Franklin, D. Kossmann, T. Kraska, S. Ramesh, and R. Xin, "CrowdDB: Answering queries with crowdsourcing, " in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2011, pp. 61-72.
22. A. Ghosh, "Game theory and incentives in human computation systems, " in Handbook of Human Computation. New York, NY, USA: Springer, 2013.
23. S. Guo, A. G. Parameswaran, and H. Garcia-Molina, "So who won?: Dynamic max discovery with the crowd, " in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2012, pp. 385-396.
24. N. Q. V. Hung, D. C. Thang, M. Weidlich, and K. Aberer, "Minimizing efforts in validating crowd answers, " in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2015, pp. 999-1014.
25. M. Joglekar, H. Garcia-Molina, and A. Parameswaran, "Comprehensive and reliable crowd assessment algorithms, " in Proc. IEEE 31st Int. Conf. Data Eng., 2015, pp. 195-206.
26. D. R. Karger, S. Oh, and D. Shah, "Efficient crowdsourcing for multi-class labeling, " in Proc. ACM SIGMETRICS/Int. Conf. Meas. Model. Comput. Syst., 2013, pp. 81-92.
27. A. Kittur, E. H. Chi, and B. Suh, "Crowdsourcing user studies with mechanical turk, " in Proc. SIGCHI Conf. Human Factors Comput. Syst., 2008, pp. 453-456.
28. A. Kittur, B. Smus, and R. Kraut, "CrowdForge: Crowdsourcing complex work, " in Proc. CHI Extended Abstracts, 2011, pp. 1801-1806.
29. A. P. Kulkarni, M. Can, and B. Hartmann, "Collaboratively crowdsourcing workflows with turkomatic, " in Proc. Conf. Comput. Supported Cooperative Work, 2012, pp. 1003-1012.
30. G. Little, L. B. Chilton, M. Goldman, and R. C. Miller, "TurKit: Human computation algorithms on mechanical turk, " in Proc. 23rd Annu. ACM Symp. User Interface Softw. Technol., 2010, pp. 57-66.
31. A. Marcus and A. Parameswaran, "Crowdsourced data management: Industry and academic perspectives, " Found. Trends Databases, vol. 6, nos. 1/2, pp. 1-161, 2015.
32. A. Marcus, E. Wu, D. R. Karger, S. Madden, and R. C. Miller, "Human-powered sorts and joins, " Proc. VLDB Endowment, vol. 5, no. 1, pp. 13-24, 2011.
33. A. Marcus, E. Wu, S. Madden, and R. C. Miller, "Crowdsourced databases: Query processing with people, " in Proc. 5th Biennial Conf. Innovative Data Syst. Res., 2011, pp. 211-214.
34. M. R. Morris and J. Teevan, Collaborative Web Search: Who, What, Where, When, and Why. San Rafael, CA, USA: Morgan & Claypool Publishers, 2009.
35. B. Mozafari, P. Sarkar, M. J. Franklin, M. I. Jordan, and S. Madden, "Active learning for crowd-sourced databases, " arXiv preprint arXiv:1209.3686, 2012.
36. A. Parameswaran, S. Boyd, H. Garcia-Molina, A. Gupta, N. Polyzotis, and J. Widom. (2013). Optimal crowd-powered rating and filtering algorithms. Stanford Univ., Stanford, CA, USA, Tech. Rep. http://ilpubs.stanford.edu:8090/1078/
37. A. G. Parameswaran, H. Garcia-Molina, H. Park, N. Polyzotis, A. Ramesh, and J. Widom, "Crowdscreen: Algorithms for filtering data with humans, " in Proc. SIGMOD Conf., 2012, pp. 361-372.
38. A. G. Parameswaran, M. H. Teh, H. Garcia-Molina, and J. Widom, "DataSift: An expressive and accurate crowd-powered search toolkit, " presented at the 1st AAAI Conf. Human Comput. Crowdsourcing, Palm Springs, CA, USA, Nov. 7-9, 2013.
39. A. G. Parameswaran, M. H. Teh, H. Garcia-Molina, and J. Widom, "Datasift: A crowd-powered search toolkit, " in Proc. Int. Conf. Manage. Data, Snowbird, UT, USA, Jun. 22-27, 2014, pp. 885-888.
40. H. Park, R. Pang, A. Parameswaran, H. Garcia-Molina, N. Polyzotis, and J. Widom, "An overview of the deco system: Data model and query language; query processing and optimization, " ACM SIGMOD Rec., no. 41, no. 4, Dec. 2012.
41. M. J. Raddick, G. Bracey, P. L. Gay, C. J. Lintott, P. Murray, K. Schawinski, A. S. Szalay, and J. Vandenberg. (2009, Sep.). Galaxy zoo: Exploring the motivations of citizen science volunteers [Online]. Available: http://arxiv.org/abs/0909.2925
42. D. Retelny, S. Robaszkiewicz, A. To, W. S. Lasecki, J. Patel, N. Rahmati, T. Doshi, M. Valentine, and M. S. Bernstein, "Expert crowdsourcing with flash teams, " in Proc. 27th Annu. ACM Symp. User Interface Softw. Technol., 2014, pp. 75-85
43. A. D. Sarma, A. Jain, A. Nandi, A. Parameswaran, and J. Widom. (2015). Surpassing humans and computers with jellybean: Crowdvision-hybrid counting algorithms. Stanford Univ., Stanford, CA, USA, Tech. Rep. [Online]. Available: http://ilpubs.stanford. edu:8090/1128/
44. A. D. Sarma, A. Parameswaran, H. Garcia-Molina, and A. Halevy, "Crowd-powered find algorithms, " in Proc. IEEE 30th Int. Conf. Data Eng., 2014, pp. 964-975.
45. L. Schmidt, "Crowdsourcing for human subjects research, " in Proc. CrowdConf., 2010.
46. B. Settles, Active Learning, Synthesis Lectures on Artificial Intelligence and Machine Learning. San Rafael, CA, USA: Morgan & Claypool Publishers, 2012.
47. V. S. Sheng, F. J. Provost, and P. G. Ipeirotis, "Get another label? improving data quality and data mining using multiple, noisy labelers, " in Proc. 14th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2008, pp. 614-622.
48. A. Singhal, "Introducing the knowledge graph: Things, not strings, " Official Google Blog, May, 2012.
49. R. Snow, B. O'Connor, D. Jurafsky, and A. Y. Ng, "Cheap and fast - but is it good?: Evaluating non-expert annotations for natural language tasks, " in Proc. Conf. Empirical Methods Natural Language Process., 2008, pp. 254-263.
50. C. Sun, N. Rampalli, F. Yang, and A. Doan, "Chimera: Large-scale classification using machine learning, rules, and crowdsourcing, " Proc. VLDB Endowment, vol. 7, no. 13, pp. 1529-1540, 2014.
51. B. Trushkowsky, T. Kraska, M. J. Franklin, and P. Sarkar, "Crowdsourced enumeration queries, " in Proc. IEEE 29th Int. Conf. Data Eng., 2013, pp. 673-684.
52. P. Venetis and H. Garcia-Molina. (2012, Aug.). Dynamic max algorithms in crowdsourcing environments. Stanford Univ., Stanford, CA, USA, Tech. Rep. [Online]. Available: http://ilpubs.stanford. edu:8090/1050/
53. V. Verroios and H. Garcia-Molina, "Entity resolution with crowd errors, " in Proc. 31st IEEE Int. Conf. Data Eng., Seoul, South Korea, Apr. 13-17, 2015, pp. 219-230.
54. V. Verroios, P. Lofgren, and H. Garcia-Molina, "tdp: An optimallatency budget allocation strategy for crowdsourced MAXIMUM operations, " in Proc. ACM SIGMOD Int. Conf. Manage. Data, Melbourne, Vic., Australia, May 31-Jun. 4, 2015, pp. 1047-1062.
55. L. von Ahn, M. Blum, N. J. Hopper, and J. Langford, "Captcha: Using hard ai problems for security, " in Proc. Int. Conf. Theory Appl. Cryptographic Techn., 2003, pp. 294-311.
56. L. von Ahn and L. Dabbish, "Designing games with a purpose, " Commun. ACM, vol. 51, no. 8, pp. 58-67, 2008.
57. J. Wang, T. Kraska, M. J. Franklin, and J. Feng, "CrowdER: Crowdsourcing Entity Resolution, " Proc. VLDB Endowment, vol. 5, pp. 1483-1494, 2012.
58. J. Wang, G. Li, T. Kraska, M. J. Franklin, and J. Feng, "Leveraging transitive relations for crowdsourced joins, " in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2013, pp. 229-240
59. Q. Wang, P. Cavanagh, and M. Green, "Familiarity and pop-out in visual search, " Perception Psychophys., no. 56, no 5, pp. 495-500, 1994.
60. S. E. Whang, P. Lofgren, and H. Garcia-Molina, "Question selection for crowd entity resolution, " in Proc. VLDB Endowment, no. 6, pp. 349-360, Apr. 2013.
61. T. Wu, L. Chen, P. Hui, C. J. Zhang, and W. Li, "Hear the whole story: Towards the diversity of opinion in crowdsourcing markets, " Proc. VLDB Endowment, vol. 8, no. 5, pp. 485-496, 2015.
62. Y. Yan, R. Rosales, G. Fung, and J. G. Dy, "Active learning from crowds, " in Proc. 28th Int. Conf. Mach. Learn., Bellevue, WA, USA, Jun. 28-Jul. 2, 2011, pp. 1161-1168.
63. C. J. Zhang, L. Chen, Y. Tong, and Z. Liu, "Cleaning uncertain data with a noisy crowd, " in Proc. 31st IEEE Int. Conf. Data Eng., Seoul, South Korea, Apr. 13-17, 2015, pp. 6-17.
64. Y. Zheng, R. Cheng, S. Maniu, and L. Mo, "On optimality of jury selection in crowdsourcing, " in Proc. 18th Int. Conf. Extending Database Technol., 2015, pp. 193-204.
65. Y. Zheng, J.Wang, G. Li, R. Cheng, and J. Feng, "QASCA: A qualityaware task assignment system for crowdsourcing applications, " in Proc. ACMSIGMODInt. Conf.Manage. Data, 2015, pp. 1031-1046.
66. H. Zhuang, A. Parameswaran, D. Roth, and J. Han, "Debiasing crowdsourced batches, " Proc. 21st ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2015, pp. 1593-1602.