Crowdsourced Data Management: A Survey

IEEE Transactions on Knowledge and Data Engineering

Volumn 28, Issue 9, 2016, Pages 2296-2319

  Li, Guoliang ^a   Wang, Jiannan ^b   Zheng, Yudian ^c   Franklin, Michael J ^d  

^a TSINGHUA UNIVERSITY   (China)

^b SIMON FRASER UNIVERSITY   (Canada)

^c UNIVERSITY OF HONG KONG   (Hong Kong)

^d UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

cost control; Crowdsourcing; data management; human computation; latency control; quality control

Indexed keywords

AUTOMATION; COST EFFECTIVENESS; COSTS; CROWDSOURCING; IMAGE RECOGNITION; INDUSTRIAL RESEARCH; INFORMATION MANAGEMENT; SURVEYS;

COST CONTROLS; DATA MANIPULATIONS; ENTITY RESOLUTIONS; HUMAN COGNITIVE ABILITIES; HUMAN COMPUTATION; LATENCY CONTROL; MULTIPLE OPERATOR; SENTIMENT ANALYSIS;

QUALITY CONTROL;

EID: 84982126145     PISSN: 10414347     EISSN: None     Source Type: Journal    
DOI: 10.1109/TKDE.2016.2535242     Document Type: Article

References (141)

1. Mturk. (2016). [Online]. Available: https://www.mturk.com/
2. Crowdflower. (2016). [Online]. Available: http://www.crowdflower. com
3. Upwork. (2016). [Online]. Available: https://www.upwork.com
4. Chinacrowd. (2016). [Online]. Available: http://www.chinacrowds. Com
5. Waze. (2016). [Online]. Available: https://www.waze.com
6. R. M. Adelsman and A. B. Whinston, "Sophisticated voting with information for two voting functions," J. Econ. Theory, vol. 15, no. 1, pp. 145-159, 1977.
7. R. Agrawal, T. Imielinski, and A. N. Swami, "Mining association rules between sets of items in large databases," in Proc. ACM SIGMOD Int. Conf. Manage. Data, 1993, pp. 207-216.
8. A. Amarilli, Y. Amsterdamer, and T. Milo, "On the complexity of mining itemsets from the crowd using taxonomies," in Proc. Int. Conf. Database Theory, 2015, pp. 15-25.
9. Y. Amsterdamer, S. Davidson, A. Kukliansky, T. Milo, S. Novgorodov, and A. Somech, "Managing general and individual knowledge in crowd mining applications," in Proc. Conf. Innovative Data Syst. Res., 2015, pp. 1-6.
10. Y. Amsterdamer, S. B. Davidson, T. Milo, S. Novgorodov, and A. Somech, "Oassis: Query driven crowd mining," in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2014, pp. 589-600.
11. Y. Amsterdamer, S. B. Davidson, T. Milo, S. Novgorodov, and A. Somech, "Ontology assisted crowd mining." Proc. VLDB Endowment, vol. 7, no. 13, pp. 1597-1600, 2014.
12. Y. Amsterdamer, Y. Grossman, T. Milo, and P. Senellart, "Crowd mining," in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2013, pp. 241-252.
13. Y. Amsterdamer, Y. Grossman, T. Milo, and P. Senellart, "Crowdminer: Mining association rules from the crowd," Proc. VLDB Endowment, vol. 6, no. 12, pp. 1250-1253, 2013.
14. Y. Amsterdamer, A. Kukliansky, and T. Milo, "Nl2cm: A natural language interface to crowd mining," in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2015, pp. 1433-1438.
15. A. P. Dawid and A. M. Skene, "Maximum likelihood estimation of observer error-rates using em algorithm," Appl. Statist., vol. 28, no. 1 pp. 20-28, 1979.
16. A. Artikis, M. Weidlich, F. Schnitzler, I. Boutsis, T. Liebig, N. Piatkowski, C. Bockermann, K. Morik, V. Kalogeraki, J. Marecek, et al., "Heterogeneous stream processing and crowdsourcing for urban traffic management," in Proc. Int. Conf. Extending Database Technol.: Adv. Database Technol., 2014, pp. 712-723.
17. B. I. Aydin, Y. S. Yilmaz, Y. Li, Q. Li, J. Gao, and M. Demirbas, "Crowdsourcing for multiple-choice question answering," in Proc. Nat. Conf. Artif. Intell., 2014, pp. 2946-2953.
18. R. Boim, O. Greenshpan, T. Milo, S. Novgorodov, N. Polyzotis, and W. C. Tan, "Asking the right questions in crowd data sourcing," in Proc. 28th Int. Conf. Data Eng., 2012, pp. 1261-1264.
19. S. Börzsönyi, D. Kossmann, and K. Stocker, "The skyline operator," in Proc. Int. Conf. Data Eng., 2001, pp. 421-430.
20. R. A. Bradley and M. E. Terry, "Rank analysis of incomplete block designs: I. the method of paired comparisons," Biometrika, 1952, pp. 324-345.
21. R. Busa-Fekete, B. Szorenyi, W. Cheng, P. Weng, and E. Hullermeier, "Top-k selection based on adaptive sampling of noisy preferences," in Proc. Int. Conf.Mach. Learn., 2013, pp. 1094-1102.
22. C. C. Cao, J. She, Y. Tong, and L. Chen, "Whom to ask? jury selection for decision making tasks on micro-blog services," Proc. VLDB Endowment, vol. 5, no. 11 pp. 1495-1506, 2012.
23. X. Chen, P. N. Bennett, K. Collins-Thompson, and E. Horvitz, "Pairwise ranking aggregation in a crowdsourced setting," in Proc. ACM Int. Conf. Web Search Data Mining, 2013, pp. 193-202.
24. X. Chen, Q. Lin, and D. Zhou, "Optimistic knowledge gradient policy for optimal budg et al.ocation in crowdsourcing," in Proc. Int. Conf. Mach. Learn., 2013, pp. 64-72.
25. Z. Chen, R. Fu, Z. Zhao, Z. Liu, L. Xia, L. Chen, P. Cheng, C. C. Cao, Y. Tong, and C. J. Zhang, "Gmission: A general spatial crowdsourcing platform," Proc. VLDB Endowment, vol. 7, no. 13, pp. 1629-1632, 2014.
26. P. Dai, C. H. Lin, Mausam, and D. S. Weld, "Pomdp-based control of workflows for crowdsourcing," Artif. Intell., vol. 202, pp. 52-85, 2013.
27. S. B. Davidson, S. Khanna, T. Milo, and S. Roy, "Using the crowd for top-k and group-by queries," in Proc. Int. Conf. Database Theory, 2013, pp. 225-236.
28. G. Demartini, D. E. Difallah, and P. Cudré-Mauroux, "Zencrowd: Leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking," in Proc. Int. Conf. World Wide Web, 2012, pp. 469-478.
29. A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum likelihood from incomplete data via the em algorithm," J. R. Statist. Soc. B, vol. 30, no. 1, pp. 1-38, 1977.
30. D. Deng, C. Shahabi, and U. Demiryurek, "Maximizing the number of worker's self-selected tasks in spatial crowdsourcing," in SIGSPATIAL., 2013, pp. 324-333.
31. C. B. Eiben, J. B. Siegel, J. B. Bale, S. Cooper, F. Khatib, B. W. Shen, F. Players, B. L. Stoddard, Z. Popovic, and D. Baker, "Increased diels-alderase activity through backbone remodeling guided by foldit players," Nature Biotechnol., vol. 30, no. 2, pp. 190-192, 2012.
32. A. E. Elo, The Rating of Chessplayers, Past and Present, volume 3. London, U.K.: Batsford, 1978.
33. B. Eriksson, "Learning to top-k search using pairwise comparisons," in Proc. Int. Conf. Artif. Intell. Stat., 2013, pp. 265-273.
34. R. Fagin, A. Lotem, and M. Naor, "Optimal aggregation algorithms for middleware," J. Comput. Syst. Sci., vol. 66, no. 4, pp. 614-656, 2003.
35. J. Fan, G. Li, B. C. Ooi, K. Tan, and J. Feng, "iCrowd: An adaptive crowdsourcing framework," in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2015, pp. 1015-1030.
36. J. Fan, M. Lu, B. C. Ooi, W.-C. Tan, and M. Zhang, "A hybrid machine-crowdsourcing system for matching web tables," in Proc. Int. Conf. Data Eng., 2014, pp. 976-987.
37. M. Fang, J. Yin, and D. Tao, "Active learning for crowdsourcing using knowledge transfer," in Nat. Conf. Artif. Intell., 2014, pp. 1809-1815.
38. Y. Fang, H. Sun, G. Li, R. Zhang, and J. Huai, "Effective result inference for context-sensitive tasks in crowdsourcing," in Proc. Int. Conf. Database Syst. Adv. Appl., 2016, pp. 33-48.
39. S. Faradani, B. Hartmann, and P. G. Ipeirotis, "What's the right price? pricing tasks for finishing on time," in Proc. 11th Nat. Conf. Artif. Intell. Workshop, 2011, pp. 26-31.
40. U. Feige, P. Raghavan, D. Peleg, and E. Upfal, "Computing with noisy information," SIAM J. Comput., vol. 23, no. 5, pp. 1001-1018, 1994.
41. J. Feng, G. Li, H. Wang, and J. Feng, "Incremental quality inference in crowdsourcing," in Proc. Int. Conf. Database Syst. Adv. Appl., 2014, pp. 453-467.
42. M. J. Franklin, D. Kossmann, T. Kraska, S. Ramesh, and R. Xin, "Crowddb: Answering queries with crowdsourcing," in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2011, pp. 61-72.
43. J. Gao, X. Liu, B. C. Ooi, H. Wang, and G. Chen, "An online cost sensitive decision-making method in crowdsourcing systems," in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2013, pp. 217-228.
44. Y. Gao and A. G. Parameswaran, "Finish them!: Pricing algorithms for human computation," Proc. VLDB Endowment, vol. 7, no. 14, pp. 1965-1976, 2014.
45. C. Gokhale, S. Das, A. Doan, J. F. Naughton, N. Rampalli, J. W. Shavlik, and X. Zhu, "Corleone: Hands-off crowdsourcing for entity matching," in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2014, pp. 601-612.
46. R. Gomes, P. Welinder, A. Krause, and P. Perona, "Crowdclustering," in Proc. Conf. Adv. Neural Inf. Process. Syst., 2011, pp. 558-566.
47. B. Groz and T. Milo, "Skyline queries with noisy comparisons," in Proc. ACM Symp. Principles Database Syst., pp. 185-198, 2015.
48. A. Gruenheid, D. Kossmann, S. Ramesh, and F. Widmer, "Crowdsourcing entity resolution:When isA= B? Technical report, ETH Zürich," http://e-collection.library.ethz.ch/view/eth:6833
49. S. Guo, A. G. Parameswaran, and H. Garcia-Molina, "So who won?: Dynamic max discovery with the crowd," in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2012, pp. 385-396.
50. D. Haas, J. Ansel, L. Gu, and A. Marcus, "Argonaut: Macrotask crowdsourcing for complex data processing," Proc. VLDB Endowment, vol. 8, no. 12, pp. 1642-1653, 2015.
51. D. Haas, J. Wang, E. Wu, and M. J. Franklin, "Clamshell: Speeding up crowds for low-latency data labeling," Proc. VLDB Endowment, vol. 9, no. 4, pp. 372-383, 2015.
52. H. Heikinheimo and A. Ukkonen, "The crowd-median algorithm," in HCOMP, 2013, pp. 69-77.
53. R. Herbrich, T. Minka, and T. Graepel, "Trueskill: A bayesian skill rating system," in Proc. Conf. Adv. Neural Inf. Process. Syst., 2006, pp. 569-576.
54. C.-J. Ho, S. Jabbari, and J. W. Vaughan, "Adaptive task assignment for crowdsourced classification," in Proc. Int. Conf. Mach. Learn., 2013, pp. 534-542.
55. C.-J. Ho and J. W. Vaughan, "Online task assignment in crowdsourcing markets," in Proc. Nat. Conf. Artif. Intell., 2012, pp. 45-51.
56. H. Hu, G. Li, Z. Bao, and J. Feng, "Crowdsourcing-based realtime urban traffic speed estimation: From speed to trend," in Proc. Int. Conf. Data Eng., 2016, pp. 883-894.
57. H. Hu, Y. Zheng, Z. Bao, G. Li, and J. Feng, "Crowdsourced poi labelling: Location-aware result inference and task assignment," in Proc. Int. Conf. Data Eng., 2016, pp. 61-72.
58. P. Ipeirotis, F. Provost, and J. Wang, "Qualitymanagement on amazonmechanical turk," in Proc. SIGKDDWorkshop, 2010, pp. 64-67.
59. S. R. Jeffery, M. J. Franklin, and A. Y. Halevy, "Pay-as-you-go user feedback for dataspace systems," in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2008, pp. 847-860.
60. X. Jiang, L.-H. Lim, Y. Yao, and Y. Ye, "Statistical ranking and combinatorial hodge theory," Math. Program., 2011, pp. 203-244.
61. M. Joglekar, H. Garcia-Molina, and A. G. Parameswaran, "Evaluating the crowd with confidence," in Proc. SIGKDD, 2013, pp. 686-694.
62. M. Joglekar, H. Garcia-Molina, and A. G. Parameswaran, "Comprehensive and reliable crowd assessment algorithms," in Proc. Int. Conf. Data Eng., 2015, pp. 195-206.
63. H. Kaplan, I. Lotosh, T. Milo, and S. Novgorodov, "Answering planning queries with the crowd," Proc. VLDB Endowment, vol. 6, no. 9, pp. 697-708, 2013.
64. D. R. Karger, S. Oh, and D. Shah, "Iterative learning for reliable crowdsourcing systems," in Proc. Adv. Neural Inf. Process. Syst., 2011, pp. 1953-1961.
65. L. Kazemi and C. Shahabi, "Geocrowd: Enabling query answering with spatial crowdsourcing," in Proc. Int. Conf. Adv. Geographic Inf. Syst., 2012, pp. 189-198.
66. L. Kazemi, C. Shahabi, and L. Chen, "Geotrucrowd: Trustworthy query answering with spatial crowdsourcing," in Proc. Int. Conf. Adv. Geographic Inf. Syst., 2013, pp. 304-313.
67. A. R. Khan and H. Garcia-Molina, "Hybrid strategies for finding the max with the crowd," Stanford University, 2014, http:// ilpubs.stanford.edu:8090/1090/
68. D. Koller and N. Friedman, Probabilistic Graphical Models - Principles and Techniques. Cambridge, MA: MIT Press, 2009.
69. L. I. Kuncheva, C. J. Whitaker, and C. A. Shipp, "Limits on the majority vote accuracy in classifier fusion," Pattern Anal. Appl., vol. 6, no. 1, pp. 22-31, 2003.
70. A. Kurve, D. J. Miller, and G. Kesidis, "Multicategory crowdsourcing accounting for variable task difficulty, worker skill, and worker intention," IEEE Trans. Knowl. Data Eng., vol. 27, no. 3, pp. 794-809, Mar. 2015.
71. G. Li, D. Deng, J. Wang, and J. Feng, "PASS-JOIN: A partitionbased method for similarity joins," Proc. VLDB Endowment, vol. 5, no. 3, pp. 253-264, 2011.
72. N. Littlestone and M. K. Warmuth, "The weighted majority algorithm," Inf. Comput., vol. 108, no. 2, pp. 212-261, 1994.
73. Q. Liu, J. Peng, and A. T. Ihler, "Variational inference for crowdsourcing," in Proc. Adv. Neural Inf. Process. Syst., 2012, pp. 701-709.
74. X. Liu, M. Lu, B. C. Ooi, Y. Shen, S. Wu, and M. Zhang, "CDAS: A crowdsourcing data analytics system," Proc. VLDB Endowment, vol. 5, no. 10, pp. 1040-1051, 2012.
75. C. Lofi, K. E. Maarry, and W. Balke, "Skyline queries in crowdenabled databases," in Proc. Int. Conf. Extending Database Technol.: Adv. Database Technol., 2013, pp. 465-476.
76. C. Lofi, K. E. Maarry, and W. Balke, "Skyline queries over incomplete data - error models for focused crowd-sourcing," in ER, 2013, pp. 298-312.
77. I. Lotosh, T. Milo, and S. Novgorodov, "Crowdplanr: Planning made easy with crowd," in Proc. Int. Conf. Data Eng., 2013, pp. 1344-1347.
78. A. Marcus, D. R. Karger, S. Madden, R. Miller, and S. Oh, "Counting with the crowd," Proc. VLDB Endowment, vol. 6, no. 2, pp. 109-120, 2012.
79. A. Marcus, E. Wu, D. R. Karger, S. Madden, and R. C. Miller, "Human-powered sorts and joins," Proc. VLDB Endowment, vol. 5, no. 1, pp. 13-24, 2011.
80. A. Marcus, E. Wu, S. Madden, and R. C. Miller, "Crowdsourced databases: Query processing with people," in Proc. Conf. Innovative Data Syst. Res., 2011, pp. 211-214.
81. L. Mo, R. Cheng, B. Kao, X. S. Yang, C. Ren, S. Lei, D. W. Cheung, and E. Lo, "Optimizing plurality for human intelligence tasks," in Proc. ACMConf. Inf. Knowl.Manage., 2013, pp. 1929-1938.
82. B. Mozafari, P. Sarkar, M. Franklin, M. Jordan, and S. Madden, "Scaling up crowd-sourcing to very large datasets:Acase for active learning," Proc. VLDBEndowment, vol. 8, no. 2, pp. 125-136, 2014.
83. D. Nau, M. Ghallab, and P. Traverso, Automated Planning: Theory & Practice. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 2004.
84. R. Neal and G. E. Hinton, "A view of the em algorithm that justifies incremental, sparse, and other variants," in Learning in Graphical Models. Berlin, Germany: Kluwer Academic Publishers, 1998, pp. 355-368.
85. S. Negahban, S. Oh, and D. Shah, "Iterative ranking from pairwise comparisons," in Proc. Adv. Neural Inf. Process. Syst., 2012, pp. 2483-2491.
86. Q. V. H. Nguyen, T. T. Nguyen, Z. Miklós, K. Aberer, A. Gal, and M. Weidlich, "Pay-as-you-go reconciliation in schema matching networks," in Proc. Int. Conf. Data Eng., 2014, pp. 220-231.
87. W. R. Ouyang, L. M. Kaplan, P. Martin, A. Toniolo, M. B. Srivastava, and T. J. Norman, "Debiasing crowdsourced quantitative characteristics in local businesses and services," in Proc. Int. Conf. Inf. Process. Sensor Netw., 2015, pp. 190-201.
88. A. G. Parameswaran, S. Boyd, H. Garcia-Molina, A. Gupta, N. Polyzotis, and J. Widom, "Optimal crowd-powered rating and filtering algorithms," Proc. VLDB Endowment, vol. 7, no. 9, pp. 685-696, 2014.
89. A. G. Parameswaran, H. Garcia-Molina, H. Park, N. Polyzotis, A. Ramesh, and J. Widom, "Crowdscreen: Algorithms for filtering data with humans," in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2012, pp. 361-372.
90. A. G. Parameswaran, H. Park, H. Garcia-Molina, N. Polyzotis, and J. Widom, "Deco: Declarative crowdsourcing," in Proc. ACM Conf. Inf. Knowl. Manage., 2012, pp. 1203-1212.
91. A. G. Parameswaran, A. D. Sarma, H. Garcia-Molina, N. Polyzotis, and J. Widom, "Human-assisted graph search: it's okay to ask questions," Proc. VLDB Endowment, vol. 4, no. 5, pp. 267-278, 2011.
92. H. Park and J. Widom, "Query optimization over crowdsourced data," Proc. VLDB Endowment, vol. 6, no. 10, pp. 781-792, 2013.
93. H. Park and J. Widom, "Crowdfill: Collecting structured data from the crowd," in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2014, pp. 577-588.
94. J. Pearl, "Reverend bayes on inference engines:Adistributed hierarchical approach," in Proc. Nat. Conf.Artif. Intell., 1982, pp. 133-136.
95. T. Pfeiffer, X. A. Gao, Y. Chen, A. Mao, and D. G. Rand, "Adaptive polling for information aggregation," in Proc. Nat. Conf. Artif. Intell., 2012, pp. 122-128.
96. J.-C. Pomerol and S. Barba-Romero, Multicriterion Decision in Management: Principal and Practice, vol 25. New York, NY, USA: Springer, 2000.
97. L. Pournajaf, L. Xiong, V. Sunderam, and S. Goryczka, "Spatial task assignment for crowd sensing with cloaked locations," in Proc. IEEE Int. Conf. Mobile Data Manage., 2014, vol. 1, pp. 73-82.
98. E. Rahm and P. A. Bernstein, "A survey of approaches to automatic schema matching," VLDBJ, vol. 10, no. 4, pp. 334-350, 2001.
99. V. C. Raykar and S. Yu, "Eliminating spammers and ranking annotators for crowdsourced labeling tasks," J. Mach. Learning Res., vol. 13, pp. 491-518, 2012.
100. V. C. Raykar, S. Yu, L. H. Zhao, A. K. Jerebko, C. Florin, G. H. Valadez, L. Bogoni, and L. Moy, "Supervised learning from multiple experts: Whom to trust when everyone lies a bit," in Proc. Int. Conf. Mach. Learn., 2009, pp. 889-896.
101. S. Sarawagi and A. Bhamidipaty, "Interactive deduplication using active learning," in Proc. SIGKDD, 2002, pp. 269-278.
102. A. D. Sarma, A. Jain, A. Nandi, A. G. Parameswaran, and J. Widom, "Surpassing humans and computers with JELLYBEAN: Crowd-vision-hybrid counting algorithms," HCOMP, pp. 178-187, 2015.
103. A. D. Sarma, A. G. Parameswaran, H. Garcia-Molina, and A. Y. Halevy, "Crowd-powered find algorithms," in Proc. Int. Conf. Data Eng., 2014, pp. 964-975.
104. P. Smyth, U. M. Fayyad, M. C. Burl, P. Perona, and P. Baldi, "Inferring ground truth fromsubjective labelling of venus images," in Proc. Adv.Neural Inf. Process. Syst., 1994, pp. 1085-1092.
105. H. Su, K. Zheng, J. Huang, H. Jeung, L. Chen, and X. Zhou, "Crowdplanner: A crowd-based route recommendation system," in Proc. Int. Conf. Data Eng., 2014, pp. 1144-1155.
106. H. Su, K. Zheng, J. Huang, T. Liu, H. Wang, and X. Zhou, "A crowd-based route recommendation system-crowdplanner," in Proc. Int. Conf. Data Eng., 2014, pp. 1178-1181.
107. R. S. Sutton and A. G. Barto, "Reinforcement learning: An introduction," IEEE Trans. Neural Netw., vol. 9, no. 5, pp. 1054-1054, Sep. 1998.
108. K. Talamadupula, S. Kambhampati, Y. Hu, T. A. Nguyen, and H. H. Zhuo," Herding the crowd: Automated planning for crowdsourced planning," in HCOMP, 2013, pp. 70-71.
109. H. To, G. Ghinita, and C. Shahabi," A framework for protecting worker location privacy in spatial crowdsourcing," Proc. VLDB Endowment, vol. 7, no. 10, pp. 919-930, 2014.
110. B. Trushkowsky, T. Kraska, M. J. Franklin, and P. Sarkar, "Crowdsourced enumeration queries," in Proc. Int. Conf. Data Eng., 2013, pp. 673-684.
111. U. ul Hassan and E. Curry, "A multi-armed bandit approach to online spatial task assignment," in Proc. Ubiquitous Intell. Comput., 2014, pp. 212-219.
112. M. Venanzi, J. Guiver, G. Kazai, P. Kohli, and M. Shokouhi, "Community-based bayesian aggregation models for crowdsourcing," in Proc. Int. Conf.WorldWideWeb, 2014, pp. 155-164.
113. P. Venetis, H. Garcia-Molina, K. Huang, and N. Polyzotis, "Max algorithms in crowdsourcing environments," in Proc. Int. Conf. World Wide Web, 2012, pp. 989-998.
114. V. Verroios and H. Garcia-Molina, "Entity resolution with crowd errors," in Proc. Int. Conf. Data Eng., 2015, pp. 219-230.
115. V. Verroios, P. Lofgren, and H. Garcia-Molina, "TDP: An optimal- latency budg et al.ocation strategy for crowdsourced MAXIMUM operations," in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2015, pp. 1047-1062.
116. N. Vesdapunt, K. Bellare, and N. N. Dalvi," Crowdsourcing algorithms for entity resolution," Proc. VLDB Endowment, vol. 7, no. 12, pp. 1071-1082, 2014.
117. L. Von Ahn, B. Maurer, C. McMillen, D. Abraham, and M. Blum, "reCAPTCHA: Human-based character recognition via web security measures," Science, vol. 321, no. 5895, pp. 1465-1468, 2008.
118. J. Wang, T. Kraska, M. J. Franklin, and J. Feng, "CrowdER: crowdsourcing entity resolution," Proc. VLDB Endowment, vol. 5, no. 11, pp. 1483-1494, 2012.
119. J. Wang, S. Krishnan, M. J. Franklin, K. Goldberg, T. Kraska, and T. Milo, "A sample-and-clean framework for fast and accurate query processing on dirty data," in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2014, pp. 469-480.
120. J. Wang, G. Li, and J. Feng, "Can we beat the prefix filtering?: An adaptive framework for similarity join and search," in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2012, pp. 85-96.
121. J. Wang, G. Li, T. Kraska, M. J. Franklin, and J. Feng," Leveraging transitive relations for crowdsourced joins," in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2013, pp. 229-240.
122. S. Wang, X. Xiao, and C. Lee, "Crowd-based deduplication: An adaptive approach," in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2015, pp. 1263-1277.
123. F. L. Wauthier, M. I. Jordan, and N. Jojic, "Efficient ranking from pairwise comparisons," in Proc. Int. Conf. Mach. Learn. Appl., 2013, pp. 109-117.
124. P. Welinder and P. Perona, "Online crowdsourcing: Rating annotators and obtaining cost-effective labels," in Proc. IEEE Conf. Comput. Vis. Pattern Recog. Workshop, 2010, pp. 25-32.
125. S. E. Whang, P. Lofgren, and H. Garcia-Molina," Question selection for crowd entity resolution," Proc. VLDB Endowment, vol. 6, no. 6, pp. 349-360, 2013.
126. S. E. Whang, J. McAuley, and H. Garcia-Molina, "Compare me maybe: Crowd entity resolution interfaces," Stanford University, Stanford, CA, 2012, http://ilpubs.stanford.edu:8090/1061/
127. J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. R. Movellan, "Whose vote should count more: Optimal integration of labels from labelers of unknown expertise," in Proc. Adv. Neural Inf. Process. Syst., 2009, pp. 2035-2043.
128. S. Wu, X. Wang, S. Wang, Z. Zhang, and A. K. H. Tung, "Kanonymity for crowdsourcing database," IEEE Trans. Knowl. Data Eng., vol. 26, no. 9, pp. 2207-2221, Sep. 2014.
129. T. Yan, V. Kumar, and D. Ganesan, "Crowdsearch: Exploiting crowds for accurate real-time image search on mobile phones," in Proc. MobiSys, 2010, pp. 77-90.
130. Y. Yan, G. M. Fung, R. Rosales, and J. G. Dy, "Active learning from crowds," in Proc. Int. Conf. Mach. Learn., 2011, pp. 1161-1168.
131. P. Ye, U. EDU, and D. Doermann," Combining preference and absolute judgements in a crowd-sourced setting," in Proc. Int. Conf. Mach. Learn. Workshop, 2013, pp. 1-6.
132. J. Yi, R. Jin, A. K. Jain, S. Jain, and T. Yang, "Semi-crowdsourced clustering: Generalizing crowd labeling by robust distance metric learning," in Proc. Adv. Neural Inf. Process. Syst., 2012, pp. 1781-1789.
133. C. J. Zhang, L. Chen, H. V. Jagadish, and C. C. Cao, "Reducing uncertainty of schema matching via crowdsourcing," Proc. VLDB Endowment, vol. 6, no. 9, pp. 757-768, 2013.
134. C. J. Zhang, Y. Tong, and L. Chen, "Where to: Crowd-aided path selection," Proc. VLDB Endowment, vol. 7, no. 14, pp. 2005-2016, 2014.
135. X. Zhang, G. Li, and J. Feng, "Crowdsourced top-k algorithms: An experimental evaluation," Proc. VLDB Endowment, vol. 9, no. 4, pp. 612-623, 2015.
136. Z. Zhao, F. Wei, M. Zhou, W. Chen, and W. Ng, "Crowdselection query processing in crowdsourcing databases: A taskdriven approach," in Proc. Int. Conf. Extending Database Technol.: Adv. Database Technol., 2015, pp. 397-408.
137. Z. Zhao, D. Yan, W. Ng, and S. Gao, "A transfer learning based framework of crowd-selection on twitter," in Proc. SIGKDD, 2013, pp. 1514-1517.
138. Y. Zheng, R. Cheng, S. Maniu, and L. Mo, "On optimality of jury selection in crowdsourcing," in Proc. Int. Conf. Extending Database Technol.: Adv. Database Technol., 2015, pp. 193-204.
139. Y. Zheng, J. Wang, G. Li, R. Cheng, and J. Feng, "QASCA: A quality- aware task assignment systemfor crowdsourcing applications," in Proc. ACMSIGMODInt. Conf.Manage. Data, 2015, pp. 1031-1046.
140. J. Zhong, K. Tang, and Z. Zhou, "Active learning from crowds with unsure option," in Proc. Int. Joint Conf. Artif. Intell., 2015, pp. 1061-1068.
141. H. H. Zhuo, "Crowdsourced action-model acquisition for planning," in Proc. Nat. Conf. Artif. Intell., 2015, pp. 3439-3446.