The importance of being expert: Efficient max-finding in crowdsourcing

Proceedings of the ACM SIGMOD International Conference on Management of Data

Volumn 2015-May, Issue , 2015, Pages 983-998

  Anagnostopoulos, Aris ^a   Becchetti, Luca ^a   Fazzone, Adriano ^a   Mele, Ida ^b   Riondato, Matteo ^c  

^a SAPIENZA UNIVERSITY OF ROME   (Italy)

^b MAX PLANCK INSTITUTE FOR INFORMATICS   (Germany)

^c STANFORD UNIVERSITY   (United States)

Author keywords

Crowdsourcing; Human computation; Max algorithm; Worker models

Indexed keywords

APPROXIMATION ALGORITHMS; COMPUTATION THEORY; COMPUTATIONAL EFFICIENCY;

COMPUTATIONAL MODELLING; COMPUTATIONAL PARADIGM; ERROR MODELING; HUMAN COMPUTATION; MAX ALGORITHMS; SPECIALISATION; UNCERTAINTY; WISDOM OF THE CROWDS; WORKER MODELS; WORKERS';

CROWDSOURCING;

EID: 84957603811     PISSN: 07308078     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2723372.2723722     Document Type: Conference Paper

References (37)

1. M. Aigner. Finding the maximum and minimum. Discrete Applied Mathematics, 74(1):1-12, 1997.
2. M. Ajtai, V. Feldman, A. Hassidim, and J. Nelson. Sorting and selection with imprecise comparisons. In Automata, Languages and Programming, volume 5555 of Lecture Notes in Computer Science, pages 37-48, 2009.
3. S. Assaf and E. Upfal. Fault tolerant sorting networks. SIAM Journal on Discrete Mathematics, 4(4):472-480, 1991.
4. A. Bozzon, M. Brambilla, S. Ceri, M. Silvestri, and G. Vesci. Choosing the right crowd: Expert finding in social networks. In Proceedings of the 16th International Conference on Extending Database Technology, EDBT '13, pages 637-648, New York, NY, USA, 2013. ACM.
5. C. C. Cao, J. She, Y. Tong, and L. Chen. Whom to ask?: Jury selection for decision making tasks on micro-blog services. Proc. VLDB Endow., 5(11):1495-1506, July 2012.
6. CSE 2010. Proceedings of the 1st SIGIR Workshop on Crowdsourcing for Information Retrieval (CSE 2010), 2010.
7. A. Das Sarma, A. Parameswaran, H. Garcia-Molina, and A. Halevy. Crowd-powered find algorithms. In Data Engineering (ICDE), 2014 IEEE 30th International Conference on, pages 964-975, March 2014.
8. S. B. Davidson, S. Khanna, T. Milo, and S. Roy. Using the crowd for top-k and group-by queries. In Proceedings of the 16th International Conference on Database Theory, ICDT '13, pages 225-236, New York, NY, USA, 2013. ACM.
9. D. E. Difallah, G. Demartini, and P. Cudré-Mauroux. Pick-a-crowd: Tell me what you like, and i'll tell you what to do. In Proceedings of the 22nd International Conference on World Wide Web, WWW '13, pages 367-374, Republic and Canton of Geneva, Switzerland, 2013. International World Wide Web Conferences Steering Committee.
10. D. P. Dubhashi and A. Panconesi. Concentration of measure for the analysis of randomized algorithms. Cambridge University Press, 2009.
11. U. Feige, P. Raghavan, D. Peleg, and E. Upfal. Computing with noisy information. SIAM Journal on Computing, 23(5):1001-1018, 1994.
12. I. Finocchi and G. F. Italiano. Sorting and searching in the presence of memory faults (without redundancy). In Proceedings of the thirty-sixth annual ACM symposium on Theory of computing, pages 101-110. ACM, 2004.
13. I. Finocchi, F. Grandoni, and G. F. Italiano. Designing reliable algorithms in unreliable memories. Computer Science Review, 1(2):77-87, 2007.
14. M. J. Franklin, D. Kossmann, T. Kraska, S. Ramesh, and R. Xin. Crowddb: Answering queries with crowdsourcing. In Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data, SIGMOD '11, pages 61-72, New York, NY, USA, 2011. ACM.
15. S. Guo, A. Parameswaran, and H. Garcia-Molina. So who won?: dynamic max discovery with the crowd. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, SIGMOD '12, pages 385-396, New York, NY, USA, 2012. ACM.
16. P.-Y. Hsueh, P. Melville, and V. Sindhwani. Data quality from crowdsourcing: a study of annotation selection criteria. In Proceedings of the NAACL HLT 2009 workshop on active learning for natural language processing, pages 27-35. Association for Computational Linguistics, 2009.
17. D. R. Karger, S. Oh, and D. Shah. Iterative learning for reliable crowdsourcing systems. In J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. C. N. Pereira, and K. Q. Weinberger, editors, NIPS, pages 1953-1961, 2011.
18. D. R. Karger, S. Oh, and D. Shah. Budget-optimal crowdsourcing using low-rank matrix approximations. In Communication, Control, and Computing (Allerton), 2011 49th Annual Allerton Conference on, pages 284-291. IEEE, 2011.
19. H. Li, B. Zhao, and A. Fuxman. The wisdom of minority: Discovering and targeting the right group of workers for crowdsourcing. In Proceedings of the 23rd International Conference on World Wide Web, WWW '14, pages 165-176, Republic and Canton of Geneva, Switzerland, 2014. International World Wide Web Conferences Steering Committee.
20. A. Marcus, E. Wu, D. Karger, S. Madden, and R. Miller. Human-powered sorts and joins. Proc. VLDB Endow., 5(1):13-24, Sept. 2011.
21. A. Marcus, E. Wu, D. R. Karger, S. Madden, and R. C. Miller. Crowdsourced databases: Query processing with people. In CIDR, 2011.
22. W. Mason and D. J. Watts. Financial incentives and the "performance of crowds". SIGKDD Explor. Newsl., 11(2):100-108, May 2010.
23. L. Mo, R. Cheng, X. S. Yang, C. Ren, S. Lei, E. Lo, B. Kao, and D. W. Cheung. Optimizing task assignment for crowdsourcing environments. Technical Report HKU CS TR-2013-01, University of Hong Kong, 2013.
24. A. Parameswaran, S. Boyd, H. Garcia-Molina, A. Gupta, N. Polyzotis, and J. Widom. Optimal crowd-powered rating and filtering algorithms. Technical report, Stanford University, 2014. URL http://ilpubs.stanford.edu:8090/1078/.
25. A. G. Parameswaran, H. Garcia-Molina, H. Park, N. Polyzotis, A. Ramesh, and J. Widom. Crowdscreen: Algorithms for filtering data with humans. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, SIGMOD '12, pages 361-372, New York, NY, USA, 2012. ACM.
26. A. Pelc. Searching games with errors - fifty years of coping with liars. Theoretical Computer Science, 270(1):71-109, 2002.
27. A. Ramesh, A. Parameswaran, H. Garcia-Molina, and N. Polyzotis. Identifying reliable workers swiftly. Technical report, Stanford University, 2012. URL http://ilpubs.stanford.edu:8090/1043/.
28. B. Ravikumar, K. Ganesan, and K. B. Lakshmanan. On selecting the largest element in spite of erroneous information. In STACS 87, volume 247 of Lecture Notes in Computer Science, pages 88-99, 1987.
29. Y.-A. Sun, S. Roy, and G. Little. Beyond independent agreement: A tournament selection approach for quality assurance of human computation tasks. In Human Computation, volume WS-11-11 of AAAI Workshops. AAAI, 2011.
30. J. Surowiecki. The wisdom of crowds. Random House LLC, 2005.
31. L. L. Thurstone. A law of comparative judgment. Psychological review, 34(4):273, 1927.
32. B. Trushkowsky, T. Kraska, M. J. Franklin, and P. Sarkar. Crowdsourced enumeration queries. 2013 IEEE 29th International Conference on Data Engineering (ICDE), pages 673-684, 2013.
33. P. Venetis and H. Garcia-Molina. Quality control for comparison microtasks. In CrowdKDD 2012. Stanford InfoLab, August 2012. URL http://ilpubs.stanford.edu:8090/1044/.
34. P. Venetis and H. Garcia-Molina. Dynamic max algorithms in crowdsourcing environments. Technical report, Stanford University, August 2012. URL http://ilpubs.stanford.edu:8090/1050/.
35. P. Venetis, H. Garcia-Molina, K. Huang, and N. Polyzotis. Max algorithms in crowdsourcing environments. In Proceedings of the 21st international conference on World Wide Web, WWW '12, pages 989-998, New York, NY, USA, 2012. ACM.
36. A. Yao and F. Yao. On fault-tolerant networks for sorting. SIAM Journal on Computing, 14(1):120-128, 1985. doi: 10.1137/0214009.
37. L. Zhao, G. Sukthankar, and R. Sukthankar. Robust active learning using crowdsourced annotations for activity recognition. In Human Computation, 2011.