Combining crowd and machines for multi-predicate item screening

Proceedings of the ACM on Human-Computer Interaction

Volumn 2, Issue CSCW, 2018, Pages

  Krivosheev, Evgeny ^a   Casati, Fabio ^a,b   Baez, Marcos ^a   Benatallah, Boualem ^c  

^a UNIVERSITY OF TRENTO   (Italy)

^b NATIONAL RESEARCH TOMSK POLYTECHNIC UNIVERSITY   (Russian Federation)

^c UNIVERSITY OF NEW SOUTH WALES   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

BUDGET CONTROL; COST ESTIMATING; ECONOMIC AND SOCIAL EFFECTS;

CLASSIFICATION ALGORITHM; ENSEMBLE OF CLASSIFIERS; TEST DATA; TRADE OFF;

COST BENEFIT ANALYSIS;

EID: 85064888586     PISSN: None     EISSN: 25730142     Source Type: Journal    
DOI: 10.1145/3274366     Document Type: Article

References (45)

1. Ron Avnur and Joseph M. Hellerstein. 2000. Eddies: Continuously Adaptive Query Processing. SIGMOD Rec. 29, 2 (May 2000), 261–272. https://doi.org/10.1145/335191.335420
2. Shivnath Babu, Rajeev Motwani, Kamesh Munagala, Itaru Nishizawa, and Jennifer Widom. 2004. Adaptive Ordering of Pipelined Stream Filters. In Proceedings of ACM SIGMOD. ACM.
3. Justin Cheng and Michael S. Bernstein. 2015. Flock: Hybrid Crowd-Machine Learning Classifiers. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW’15). ACM, New York, NY, USA, 600–611. https://doi.org/10.1145/2675133.2675214
4. Peng Dai, Christopher H. Lin, Mausam, and Daniel S. Weld. 2013. POMDP-based Control of Workflows for Crowdsourcing. Artif. Intell. 202, 1 (Sept. 2013), 52–85. https://doi.org/10.1016/j.artint.2013.06.002
5. A. P. Dawid and A. M. Skene. 1979. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Journal of the Royal Statistical Society. Series C Applied Statistics 28, 1 (1979).
6. A. P. Dempster, N. M. Laird, and D. B. Rubin. 1977. Maximum Likelihood from Incomplete Data via the EM Algorithm. Journal of the Royal Statistical Society. Series B (Methodological) 39, 1 (1977).
7. Amol Deshpande, Zachary Ives, and Vijayshankar Raman. 2007. Adaptive Query Processing. Foundations and TrendsÂő in Databases 1, 1 (2007), 1–140. https://doi.org/10.1561/1900000001
8. Thomas G Dietterich. 2000. An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees: Bagging, Boosting, and Randomization. Machine Learning 40, 2 (2000).
9. Xin Luna Dong, Laure Berti-Equille, and Divesh Srivastava. 2013. Data Fusion: Resolving Conflicts from Multiple Sources. In Procs of WAIM2013. Springer. https://doi.org/10.1007/978-3-642-36257-6
10. Saso Džeroski and Bernard Ženko. 2004. Is Combining Classifiers with Stacking Better than Selecting the Best One? Machine Learning 54, 3 (01 Mar 2004), 255–273.
11. Carsten Eickhoff and Arjen P de Vries. 2013. Increasing cheat robustness of crowdsourcing tasks. Information retrieval 16, 2 (2013), 121–137.
12. Michael J. Franklin, Donald Kossmann, Tim Kraska, Sukriti Ramesh, and Reynold Xin. 2011. CrowdDB: Answering Queries with Crowdsourcing. In Proceedings of ACM SIGMOD. ACM.
13. Ryan Gomes, Peter Welinder, Andreas Krause, and Pietro Perona. 2011. Crowdclustering. In Procs of Nips 2011.
14. Joseph M. Hellerstein and Michael Stonebraker. 1993. Predicate Migration: Optimizing Queries with Expensive Predicates. In Proceedings of ACM SIGMOD. ACM.
15. Matthias Hirth, Tobias Hoßfeld, and Phuoc Tran-Gia. 2011. Cost-optimal validation mechanisms and cheat-detection for crowdsourcing platforms. In Innovative Mobile and Internet Services in Ubiquitous Computing (IMIS), 2011 Fifth International Conference on. IEEE, 316–321.
16. Matthias Hirth, Tobias Hoßfeld, and Phuoc Tran-Gia. 2013. Analyzing costs and accuracy of validation mechanisms for crowdsourcing platforms. Mathematical and Computer Modelling 57, 11 (2013), 2918–2932.
17. Ece Kamar, Severin Hacker, and Eric Horvitz. 2012. Combining Human and Machine Intelligence in Large-scale Crowdsourcing. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems (AAMAS’12). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 467–474.
18. Ece Kamar, Ashish Kapoor, and Eric Horvitz. 2013. Lifelong learning for acquiring the wisdom of the crowd. In JCAI.
19. David R Karger, Sewoong Oh, and Devavrat Shah. 2011. Budget-optimal crowdsourcing using low-rank matrix approximations. In Communication, Control, and Computing (Allerton), 2011 49th Annual Allerton Conference on. IEEE, 284–291.
20. David R Karger, Sewoong Oh, and Devavrat Shah. 2011. Iterative learning for reliable crowdsourcing systems. In Advances in neural information processing systems. 1953–1961.
21. Evgeny Krivosheev, Boualem Benatallah, and Fabio Casati. 2018. Crowd-based Multi-predicate Screening of Papers in Literature Reviews. In Proceedings of WWW2018. International World Wide Web Conferences Steering Committee.
22. Evgeny Krivosheev, Valentina Caforio, Boualem Benatallah, and Fabio Casati. 2017. Crowdsourcing Paper Screening in Systematic Literature Reviews. In Procs of Hcomp2017. AAAI.
23. Doren Lan, Katherine Reed, Austin Shin, and Beth Trushkowsky. 2017. Dynamic Filter: Adaptive Query Processing with the Crowd. In Procs of Hcomp2017. AAAI.
24. Hongwei Li, Bin Yu, and Dengyong Zhou. 2013. Error Rate Analysis of Labeling by Crowdsourcing. In Procs of ICML2013.
25. Chao Liu and Yi Min Wang. 2012. TrueLabel + Confusions: A Spectrum of Probabilistic Models in Analyzing Multiple Ratings. In Procs of ICML2012. ICML.
26. Qiang Liu, Alexander T Ihler, and Mark Steyvers. 2013. Scoring workers in crowdsourcing: How many control questions are enough?. In Advances in Neural Information Processing Systems. 1914–1922.
27. Michael L. Mortensen, Gaelen P. Adam, Thomas A. Trikalinos, Tim Kraska, and Byron C. Wallace. 2016. An exploration of crowdsourcing citation screening for systematic reviews. Research Synthesis Methods (2016). RSM-02-2016-0006.R4.
28. An Thanh Nguyen, Matthew Halpern, Byron C. Wallace, and Matthew Lease. 2015. Probabilistic Modeling for Crowdsourcing Partially-Subjective Ratings. In Procs of HComp2015. AAAI Publications.
29. An T Nguyen, Byron C Wallace, and Matthew Lease. 2015. Combining Crowd and Expert Labels using Decision Theoretic Active Learning. Proceedings of the 3rd AAAI Conference on Human Computation (HCOMP) (2015), 120–129.
30. Besmira Nushi, Adish Singla, Anja Gruenheid, Erfan Zamanian, Andreas Krause, and Donald Kossmann. 2015. Crowd Access Path Optimization: Diversity Matters. In HCOMP.
31. Jungseul Ok, Sewoong Oh, Jinwoo Shin, and Yung Yi. 2016. Optimality of Belief Propagation for Crowdsourced Classification. In Procs of ICML2016.
32. Aditya Parameswaran, Stephen Boyd, Hector Garcia-Molina, Ashish Gupta, Neoklis Polyzotis, and Jennifer Widom. 2014. Optimal crowd-powered rating and filtering algorithms. In Proceedings of VLDB. VLDB Endowment.
33. Aditya Parameswaran, Hector Garcia-Molina, Hyunjung Park, Neoklis Polyzotis, Aditya Ramesh, and Jennifer Widom. 2012. CrowdScreen: Algorithms for Filtering Data with Humans. In Proceedings of ACM SIGMOD. ACM.
34. Hyunjung Park and Jennifer Widom. 2013. Query Optimization over Crowdsourced Data. Proc. VLDB Endow. 6, 10 (Aug. 2013), 781–792. https://doi.org/10.14778/2536206.2536207
35. Carlos Rodriguez, Florian Daniel, and Fabio Casati. 2014. Crowd-Based Mining of Reusable Process Model Patterns. In Business Process Management, Shazia Sadiq, Pnina Soffer, and Hagen Völzer (Eds.). Springer International Publishing, 51–66.
36. Lior Rokach. 2010. Ensemble-based Classifiers. Artif. Intell. Rev. 33, 1-2 (Feb. 2010), 1–39. https://doi.org/10.1007/ s10462-009-9124-7
37. Padhraic Smyth, Usama Fayyad, Michael Burl, Pietro Perona, and Pierre Baldi. 1995. Inferring ground truth from subjective labelling of venus images. Advances in neural information processing systems 7 (1995), 1085–1092.
38. Yalin Sun, Pengxiang Cheng, Shengwei Wang, Hao Lyu, Matthew Lease, Iain Marshall, and Byron C. Wallace. 2016. Crowdsourcing Information Extraction for Biomedical Systematic Reviews. In 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP): Works-in-Progress Track. http://arxiv.org/abs/1609.01017 3 pages. arXiv:1609.01017.
39. Jennifer Wortman Vaughan. 2017. Making Better Use of the Crowd: How Crowdsourcing Can Advance Machine Learning Research. Survey and Position Paper. Microsoft Research. Available at http://www.jennwv.com/projects/crowdtutorial.html.
40. Norases Vesdapunt, Kedar Bellare, and Nilesh Dalvi. 2014. Crowdsourcing Algorithms for Entity Resolution. In Proceedings of VLDB. VLDB Endowment.
41. Ramya Korlakai Vinayak and Babak Hassibi. 2016. Crowdsourced clustering: Querying edges vs triangles. In Procs of Nips 2016.
42. Byron C Wallace, A Noel-Storr, IJ Marshall, AM Cohen, NR Smalheiser, and J Thomas. 2017. Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach. J Am Med Inform Assoc (2017).
43. Jiannan Wang, Tim Kraska, Michael J. Franklin, and Jianhua Feng. 2012. CrowdER: Crowdsourcing Entity Resolution. In Proceedings of VLDB. VLDB Endowment.
44. Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier R Movellan, and Paul L Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. (2009), 2035–2043.
45. D. Zhou, J. Platt, S. Basu, and Y. Mao. 2012. Learning from the wisdom of crowds by minimax entropy. In Procs of Nips 2012.