Crowd-based multi-predicate screening of papers in literature reviews

The Web Conference 2018 - Proceedings of the World Wide Web Conference, WWW 2018

Volumn , Issue , 2018, Pages 55-64

  Krivosheev, Evgeny ^a   Casati, Fabio ^a   Benatallah, Boualem ^b  

^a UNIVERSITY OF TRENTO   (Italy)

^b UNIVERSITY OF NEW SOUTH WALES   (Australia)

Author keywords

Crowdsourcing; Human computation; Literature reviews

Indexed keywords

CROWDSOURCING; WORLD WIDE WEB;

ADAPTIVE APPROACH; ADAPTIVE STRATEGY; LITERATURE REVIEWS; RESEARCH COMMUNITIES; SCIENTIFIC RESEARCHES; STATISTICAL PROPERTIES; SYSTEMATIC LITERATURE REVIEW;

DIAGNOSIS;

EID: 85066423332     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/3178876.3186036     Document Type: Conference Paper

References (43)

1. H. Arkes and C. Blumer. 1985. The psychology of sunk cost. Organizational Behavior and Human Decision Processes 35, 1 (1985).
2. Jonathan Bragg, Mausam, and Daniel S. Weld. 2016. Optimal Testing for Crowd Workers. In Proceedings of the 2016 International Conference on Autonomous Agents &38; Multiagent Systems (AAMAS '16). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 966-974. http://dl. acm.org/citation.cfm?id=2937029.2937066
3. Andrew W. Brown and David B. Allison. 2014. Using Crowdsourcing to Evaluate Published Scientific Literature: Methods and Example. Plos One 9, 7 (2014).
4. Perrine Créquit, Ludovic Trinquart, Amélie Yavchitz, and Philippe Ravaud. 2016. Wasted research when systematic reviews fail to provide a complete and up-todate evidence synthesis: the example of lung cancer. BMC Medicine 14, 1 (2016), 8. https://doi.org/10.1186/s12916-016-0555-0
5. A. P. Dawid and A. M. Skene. 1979. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Journal of the Royal Statistical Society. Series C Applied Statistics 28, 1 (1979).
6. A. P. Dempster, N. M. Laird, and D. B. Rubin. 1977. Maximum Likelihood from Incomplete Data via the EM Algorithm. Journal of the Royal Statistical Society. Series B (Methodological) 39, 1 (1977).
7. Xin Luna Dong, Laure Berti-Equille, and Divesh Srivastava. 2013. Data Fusion : Resolving Conflicts from Multiple Sources. In Procs of WAIM2013. Springer. https://doi.org/10.1007/978-3-642-36257-6
8. Carsten Eickhoff and Arjen P de Vries. 2013. Increasing cheat robustness of crowdsourcing tasks. Information retrieval 16, 2 (2013), 121-137.
9. M.J. Grant and A. Booth. 2009. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Info Libr J 26, 2 (2009), 91-108.
10. JPT Higginsand S. Green. 2011. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. The Cochrane Collaboration. Available from www.handbook.cochrane.org.
11. A. Haidich. 2010. Meta-analysis in medical research. Hippokratia 14, Suppl 1 (2010), 29-37.
12. Joseph M. Hellerstein and Michael Stonebraker. 1993. Predicate Migration: Optimizing Queries with Expensive Predicates. In Proceedings of ACM SIGMOD. ACM.
13. Lorna K Henderson, Jonathan C Craig, Narelle S Willis, David Tovey, and Angela C Webster. 2010. How to write a Cochrane systematic review. Nephrology 15, 6 (2010), 617-624.
14. Matthias Hirth, Tobias Hoßfeld, and Phuoc Tran-Gia. 2011. Cost-optimal validation mechanisms and cheat-detection for crowdsourcing platforms. In Innovative Mobile and Internet Services in Ubiquitous Computing (IMIS), 2011 Fifth International Conference on. IEEE, 316-321.
15. Matthias Hirth, Tobias Hoßfeld, and Phuoc Tran-Gia. 2013. Analyzing costs and accuracy of validation mechanisms for crowdsourcing platforms. Mathematical and Computer Modelling 57, 11 (2013), 2918-2932.
16. Yuan Jin, Mark Carman, Dongwoo Kim, and Lexing Xie. 2017. Leveraging Side Information to Improve Label Quality Control in Crowd-Sourcing. In Procs of Hcomp2017. AAAI.
17. David R Karger, Sewoong Oh, and Devavrat Shah. 2011. Budget-optimal crowdsourcing using low-rank matrix approximations. In Communication, Control, and Computing (Allerton), 2011 49th Annual Allerton Conference on. IEEE, 284-291.
18. David R Karger, Sewoong Oh, and Devavrat Shah. 2011. Iterative learning for reliable crowdsourcing systems. In Advances in neural information processing systems. 1953-1961.
19. Khalid S Khan, Regina Kunz, Jos Kleijnen, and Gerd Antes. 2003. Five steps to conducting a systematic review. Journal of the Royal Society of Medicine 96, 3 (2003), 118-121.
20. Evgeny Krivosheev, Boualem Benatallah Bahareh Harandizadeh, and Fabio Casati. 2018. Crowd-based Multi-predicate Screening of Papers in Literature Reviews (poster). In Proceedings ofWWW2018. InternationalWorld WideWeb Conferences Steering Committee.
21. Evgeny Krivosheev, Valentina Caforio, Boualem Benatallah, and Fabio Casati. 2017. Crowdsourcing Paper Screening in Systematic Literature Reviews. In Procs of Hcomp2017. AAAI.
22. Edith Law, Krzysztof Z. Gajos, Andrea Wiggins, Mary L. Gray, and Alex Williams. 2017. Crowdsourcing As a Tool for Research: Implications of Uncertainty. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17). ACM, New York, NY, USA, 1544-1561. https://doi.org/10.1145/2998181.2998197
23. Hongwei Li, Bin Yu, and Dengyong Zhou. 2013. Error Rate Analysis of Labeling by Crowdsourcing. In Procs of ICML2013.
24. Chao Liu and Yi Min Wang. 2012. TrueLabel + Confusions: A Spectrum of Probabilistic Models in Analyzing Multiple Ratings. In Procs of ICML2012. ICML.
25. Qiang Liu, Alexander T Ihler, and Mark Steyvers. 2013. Scoring workers in crowdsourcing: How many control questions are enough?. In Advances in Neural Information Processing Systems. 1914-1922.
26. FJ Mateen, J Oh, AI Tergas, NH Bhayani, and BB Kamdar. 2013. Titles versus titles and abstracts for initial screening of articles for systematic reviews. Clin Epidemiol. 5, 1 (2013).
27. David Moher, Alessandro Liberati, Jennifer Tetzlaff, Douglas G Altman, Prisma Group, et al. 2009. Preferred reporting items for systematic reviews and metaanalyses: the PRISMA statement. PLoS med 6, 7 (2009), e1000097.
28. Michael L. Mortensen, Gaelen P. Adam, Thomas A. Trikalinos, Tim Kraska, and Byron C. Wallace. 2016. An exploration of crowdsourcing citation screening for systematic reviews. Research Synthesis Methods (2016). RSM-02-2016-0006.R4.
29. Lauren Ng, Veronica Pitt, Kit Huckvale, Ornella Clavisi, Tari Turner, Russell Gruen, and Julian H Elliott. 2014. Title and Abstract Screening and Evaluation in Systematic Reviews (TASER): a pilot randomised controlled trial of title and abstract screening by medical students. Systematic reviews 3, 1 (2014), 121. https://doi.org/10.1186/2046-4053-3-121
30. An T Nguyen, Byron C Wallace, and Matthew Lease. 2015. Combining Crowd and Expert Labels using Decision Theoretic Active Learning. Proceedings of the 3rd AAAI Conference on Human Computation (HCOMP) (2015), 120-129.
31. Jungseul Ok, Sewoong Oh, Jinwoo Shin, and Yung Yi. 2016. Optimality of Belief Propagation for Crowdsourced Classification. In Procs of ICML2016.
32. Aditya Parameswaran, Hector Garcia-Molina, Hyunjung Park, Neoklis Polyzotis, Aditya Ramesh, and Jennifer Widom. 2012. CrowdScreen: Algorithms for Filtering Data with Humans. In Proceedings of ACM SIGMOD. ACM.
33. Margaret Sampson, Kaveh G Shojania, Chantelle Garritty, Tanya Horsley, Mary Ocampo, and David Moher. 2008. Systematic reviews can be produced and published faster. Journal of clinical epidemiology 61, 6 (2008), 531-536.
34. Padhraic Smyth, Usama Fayyad, Michael Burl, Pietro Perona, and Pierre Baldi. 1995. Inferring ground truth from subjective labelling of venus images. Advances in neural information processing systems 7 (1995), 1085-1092.
35. Earl Steinberg, Sheldon Greenfield, Dianne Miller Wolman, Michelle Mancher, Robin Graham, et al. 2011. Clinical practice guidelines we can trust. National Academies Press.
36. Yalin Sun, Pengxiang Cheng, Shengwei Wang, Hao Lyu, Matthew Lease, Iain Marshall, and Byron C.Wallace. 2016. Crowdsourcing Information Extraction for Biomedical Systematic Reviews. In 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP): Works-in-Progress Track. http://arxiv.org/abs/1609. 01017 3 pages. arXiv:1609.01017.
37. Yemisi Takwoingi, Sally Hopewell, David Tovey, and Alex J Sutton. 2013. A multicomponent decision tool for prioritising the updating of systematic reviews. Bmj 7191, December (2013), 1-8. https://doi.org/10.1136/bmj.f7191
38. Nicola Veronese, Silvia Facchini, Brendon Stubbs, Claudio Luchini, Marco Solmi, Enzo Manzato, Giuseppe Sergi, Stefania Maggi, Theodore Cosco, and Luigi Fontana. 2017. Weight loss is associated with improvements in cognitive function among overweight and obese people: A systematic review and meta-analysis. Neuroscience &Biobehavioral Reviews 72 (2017), 87-94.
39. Byron C Wallace, Anna Noel-Storr, Iain J Marshall, Aaron M Cohen, Neil R Smalheiser, and James Thomas. 2017. Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach. J Am Med Inform Assoc (2017).
40. Michael Weiss. 2016. Crowdsourcing literature reviews in new domains. Technology Innovation Management Review 6, 2 (2016), 5-14.
41. Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier R Movellan, and Paul L Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. (2009), 2035-2043.
42. Jie Yang, Judith Redi, Gianluca DeMartini, and Alessandro Bozzon. 2016. Modeling Task Complexity in Crowdsourcing. In Proceedings of The Fourth AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2016). AAAI, 249-258.
43. D. Zhou, J. Platt, S. Basu, and Y. Mao. 2012. Learning from the wisdom of crowds by minimax entropy. In Procs of Nips 2012.