Crowdsourcing Paper Screening in Systematic Literature Reviews

Proceedings of the 5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017

Volumn , Issue , 2017, Pages 108-117

  Krivosheev, Evgeny ^a   Casati, Fabio ^a,b   Caforio, Valentina ^a   Benatallah, Boualem ^c  

^a UNIVERSITY OF TRENTO   (Italy)

^b NATIONAL RESEARCH TOMSK POLYTECHNIC UNIVERSITY   (Russian Federation)

^c UNIVERSITY OF NEW SOUTH WALES   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; BUDGET CONTROL;

CLASSIFICATION ERRORS; LITERATURE REVIEWS; REVIEW PROCESS; SYSTEMATIC LITERATURE REVIEW;

CROWDSOURCING;

EID: 85167572036     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (40)

1. Arora, R. K. 2015. Optimization: algorithms and applications. CRC Press.
2. Dawid, A. P., and Skene, A. M. 1979. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society. Series C Applied Statistics 28(1).
3. Dempster, A. P.; Laird, N. M.; and Rubin, D. B. 1977. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society. Series B (Methodological) 39(1).
4. Eickhoff, C., and de Vries, A. P. 2013. Increasing cheat robustness of crowdsourcing tasks. Information retrieval 16(2):121-137.
5. Eveleigh, A.; Jennett, C.; Blandford, A.; Brohan, P.; and Cox, A. L. 2014. Designing for dabblers and deterring dropouts in citizen science. In Procs of CHI 2014. ACM Press.
6. Frey, B. S., and Jegen, R. 2001. Motivation crowding theory. Journal of Economic Surveys 15(5).
7. Garneau, N. L.; Nuessle, T. M.; Sloan, M. M.; Santorico, S. A.; Coughlin, B. C.; and Hayes, J. E. 2014. Crowdsourcing taste research: genetic and phenotypic predictors of bitter taste perception as a model. Frontiers in integrative neuroscience 8:33.
8. Grant, M., and Booth, A. 2009. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Info Libr J 26(2):91-108.
9. H. Riesch H, C. P. 2014. Citizen science as seen by scientists: Methodological, epistemological and ethical dimensions. TUGBoat 23(1).
10. Haidich, A. 2010. Meta-analysis in medical research. Hippokratia 14(Suppl 1):29-37.
11. Harris, J. D.; Quatman, C. E.; Manring, M.; Siston, R. A.; and Flanigan, D. C. 2014. How to write a systematic review. The American journal of sports medicine 42(11):2761-2768.
12. Henderson, L. K.; Craig, J. C.; Willis, N. S.; Tovey, D.; and Webster, A. C. 2010. How to write a cochrane systematic review. Nephrology 15(6):617-624.
13. Hennon, C. C.; Knapp, K. R.; Schreck III, C. J.; Stevens, S. E.; Kossin, J. P.; Thorne, P. W.; Hennon, P. A.; Kruk, M. C.; Rennie, J.; Gadéa, J.-M.; et al. 2015. Cyclone center: can citizen scientists improve tropical cyclone intensity records? Bulletin of the American Meteorological Society 96(4):591-607.
14. Hirth, M.; Hoßfeld, T.; and Tran-Gia, P. 2011. Cost-optimal validation mechanisms and cheat-detection for crowdsourcing platforms. In Innovative Mobile and Internet Services in Ubiquitous Computing (IMIS), 2011 Fifth International Conference on, 316-321. IEEE.
15. Hirth, M.; Hoßfeld, T.; and Tran-Gia, P. 2013. Analyzing costs and accuracy of validation mechanisms for crowdsourcing platforms. Mathematical and Computer Modelling 57(11):2918-2932.
16. Jeff Pasternack, D. R. 2011. Making better informed trust decisions with generalized fact-finding. IJCAI International Joint Conference on Artificial Intelligence 2324-2329.
17. Karger, D. R.; Oh, S.; and Shah, D. 2011a. Budget-optimal crowdsourcing using low-rank matrix approximations. In Communication, Control, and Computing (Allerton), 2011 49th Annual Allerton Conference on, 284-291. IEEE.
18. Karger, D. R.; Oh, S.; and Shah, D. 2011b. Iterative learning for reliable crowdsourcing systems. In Advances in neural information processing systems, 1953-1961.
19. Khan, K. S.; Kunz, R.; Kleijnen, J.; and Antes, G. 2003. Five steps to conducting a systematic review. Journal of the Royal Society of Medicine 96(3):118-121.
20. Law, E.; Gajos, K. Z.; Wiggins, A.; Gray, M. L.; and Williams, A. 2017. Crowdsourcing as a tool for research: Implications of uncertainty. In Proceedings of the 2017 ACMConference on Computer Supported Cooperative Work and Social Computing, CSCW '17, 1544-1561. New York, NY, USA: ACM.
21. Lintott, C. J.; Schawinski, K.; Slosar, A.; Land, K.; Bamford, S.; Thomas, D.; Raddick, M. J.; Nichol, R. C.; Szalay, A.; Andreescu, D.; et al. 2008. Galaxy zoo: morphologies derived from visual inspection of galaxies from the sloan digital sky survey. Monthly Notices of the Royal Astronomical Society 389(3):1179-1189.
22. Liu, C., and Wang, Y. M. 2012. Truelabel + confusions: A spectrum of probabilistic models in analyzing multiple ratings. In Procs of ICML2012. ICML.
23. Liu, Q.; Ihler, A. T.; and Steyvers, M. 2013. Scoring workers in crowdsourcing: How many control questions are enough? In Advances in Neural Information Processing Systems, 1914-1922.
24. Moher, D.; Cook, D. J.; Eastwood, S.; Olkin, I.; Rennie, D.; and Stroup, D. F. 1999. Improving the quality of reports of meta-analyses of randomised controlled trials: the {QUOROM} statement. The Lancet 354(9193):1896-1900.
25. Moher, D.; Liberati, A.; Tetzlaff, J.; Altman, D. G.; Group, P.; et al. 2009. Preferred reporting items for systematic reviews and meta-analyses: the prisma statement. PLoS med 6(7):e1000097.
26. Ng, L.; Pitt, V.; Huckvale, K.; Clavisi, O.; Turner, T.; Gruen, R.; and Elliott, J. H. 2014. Title and Abstract Screening and Evaluation in Systematic Reviews (TASER): a pilot randomised controlled trial of title and abstract screening by medical students. Systematic reviews 3(1):121.
27. Nguyen, A. T.; Wallace, B. C.; and Lease, M. 2015. Combining Crowd and Expert Labels using Decision Theoretic Active Learning. Proceedings of the 3rd AAAI Conference on Human Computation (HCOMP) 120-129.
28. Sampson, M.; Shojania, K. G.; Garritty, C.; Horsley, T.; Ocampo, M.; and Moher, D. 2008. Systematic reviews can be produced and published faster. Journal of clinical epidemiology 61(6):531-536.
29. Smyth, P.; Fayyad, U.; Burl, M.; Perona, P.; and Baldi, P. 1995. Inferring ground truth from subjective labelling of venus images. Advances in neural information processing systems 7:1085-1092.
30. Steinberg, E.; Greenfield, S.; Wolman, D. M.; Mancher, M.; Graham, R.; et al. 2011. Clinical practice guidelines we can trust. National Academies Press.
31. Sun, Y.; Cheng, P.; Wang, S.; Lyu, H.; Lease, M.; Marshall, I.; and Wallace, B. C. 2016. Crowdsourcing Information Extraction for Biomedical Systematic Reviews. In 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP): Works-in-Progress Track. 3 pages. arXiv:1609.01017.
32. Swanson, A.; Kosmala, M.; Lintott, C.; Simpson, R.; Smith, A.; and Packer, C. 2015. Snapshot serengeti, high-frequency annotated camera trap images of 40 mammalian species in an african savanna. Scientific data 2:150026.
33. Takwoingi, Y.; Hopewell, S.; Tovey, D.; and Sutton, A. J. 2013. A multicomponent decision tool for prioritising the updating of systematic reviews. Bmj 7191(December):1-8.
34. Tinati, R.; Kleek, M. V.; Simperl, E.; Luczak-Roesch, M.; Simpson, R.; and Shadbolt, N. 2015. Designing for citizen data analysis: A cross-sectional case study of a multidomain citizen science platform. In Procs of CHI 2015. ACM Press.
35. Veronese, N.; Facchini, S.; Stubbs, B.; Luchini, C.; Solmi, M.; Manzato, E.; Sergi, G.; Maggi, S.; Cosco, T.; and Fontana, L. 2017. Weight loss is associated with improvements in cognitive function among overweight and obese people: A systematic review and meta-analysis. Neuroscience & Biobehavioral Reviews 72:87-94.
36. Wallace, B. C.; Dahabreh1, I. J.; Schmid, C. H.; Lau1, J.; and Trikalinos, T. A. 2013. Modernizing the systematic review process to inform comparative effectiveness: tools and methods. J. Compar. Effect. Res. 2(3).
37. Weiss, M. 2016. Crowdsourcing literature reviews in new domains. Technology Innovation Management Review 6(2):5-14.
38. Whitehill, J.; Wu, T.-f.; Bergsma, J.; Movellan, J. R.; and Ruvolo, P. L. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in neural information processing systems, 2035-2043.
39. Wright, R. W.; Brand, R. A.; Dunn, W.; and Spindler, K. P. 2007. How to write a systematic review. Clinical orthopaedics and related research 455:23-29.
40. Yang, J.; Redi, J.; DeMartini, G.; and Bozzon, A. 2016. Modeling task complexity in crowdsourcing. In Proceedings of The Fourth AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2016), 249-258. AAAI.