An exploration of crowdsourcing citation screening for systematic reviews

Research Synthesis Methods

Volumn 8, Issue 3, 2017, Pages 366-386

  Mortensen, Michael L ^a,e   Adam, Gaelen P ^b   Trikalinos, Thomas A ^b   Kraska, Tim ^c   Wallace, Byron C ^d  

^a Netcompany A S   (Denmark)

^b BROWN UNIVERSITY   (United States)

^c Brown Univerity   (United States)

^d Northeastern University   (United States)

^e AARHUS UNIVERSITY   (Denmark)

Author keywords

crowdsourcing (MeSH); evidence based medicine (MeSH); review literature as topic (MeSH); study selection; systematic review methods

Indexed keywords

ALGORITHM; CROWDSOURCING; HUMAN; LITERATURE;

ALGORITHMS; CROWDSOURCING; HUMANS; REVIEW LITERATURE AS TOPIC;

EID: 85021741349     PISSN: None     EISSN: 17592887     Source Type: Journal    
DOI: 10.1002/jrsm.1252     Document Type: Article

References (24)

1. Allen IEOI. Estimating time to conduct a meta-analysis from number of citations retrieved. JAMA. 1999;282:634-635.
2. Bastian HGP, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7: e1000326.
3. O'Mara-Eves ATJ, McnaughT J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev. 2015;4:5.
4. Tsafnat GDA, Glasziou P, Coiera E. The automation of systematic reviews. BMJ. 2013;346:f139.
5. Wallace BC, Dahabreh IJ, Schmid CH, Lau J, Trikalinos TA. Modernizing the systematic review process to inform comparative effectiveness: tools and methods. J Comp Eff Res. 2013;2:273-282.
6. Howe J. The rise of crowdsourcing. Wired magazine. 2006;14:1-4.
7. Kittur A, Chi EH, Suh B. Crowdsourcing user studies with Mechanical Turk. Proceedings of the SIGCHI conference on human factors in computing systems, 2008. ACM, 453–456.
8. Gao H, Barbier G, Goolsby R, Zeng D. Harnessing the crowdsourcing power of social media for disaster relief. 2011. DTIC Document.
9. Goodchild MF, Glennon JA. Crowdsourcing geographic information for disaster response: a research frontier. International Journal of Digital Earth. 2010;3:231-241.
10. Alonso O, Lease M. Crowdsourcing for information retrieval: principles, methods, and applications. Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, 2011. ACM, 1299–1300.
11. Alonso O, Mizzaro S. Using crowdsourcing for TREC relevance assessment. Information processing & management. 2012;48:1053-1066.
12. Silvertown J. A new dawn for citizen science. Trends Ecol Evol. 2009;24:467-471.
13. Lintott CJ, Schawinski K, Slosar A, et al. Galaxy Zoo: morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey. Monthly Notices of the Royal Astronomical Society. 2008;389:1179-1189.
14. Collaboration C. The EMBASE screening project: six months old and going strong. 2014. [Online]. Available: http://community.cochrane.org/news/news-events/current-news/embase-screening-project-six-months-old-and-going-strong Accessed 5 November 2015.
15. Terasawa TDT, Ip S, Raman G, Lau J, Trikalinos TA. Systematic review: Charged-particle radiation therapy for cancer. Ann Intern Med. 2009;151:556-565.
16. Dahabreh IJ, Adam GP, Halladay CW. et al. Diagnosis of right lower quadrant pain and suspected acute appendicitis (Protocol). 2013. Available: http://effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?productid=1827&pageaction=displayproduct [Accessed May 4, 2015].
17. Trikalinos TAWLS, Adam GP, Zgodic A, Ntzani EE. Decision aids for cancer screening and treatment, Rockville MD; 2014.
18. Balk E. Omega 3 fatty acids and cardiovascular disease—update (protocol). 2015. Available: http://effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?productid=2060&pageaction=displayproduct [Accessed May 4, 2015].
19. Guidelines for Academic Requesters. [Online]. DynamoWiki. 2014. Available: http://wiki.wearedynamo.org/index.php/Guidelines_for_Academic_Requesters [Accessed 8/14/15].
20. Ross J, Irani L, Silberman M, Zaldivar A, Tomlinson B. Who are the crowdworkers?: Shifting demographics in mechanical turk. CHI'10 Extended Abstracts on Human Factors in Computing Systems, 2010. ACM, 2863–2872.
21. Downs JS, Holbrook MB, Sheng S, Cranor LF. Are your participants gaming the system?: screening mechanical turk workers. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2010. ACM, 2399–2402.
22. Fleiss JL. Measuring nominal scale agreement among many raters. Psychol Bull. 1971;76:378
23. Heer J, Bostock M. Crowdsourcing graphical perception: using mechanical turk to assess visualization design. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2010. ACM, 203–212.
24. Nguyen AT, Wallace BC, Lease M. Combining crowd and expert labels using decision theoretic active learning. Third AAAI Conference on Human Computation and Crowdsourcing, 2015.