Crowdsourcing the citation screening process for systematic reviews: Validation study

Journal of Medical Internet Research

Volumn 21, Issue 4, 2019, Pages

  Nama, Nassr ^a,b,c,d   Sampson, Margaret ^b   Barrowman, Nicholas ^a,b   Sandarage, Ryan ^c   Menon, Kusum ^a,b   Macartney, Gail ^b   Murto, Kimmo ^a,b   Vaccani, Jean Philippe ^a,b   Katz, Sherri ^a,b   Zemek, Roger ^a,b,e   Nasr, Ahmed ^a,b   McNally, James Dayre ^a,b  

^a UNIVERSITY OF OTTAWA   (Canada)

^b CHILDREN S HOSPITAL OF EASTERN ONTARIO   (Canada)

^c UNIVERSITY OF BRITISH COLUMBIA   (Canada)

^d BC CHILDREN S HOSPITAL   (Canada)

^e Faculty of Medicine   (Canada)

Author keywords

Crowdsourcing; Meta analysis as topic; Research design; Systematic reviews as topic

Indexed keywords

ADULT; CONTROLLED STUDY; CROWDSOURCING; DIAGNOSTIC TEST ACCURACY STUDY; FEASIBILITY STUDY; FEMALE; GOLD STANDARD; HUMAN; INFORMATION RETRIEVAL; MALE; MEDICAL STUDENT; PROSPECTIVE STUDY; REVIEW; SENSITIVITY AND SPECIFICITY; SOFTWARE; SYNTHESIS; SYSTEMATIC REVIEW; SYSTEMATIC REVIEW (TOPIC); VALIDATION PROCESS; VALIDATION STUDY; MASS SCREENING; METHODOLOGY; PROCEDURES;

CROWDSOURCING; HUMANS; MASS SCREENING; RESEARCH DESIGN; VALIDATION STUDIES AS TOPIC;

EID: 85065428868     PISSN: None     EISSN: 14388871     Source Type: Journal    
DOI: 10.2196/12953     Document Type: Review

References (46)

1. Pearson A, Wiechula R, Court A, Lockwood C. The JBI model of evidence-based healthcare. Int J Evid Based Healthc 2005 Sep; 3 (8): 207-215. [doi: 10. 1111/j. 1479-6988. 2005. 00026. x] [Medline: 21631749]
2. Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep uP. PLoS Med 2010 Sep 21; 7 (9): e1000326 [FREE Full text] [doi: 10. 1371/journal. pmed. 1000326] [Medline: 20877712]
3. Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S, Moher D. How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med 2007 Aug 21; 147 (4): 224-233. [doi: 10. 7326/0003-4819-147-4-200708210-00179] [Medline: 17638714]
4. Xu X, Li Y, Sheng Y, Liu J, Tang L, Chen Z. Effect of low birth weight on childhood asthma: a meta-analysis. BMC Pediatr 2014 Oct 23; 14: 275 [FREE Full text] [doi: 10. 1186/1471-2431-14-275] [Medline: 25339063]
5. Kelley GA, Kelley KS, Pate RR. Effects of exercise on BMI z-score in overweight and obese children and adolescents: a systematic review with meta-analysis. BMC Pediatr 2014 Sep 9; 14: 225 [FREE Full text] [doi: 10. 1186/1471-2431-14-225] [Medline: 25204857]
6. Sun J, Zheng J, Tang L, Healy J, Sinnett D, Dai Y. Association between CEBPE variant and childhood acute leukemia risk: evidence from a meta-analysis of 22 studies. PLoS One 2015; 10 (5): e0125657 [FREE Full text] [doi: 10. 1371/journal. pone. 0125657] [Medline: 25938438]
7. Wallace AE, Neily J, Weeks WB, Friedman MJ. A cumulative meta-analysis of selective serotonin reuptake inhibitors in pediatric depression: did unpublished studies influence the efficacy/safety debate? J Child Adolesc Psychopharmacol 2006; 16 (1-2): 37-58. [doi: 10. 1089/cap. 2006. 16. 37] [Medline: 16553528]
8. Allen IE, Olkin I. Estimating time to conduct a meta-analysis from number of citations retrieved. J Am Med Assoc 1999 Aug 18; 282 (7): 634-635. [doi: 10. 1001/jama. 282. 7. 634] [Medline: 10517715]
9. Jonnalagadda S, Petitti D. A new iterative method to reduce workload in systematic review process. Int J Comput Biol Drug Des 2013; 6 (1-2): 5-17 [FREE Full text] [doi: 10. 1504/IJCBDD. 2013. 052198] [Medline: 23428470]
10. Matwin S, Kouznetsov A, Inkpen D, Frunza O, O'Blenis P. A new algorithm for reducing the workload of experts in performing systematic reviews. J Am Med Inform Assoc 2010; 17 (4): 446-453 [FREE Full text] [doi: 10. 1136/jamia. 2010. 004325] [Medline: 20595313]
11. Cohen AM, Hersh WR, Peterson K, Yen P. Reducing workload in systematic review preparation using automated citation classification. J Am Med Inform Assoc 2006; 13 (2): 206-219 [FREE Full text] [doi: 10. 1197/jamia. M1929] [Medline: 16357352]
12. Créquit P, Trinquart L, Yavchitz A, Ravaud P. Wasted research when systematic reviews fail to provide a complete and up-to-date evidence synthesis: the example of lung cancer. BMC Med 2016 Jan 20; 14: 8 [FREE Full text] [doi: 10. 1186/s12916-016-0555-0] [Medline: 26792360]
13. Nama N, Menon K, Iliriani K, Pojsupap S, Sampson M, O'Hearn K, et al. A systematic review of pediatric clinical trials of high dose vitamin D. PeerJ 2016; 4: e1701 [FREE Full text] [doi: 10. 7717/peerj. 1701] [Medline: 26966655]
14. Brown AW, Allison DB. Using crowdsourcing to evaluate published scientific literature: methods and example. PLoS One 2014; 9 (7): e100647 [FREE Full text] [doi: 10. 1371/journal. pone. 0100647] [Medline: 24988466]
15. Ranard BL, Ha YP, Meisel ZF, Asch DA, Hill SS, Becker LB, et al. Crowdsourcing-harnessing the masses to advance health and medicine, a systematic review. J Gen Intern Med 2014 Jan; 29 (1): 187-203 [FREE Full text] [doi: 10. 1007/s11606-013-2536-8] [Medline: 23843021]
16. Lee YJ, Arida JA, Donovan HS. The application of crowdsourcing approaches to cancer research: a systematic review. Cancer Med 2017 Nov; 6 (11): 2595-2605 [FREE Full text] [doi: 10. 1002/cam4. 1165] [Medline: 28960834]
17. Rossiter DG, Liu J, Carlisle S, Zhu A. Can citizen science assist digital soil mapping? Geoderma 2015 Dec; 259-260: 71-80. [doi: 10. 1016/j. geoderma. 2015. 05. 006]
18. Lintott C, Schawinski K, Slosar A, Land K, Bamford S, Thomas D, et al. Galaxy Zoo: morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey. Mon Not R Astron Soc 2008 Sep 21; 389 (3): 1179-1189. [doi: 10. 1111/j. 1365-2966. 2008. 13689. x]
19. Good BM, Su AI. Games with a scientific purpose. Genome Biol 2011 Dec 28; 12 (12): 135. [doi: 10. 1186/gb-2011-12-12-135] [Medline: 22204700]
20. Vachelard J, Gambarra-Soares T, Augustini G, Riul P, Maracaja-Coutinho V. A guide to scientific crowdfunding. PLoS Biol 2016 Feb; 14 (2): e1002373 [FREE Full text] [doi: 10. 1371/journal. pbio. 1002373] [Medline: 26886064]
21. Candido DR, Lynn S, Ali HR, Eccles D, Hanby A, Provenzano E, et al. Crowdsourcing the general public for large scale molecular pathology studies in cancer. EBioMedicine 2015 Jul; 2 (7): 681-689 [FREE Full text] [doi: 10. 1016/j. ebiom. 2015. 05. 009] [Medline: 26288840]
22. Mortensen ML, Adam GP, Trikalinos TA, Kraska T, Wallace BC. An exploration of crowdsourcing citation screening for systematic reviews. Res Synth Methods 2017 Sep; 8 (3): 366-386 [FREE Full text] [doi: 10. 1002/jrsm. 1252] [Medline: 28677322]
23. Nama N, Iliriani K, Xia MY, Chen BP, Zhou LL, Pojsupap S, et al. A pilot validation study of crowdsourcing systematic reviews: update of a searchable database of pediatric clinical trials of high-dose vitamin D. Transl Pediatr 2017 Jan; 6 (1): 18-26 [FREE Full text] [doi: 10. 21037/tp. 2016. 12. 01] [Medline: 28164026]
24. Elliott J, Sim I, Thomas J, Owens N, Dooley G, Riis J, et al. #CochraneTech: technology and the future of systematic reviews. Cochrane Database Syst Rev 2014 (9): ED000091. [doi: 10. 1002/14651858. ED000091] [Medline: 25288182]
25. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig L, STARD Group. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. Br Med J 2015 Oct 28; 351: h5527 [FREE Full text] [doi: 10. 1136/bmj. h5527] [Medline: 26511519]
26. Kantor N, Wayne C, Nasr A. Symptom development in originally asymptomatic CPAM diagnosed prenatally: a systematic review. Pediatr Surg Int 2018 Jun; 34 (6): 613-620. [doi: 10. 1007/s00383-018-4264-y] [Medline: 29632964]
27. Khatib F, DiMaio F, Foldit Contenders Group, Foldit Void Crushers Group, Cooper S, Kazmierczyk M, et al. Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nat Struct Mol Biol 2011 Sep 18; 18 (10): 1175-1177 [FREE Full text] [doi: 10. 1038/nsmb. 2119] [Medline: 21926992]
28. Mitry D, Peto T, Hayat S, Blows P, Morgan J, Khaw K, et al. Crowdsourcing as a screening tool to detect clinical features of glaucomatous optic neuropathy from digital photography. PLoS One 2015; 10 (2): e0117401 [FREE Full text] [doi: 10. 1371/journal. pone. 0117401] [Medline: 25692287]
29. Brady CJ, Villanti AC, Pearson JL, Kirchner TR, Gupta OP, Shah CP. Rapid grading of fundus photographs for diabetic retinopathy using crowdsourcing. J Med Internet Res 2014; 16 (10): e233 [FREE Full text] [doi: 10. 2196/jmir. 3807] [Medline: 25356929]
30. Wang X, Mudie L, Brady CJ. Crowdsourcing: an overview and applications to ophthalmology. Curr Opin Ophthalmol 2016 May; 27 (3): 256-261 [FREE Full text] [doi: 10. 1097/ICU. 0000000000000251] [Medline: 26761188]
31. Tsertsvadze A, Chen Y, Moher D, Sutcliffe P, McCarthy N. How to conduct systematic reviews more expeditiously? Syst Rev 2015 Nov 12; 4: 160 [FREE Full text] [doi: 10. 1186/s13643-015-0147-7] [Medline: 26563648]
32. Featherstone R, McDonald S, Noel-Storr A, Thomas J. Advanced Information Retrieval on the Edge Living Systematic Reviews. 2018 Presented at: Health Technology Assessment international; June 1-5, 2018; Vancouver, Canada.
33. Ross J, Irani L, Silberman M, Zaldivar A, Tomlinson B. Who are the Crowdworkers? Shifting Demographics in Mechanical Turk. In: Extended Abstracts on Human Factors in Computing Systems. 2010 Apr 10 Presented at: CHI EA'10 CHI'10; April 10-15, 2010; Atlanta, GA p. 2863-2872. [doi: 10. 1145/1753846. 1753873]
34. Noel-Storr A, Thomas J, Mavergames C, Turner T, McDonald S, Green S, et al. Cochrane Crowd: using citizen science to meet the challenge of information overload in evidence production. 2016 Oct 23 Presented at: 24th Cochrane Colloquium; October 23-26, 2016; Seoul, Korea. [doi: 10. 13140/RG. 2. 2. 11290. 34240]
35. O'Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev 2015 Jan 14; 4: 5 [FREE Full text] [doi: 10. 1186/2046-4053-4-5] [Medline: 25588314]
36. Cohen AM. Optimizing feature representation for automated systematic review work prioritization. AMIA Annu Symp Proc 2008 Nov 6: 121-125 [FREE Full text] [Medline: 18998798]
37. Com G, Carroll JL, Tang X, Melguizo MS, Bower C, Jambhekar S. Characteristics and surgical and clinical outcomes of severely obese children with obstructive sleep apnea. J Clin Sleep Med 2015 Apr 15; 11 (4): 467-474 [FREE Full text] [doi: 10. 5664/jcsm. 4608] [Medline: 25665695]
38. Roche SL, Timberlake K, Manlhiot C, Balasingam M, Wilson J, George K, et al. Angiotensin-converting enzyme inhibitor initiation and dose uptitration in children with cardiovascular disease: a retrospective review of standard clinical practice and a prospective randomized clinical trial. J Am Heart Assoc 2016 Dec 20; 5 (5): e003230 [FREE Full text] [doi: 10. 1161/JAHA. 116. 003230] [Medline: 27207965]
39. Simperl E. How to use crowdsourcing effectively: guidelines and examples. LIBER Q 2015 Aug 18; 25 (1): 18. [doi: 10. 18352/lq. 9948]
40. Raykar V, Yu S. Eliminating spammers and ranking annotators for crowdsourced labeling tasks. J Mach Learn Res 2012 Feb; 13: 491-518 [FREE Full text] [doi: 10. 1162/15324430152748236]
41. Sarti AJ, Sutherland S, Landriault A, DesRosier K, Brien S, Cardinal P. Exploring the components of physician volunteer engagement: a qualitative investigation of a national Canadian simulation-based training programme. BMJ Open 2017 Dec 23; 7 (6): e014303 [FREE Full text] [doi: 10. 1136/bmjopen-2016-014303] [Medline: 28645956]
42. Kuenzig ME, Bishay K, Leigh R, Kaplan GG, Benchimol EI, Crowdscreen SR Review Team. Co-occurrence of asthma and the inflammatory bowel diseases: a systematic review and meta-analysis. Clin Transl Gastroenterol 2018 Sep 24; 9 (9): 188 [FREE Full text] [doi: 10. 1038/s41424-018-0054-z] [Medline: 30250122]
43. Wallace B, Noel-Storr A, Marshall I, Cohen A, Smalheiser N, Thomas J. Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach. J Am Med Inform Assoc 2017 Nov 1; 24 (6): 1165-1168 [FREE Full text] [doi: 10. 1093/jamia/ocx053] [Medline: 28541493]
44. Krivosheev E, Casati F, Baez M, Benatallah B. Combining crowd and machines for multi-predicate item screening. Proc ACM Hum Comput Interact 2018 Nov 1; 2 (CSCW): 1-18. [doi: 10. 1145/3274366]
45. Vaughan J. Making better use of the crowd: how crowdsourcing can advance machine learning research. J Mach Learn Res 2018 May; 18 (193): 46 [FREE Full text]
46. Bannach-Brown A, Przyby?a P, Thomas J, Rice A, Ananiadou S, Liao J, et al. Machine learning algorithms for systematic review: reducing workload in a preclinical review of animal studies and reducing human screening error. Syst Rev 2019 Jan 15; 8 (1): 23 [FREE Full text] [doi: 10. 1186/s13643-019-0942-7] [Medline: 30646959]