Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources

Research Synthesis Methods

Volumn 11, Issue 2, 2020, Pages 181-217

  Gusenbauer, Michael ^a   Haddaway, Neal R ^b,c  

^a JOHANNES KEPLER UNIVERSITY LINZ   (Austria)

^b STOCKHOLM ENVIRONMENT INSTITUTE   (Sweden)

^c UNIVERSITY OF JOHANNESBURG   (South Africa)

Author keywords

academic search systems; discovery; evaluation; information retrieval; systematic review; systematic search

Indexed keywords

ALGORITHM; BIBLIOGRAPHIC DATABASE; FACTUAL DATABASE; INFORMATION RETRIEVAL; INTERNET; MEDICAL RESEARCH; MEDLINE; META ANALYSIS (TOPIC); METHODOLOGY; PUBLICATION; REPRODUCIBILITY; SEARCH ENGINE; SOFTWARE;

ALGORITHMS; BIOMEDICAL RESEARCH; DATABASES, BIBLIOGRAPHIC; DATABASES, FACTUAL; INFORMATION STORAGE AND RETRIEVAL; INTERNET; META-ANALYSIS AS TOPIC; PERIODICALS AS TOPIC; PUBMED; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN; SEARCH ENGINE; SOFTWARE; SYSTEMATIC REVIEWS AS TOPIC;

EID: 85078807044     PISSN: None     EISSN: 17592887     Source Type: Journal    
DOI: 10.1002/jrsm.1378     Document Type: Article

References (95)

1. Price DJS. Little Science, Big Science. New York: Columbia Univ. Press; 1963 Columbia paperback.
2. Larsen PO, von Ins M. The rate of growth in scientific publication and the decline in coverage provided by Science Citation Index. Scientometrics. 2010;84(3):575-603. https://doi.org/10.1007/s11192-010-0202-z
3. Eden D. From the editors: replication, meta-analysis, scientific progress, and AMJ's publication policy. AMJ. 2002;45(5):841-846. https://doi.org/10.5465/AMJ.2002.7718946
4. Naisbitt J, Aburdene P. Megatrends 2000: Ten New Directions for the 1990's. 1st ed. New York: Morrow; 1990.
5. Cooper HM. Research Synthesis and Meta-analysis: A Step-by-Step Approach. Applied social research methods series. Fifth ed. 2 Los Angeles: SAGE; 2017.
6. Littell JH. Conceptual and Practical Classification of Research Reviews and Other Evidence Synthesis Products; 2018.
7. Kostoff RN, Shlesinger MF. CAB: citation-assisted background. Scientometrics. 2005;62(2):199-212. https://doi.org/10.1007/s11192-005-0014-8
8. Littell JH, Corcoran J, Pillai V. Systematic Reviews and Meta-Analysis: Oxford University Press, USA; 2008. https://books.google.at/books?id=UpsRDAAAQBAJ.
9. Rethlefsen ML, Farrell AM, Osterhaus Trzasko LC, Brigham TJ. Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. J Clin Epidemiol. 2015;68(6):617-626. https://doi.org/10.1016/j.jclinepi.2014.11.025
10. Bandara W, Furtmueller E, Gorba, Gorbacheva E, Miskon S, Beekhuyzen J. Achieving rigor in literature reviews: insights from qualitative data analysis and tool-support. Communications of the Association for Information Systems. 2015;37(8):154-204. http://aisel.aisnet.org/cais/vol37/iss1/8
11. Meert D, Torabi N, Costella J. Impact of librarians on reporting of the literature searching component of pediatric systematic reviews. J Med Libr Assoc. 2016;104(4):267-277. https://doi.org/10.3163/1536-5050.104.4.004
12. Koffel JB. Use of recommended search strategies in systematic reviews and the impact of librarian involvement: a cross-sectional survey of recent authors. Plos One. 2015;10(5):1-13. https://doi.org/10.1371/journal.pone.0125931
13. Livoreil B, Glanville J, Haddaway NR, et al. Systematic searching for environmental evidence using multiple tools and sources. Environ Evid. 2017;6:1-14. https://doi.org/10.1186/s13750-017-0099-6
14. Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions; 2011; Version 5.1.0.
15. Kugley S, Wade A, Thomas J, et al. Searching for studies: GUidelines on information retrieval for Campbell Systematic Reviews; 2016; 1.
16. Pullin A, Frampton G, Livoreil B, Petrokofsky G. Guidelines and Standards for Evidence Synthesis in Environmental Management. Version 5.0; 2018.
17. Hug SE, Braendle MP. The coverage of Microsoft Academic: analyzing the publication output of a university. Scientometrics. 2017;113(3):1551-1571. https://doi.org/10.1007/s11192-017-2535-3
18. Khabsa M, Giles CL. The number of scholarly documents on the public web. Plos One. 2014;9(5):1-6. https://doi.org/10.1371/journal.pone.0093949
19. Falagas ME, Pitsouni EI, Malietzis GA, Pappas G. Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses. FASEB journal. 2008;22(2):338-342. https://doi.org/10.1096/fj.07-9492LSF
20. Orduña-Malea E, Ayllón JM, Martín-Martín A, Delgado L-CE. Methods for estimating the size of Google Scholar. Scientometrics. 2015;104(3):931-949. https://doi.org/10.1007/s11192-015-1614-6
21. Harzing A-W. A longitudinal study of Google Scholar coverage between 2012 and 2013. Scientometrics. 2014;98(1):565-575. https://doi.org/10.1007/s11192-013-0975-y
22. Meier JJ, Conkling TW. Google Scholar's coverage of the engineering literature: an empirical study. The Journal of Academic Librarianship. 2008;34(3):196-201. https://doi.org/10.1016/j.acalib.2008.03.002
23. Turnbull D, Berryman J. Relevant search: with applications for Solr and Elasticsearch. Shelter Island New York: Manning Publications Co; 2016.
24. Levay P, Ainsworth N, Kettle R, Morgan A. Identifying evidence for public health guidance: a comparison of citation searching with Web of Science and Google Scholar. Res Synth Methods. 2016;7(1):34-45. https://doi.org/10.1002/jrsm.1158
25. Boeker M, Vach W, Motschall E. Google Scholar as replacement for systematic literature searches: good relative recall and precision are not enough. BMC Med Res Methodol. 2013;13:1-12. https://doi.org/10.1186/1471-2288-13-131
26. Hjørland B. Classical databases and knowledge organization: a case for Boolean retrieval and human decision-making during searches. J Assn Inf Sci Tec. 2015;66(8):1559-1575. https://doi.org/10.1002/asi.23250
27. Weber K. Search engine bias. In: Lewandowski D, ed. Handbuch Internet-Suchmaschinen 2. AKA Verlag Heidelberg; 2011:265-285.
28. Vaughan L, Thelwall M. Search engine coverage bias: Evidence and possible causes. Information Processing & Management. 2004;40(4):693-707. https://doi.org/10.1016/S0306-4573(03)00063-3
29. Carmines EG, Zeller RA. Reliability and Validity Assessment. Quantitative applications in the social sciences. Beverly Hills, London: Sage Publications; 1979 no.07–017.
30. Jacsó P. Google Scholar: the pros and the cons. Online Information Review. 2005;29(2):208-214. https://doi.org/10.1108/14684520510598066
31. Jacsó P. Google Scholar revisited. Online Information Review. 2008;32(1):102-114. https://doi.org/10.1108/14684520810866010
32. Bethel A, Rogers M. A checklist to assess database-hosting platforms for designing and running searches for systematic reviews. Health Info Libr J. 2014;31(1):43-53. https://doi.org/10.1111/hir.12054
33. Bramer WM, Giustini D, Kramer B, Anderson P. The comparative recall of Google Scholar versus PubMed in identical searches for biomedical systematic reviews: a review of searches used in systematic reviews. Syst Rev. 2013;2:1-9. https://doi.org/10.1186/2046-4053-2-115
34. Bramer WM, Giustini D, Kramer BMR. Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study. Syst Rev. 2016;5:1-9. https://doi.org/10.1186/s13643-016-0215-7
35. Mowshowitz A, Kawaguchi A. Measuring search engine bias. Information Processing & Management. 2005;41(5):1193-1205. https://doi.org/10.1016/j.ipm.2004.05.005
36. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):1-6. https://doi.org/10.1371/journal.pmed.1000097
37. Gusenbauer M. Google Scholar to overshadow them all? Comparing the sizes of 12 academic search engines and bibliographic databases. Scientometrics. 2019;118(1):177-214. https://doi.org/10.1007/s11192-018-2958-5
38. Ortega JL. Academic Search Engines: A quantitative outlook. Chandos information professional series. Oxford, UK: Chandos Publishing/Elsevier; 2014.
39. Schöpfel J, Farace DJ. Grey literature. In: Bates MJ, Maack MN, eds. Encyclopedia of library and information sciences. 3rd ed. / edited by Boca Raton, Fla: CRC London: Taylor & Francis; 2010:2029–2039.Marcia J. Bates and Mary Niles Maack
40. Sampson M, McGowan J. Inquisitio validus Index Medicus: a simple method of validating MEDLINE systematic review searches. Res Synth Methods. 2011;2(2):103-109. https://doi.org/10.1002/jrsm.40
41. Rogers M, Bethel A, Abbott R. Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies. Res Synth Methods. 2017;9(2):579-586. https://doi.org/10.1002/jrsm.1280
42. Rader T, Mann M, Stansfield C, Cooper C, Sampson M. Methods for documenting systematic review searches: a discussion of common issues. Res Synth Methods. 2014;5(2):98-115. https://doi.org/10.1002/jrsm.1097
43. O'Mara-Eves A, Brunton G, McDaid D, Kavanagh J, Oliver S, Thomas J. Techniques for identifying cross-disciplinary and ‘hard-to-detect’ evidence for systematic review. Res Synth Methods. 2014;5(1):50-59. https://doi.org/10.1002/jrsm.1094
44. Atkinson KM, Koenka AC, Sanchez CE, Moshontz H, Cooper H. Reporting standards for literature searches and report inclusion criteria: making research syntheses more transparent and easy to replicate. Res Synth Methods. 2015;6(1):87-95. https://doi.org/10.1002/jrsm.1127
45. Mahood Q, van Eerd D, Irvin E. Searching for grey literature for systematic reviews: challenges and benefits. Res Synth Methods. 2014;5(3):221-234. https://doi.org/10.1002/jrsm.1106
46. Bar-Ilan J. On the overlap, the precision and estimated recall of search engines. A case study of the query “Erdos”. Scientometrics. 1998;42(2):207-228. https://doi.org/10.1007/BF02458356
47. Kumar BTS, Prakash JN. Precision and relative recall of search engines: a comparative study of Google and Yahoo. Singapore Journal of Library and Information Management. 2009;38:124-137.
48. Shafi SM, Rather R. Precision and recall of five search engines for retrieval of scholarly information in the field of biotechnology. Webology. 2005;2(2):1-7.
49. Usmani TA, Pant D, Bhatt AK. A comparative study of Google and Bing search engines in context of precision and relative recall parameter. International Journal on Computer Science and Engineering (IJCSE). 2012;4(1):21-34.
50. Giustini D, Boulos MNK. Google Scholar is not enough to be used alone for systematic reviews. Online J Public Health Inform. 2013;5(2):1-10. https://doi.org/10.5210/ojphi.v5i2.4623
51. Bramer WM. Variation in number of hits for complex searches in Google Scholar. Journal of the Medical Library Association. 2016;104(2):143-145. https://doi.org/10.3163/1536-5050.104.2.009
52. Brophy J, Bawden D. Is Google enough? Comparison of an internet search engine with academic library resources. Aslib Proceedings. 2005;57(6):498-512. https://doi.org/10.1108/00012530510634235
53. Gehanno J-F, Rollin L, Darmoni S. Is the coverage of Google Scholar enough to be used alone for systematic reviews. BMC Medical Informatics and Decision Making. 2013;13(7):1-5.
54. Sturm B, Sunyaev A. If you want your research done right, do you have to do it all yourself? Developing design principles for systematic literature search systems. In: Maedche A, Vom Brocke J, Hevner A, eds. Designing the Digital Transformation; 2017:138–146.
55. Chu H, Rosenthal M. Search engines for the World Wide Web: a comparative study and evaluation methodology. J. Am. Soc. Inf. Sci. 1996;33:127-135.
56. Biolcati-Rinaldi F, Molteni F, Salini S. Assessing the reliability and validity of Google Scholar indicators. The case of social sciences in Italy. In: Bonaccorsi A, ed. The Evaluation of Research in Social Sciences and Humanities: Lessons from the Italian Experience. Cham: Springer International Publishing; 2018:295-319.
57. Haddaway NR, Collins AM, Coughlin D, Kirk S. The role of Google Scholar in evidence reviews and its applicability to grey literature searching. Plos One. 2015;10(9):1-17. https://doi.org/10.1371/journal.pone.0138237
58. Aune D, Giovannucci E, Boffetta P, et al. Fruit and vegetable intake and the risk of cardiovascular disease, total cancer and all-cause mortality—a systematic review and dose–response meta-analysis of prospective studies. International Journal of Epidemiology. 2017;46(3):1029-1056. https://doi.org/10.1093/ije/dyw319
59. Barnett DW, Barnett A, Nathan A, van Cauwenberg J, Cerin E. Built environmental correlates of older adults' total physical activity and walking: a systematic review and meta-analysis. International Journal of Behavioral Nutrition and Physical Activity. 2017;14(1):1-24. https://doi.org/10.1186/s12966-017-0558-z
60. Baur D, Gladstone BP, Burkert F, et al. Effect of antibiotic stewardship on the incidence of infection and colonisation with antibiotic-resistant bacteria and Clostridium difficile infection: a systematic review and meta-analysis. Lancet Infectious Diseases. 2017;17(9):990-1001. https://doi.org/10.1016/S1473-3099(17)30325-0
61. Bediou B, Adams DM, Mayer RE, Tipton E, Green CS, Bavelier D. Meta-analysis of action video game impact on perceptual, attentional, and cognitive skills. Psychological Bulletin. 2018;144(1):77-110. https://doi.org/10.1037/bul0000130
62. Bethel MA, Patel RA, Merrill P, et al. Cardiovascular outcomes with glucagon-like peptide-1 receptor agonists in patients with type 2 diabetes: a meta-analysis. Lancet Diabetes & Endocrinology. 2018;6(2):105-113. https://doi.org/10.1016/S2213-8587(17)30412-6
63. Bourne RRA, Flaxman SR, Braithwaite T, et al. Magnitude, temporal trends, and projections of the global prevalence of blindness and distance and near vision impairment: a systematic review and meta-analysis. Lancet Global Health. 2017;5(9):E888-E897. https://doi.org/10.1016/S2214-109X(17)30293-0
64. Brunoni AR, Chaimani A, Moffa AH, et al. Repetitive transcranial magnetic stimulation for the acute treatment of major depressive episodes a systematic review with network meta-analysis. Jama Psychiatry. 2017;74(2):143-152. https://doi.org/10.1001/jamapsychiatry.2016.3644
65. Carlbring P, Andersson G, Cuijpers P, Riper H, Hedman-Lagerlof E. Internet-based vs. face-to-face cognitive behavior therapy for psychiatric and somatic disorders: an updated systematic review and meta-analysis. Cognitive Behaviour Therapy. 2018;47(1):1-18. https://doi.org/10.1080/16506073.2017.1401115
66. Chu DK, Kim LH-Y, Young PJ, et al. Mortality and morbidity in acutely ill adults treated with liberal versus conservative oxygen therapy (IOTA): a systematic review and meta-analysis. Lancet. 2018;391(10131):1693-1705. https://doi.org/10.1016/S0140-6736(18)30479-3
67. Cipriani A, Furukawa TA, Salanti G, et al. Comparative efficacy and acceptability of 21 antidepressant drugs for the acute treatment of adults with major depressive disorder: a systematic review and network meta-analysis. Lancet. 2018;391(10128):1357-1366. https://doi.org/10.1016/S0140-6736(17)32802-7
68. Stacey D, Légaré F, Lewis K, et al. Decision aids for people facing health treatment or screening decisions. Cochrane Database Syst Rev. 2017;4:1-343. https://doi.org/10.1002/14651858.CD001431.pub5
69. Oxford Wordlist. Oxford University Press; 2008.
70. Fieschi M, Coiera E, Li YCJ. Medinfo: IOS Press; 2004. https://books.google.at/books?id=bS2xdt7iufgC.
71. Meho LI, Yang K. Impact of data sources on citation counts and rankings of LIS faculty: Web of Science versus Scopus and Google Scholar. J. Am. Soc. Inf. Sci. 2007;58(13):2105-2125. https://doi.org/10.1002/asi.20677
72. Martín-Martín A, Orduna-Malea E, Thelwall M, López-Cózar ED. Google Scholar, Web of Science, and Scopus: a systematic comparison of citations in 252 subject categories. Journal of Informetrics. 2018;12(4):1160-1177. https://doi.org/10.31235/osf.io/42nkm
73. Bakkalbasi N, Bauer K, Glover J, Wang L. Three options for citation tracking: Google Scholar, Scopus and Web of Science. Biomed Digit Libr. 2006;3(7):1-8. https://doi.org/10.1186/1742-5581-3-7
74. Haddaway NR, Macura B, Whaley P, Pullin AS. ROSES RepOrting standards for Systematic Evidence Syntheses: pro forma, flow-diagram and descriptive summary of the plan and conduct of environmental systematic reviews and systematic maps. Environ Evid. 2018;7(7):1-8. https://doi.org/10.1186/s13750-018-0121-7
75. White RW, Roth RA. Exploratory Search: Beyond the Query-Response Paradigm. Morgan & Claypool; 2009.
76. Jansen BJ, Spink A. In: Langendoerfer P, Droegehorn O, eds. IC'2003An Analysis of Web Documents Retrieved and Viewed; 2003:65-69.
77. Jansen BJ, Spink A. How are we searching the World Wide Web?: a comparison of nine search engine transaction logs. Information Processing & Management. 2006;42(1):248-263. https://doi.org/10.1016/j.ipm.2004.10.007
78. Athukorala K, Głowacka D, Jacucci G, Oulasvirta A, Vreeken J. Is exploratory search different?: a comparison of information search behavior for exploratory and lookup tasks. J Assn Inf Sci Tec. 2016;67(11):2635-2651. https://doi.org/10.1002/asi.23617
79. Hemminger BM, Lu D, Vaughan KTL, Adams SJ. Information seeking behavior of academic scientists. J. Am. Soc. Inf. Sci. 2007;58(14):2205-2225. https://doi.org/10.1002/asi.20686
80. Athukorala K, Hoggan E, Lehtiö A, Ruotsalo T, Jacucci G. Information-seeking behaviors of computer scientists: challenges for electronic literature search tools. Proc. Am. Soc. Info. Sci. Tech. 2013;50(1):1-11. https://doi.org/10.1002/meet.14505001041
81. Nicholas D, Boukacem-Zeghmouri C, Rodríguez-Bravo B, et al. Where and how early career researchers find scholarly information. Learned Publishing. 2017;30(1):19-29. https://doi.org/10.1002/leap.1087
82. Niu X, Hemminger BM. A study of factors that affect the information-seeking behavior of academic scientists. J. Am. Soc. Inf. Sci. 2012;63(2):336-353. https://doi.org/10.1002/asi.21669
83. Sapa R, Krakowska M, Janiak M. Information seeking behaviour of mathematicians: scientists and students. Information Research: An International Electronic Journal. 2014;19(4):1-11.
84. Fast KV, Campbell DG. “I still like Google”: University student perceptions of searching OPACs and the web. Proceedings of the American Society for Information Science and Technology. 2004;41(1):138-146. https://doi.org/10.1002/meet.1450410116
85. Kuiper E, Volman M, Terwel J. Students' use of Web literacy skills and strategies: searching, reading and evaluating Web information. Information Research. 2008;13(3):1-18.
86. Ingwersen P. Cognitive perspectives of information retrieval interaction: elements of a cognitive IR theory. Journal of Documentation. 1996;52(1):3-50. https://doi.org/10.1108/eb026960
87. Wellings S, Casselden B. An exploration into the information-seeking behaviours of engineers and scientists. Journal of Librarianship and Information Science. 2017;9(2):1-12. https://doi.org/10.1177/0961000617742466
88. Rowlands I, Nicholas D, Williams P, et al. The Google generation: the information behaviour of the researcher of the future. AP. 2008;60(4):290-310. https://doi.org/10.1108/00012530810887953
89. Kingsley K, Galbraith GM, Herring M, Stowers E, Stewart T, Kingsley KV. Why not just Google it? An assessment of information literacy skills in a biomedical science curriculum. BMC Med Educ. 2011;11(17):1-8. https://doi.org/10.1186/1472-6920-11-17
90. Kurbanoğlu S, Boustany J, Špiranec S, Grassian E, Mizrachi D, Roy L, eds. Information literacy: moving toward sustainability: Third European conference, ECIL 2015, Tallinn, Estonia, October 19–22, 2015: revised selected papers. Cham, Heidelberg, New York: Springer; 2015. Communications in computer and information science; 552.
91. Kurbanoğlu S, Boustany J, Špiranec S, et al. (Eds). Search Engine Literacy: Information Literacy in the. Workplace: Springer International Publishing; 2018.
92. Brindesi H, Monopoli M, Kapidakis S. Information seeking and searching habits of Greek physicists and astronomers: a case study of undergraduate students. Procedia - Social and Behavioral Sciences. 2013;73:785-793. https://doi.org/10.1016/j.sbspro.2013.02.119
93. Kirschner PA, de Bruyckere P. The myths of the digital native and the multitasker. Teaching and Teacher Education. 2017;67:135-142. https://doi.org/10.1016/j.tate.2017.06.001
94. Georgas H. Google vs. the library (part II): student search patterns and behaviors when using Google and a federated search tool. Portal: Libraries and the Academy. 2014;14(4):503-532.
95. Halevi G, Moed H, Bar-Ilan J. Suitability of Google Scholar as a source of scientific information and as a source of data for scientific evaluation: review of the literature. Journal of Informetrics. 2017;11(3):823-834. https://doi.org/10.1016/j.joi.2017.06.005