Comparative recall and precision of simple and expert searches in google scholar and eight other databases

Portal

Volumn 11, Issue 4, 2011, Pages 972-1006

  Walters, William H ^a  

^a Menlo College   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 80053923258     PISSN: 15312542     EISSN: 15307131     Source Type: Journal    
DOI: 10.1353/pla.2011.0042     Document Type: Article

References (76)

1. Roger C. Schonfeld and Ross Housewright, Faculty Survey 2009: Key Strategic Insights for Libraries, Publishers, and Societies (New York: Ithaka, 2010), http://www.ithaka.org/ithakas-r/research/faculty-surveys-2000-2009/Faculty%20Study%202009.pdf (accessed May 25, 2011).
2. Laura Bowering Mullen and Karen A. Hartman, "Google Scholar and the Library Web Site: The Early Response by ARL Libraries," College & Research Libraries 67, 2 (2006): 106-122.
3. Joan Giglierano, "Attitudes of OhioLINK Librarians Toward Google Scholar," Journal of Library Administration 47, 1/2 (2008): 101-113.
4. Karen A. Hartman and Laura Bowering Mullen, "Google Scholar and Academic Libraries: An Update," New Library World 109, 5/6 (2008): 211-222.
5. Chris Neuhaus, Ellen Neuhaus, and Alan Asher, "Google Scholar Goes to School: The Presence of Google Scholar on College and University Web Sites," Journal of Academic Librarianship 34, 1 (2008): 39-51.
6. Google, "About Google Scholar" (2011), http://scholar.google.com/intl/en/scholar/about.html (accessed May 25, 2011).
7. Mick O'Leary, "Google Scholar: What's in It for You?" Information Today 22, 7 (2005): 35-39.
8. Dean Giustini and Eugene Barsky, "A Look at Google Scholar, PubMed, and Scirus: Comparisons and Recommendations," Journal of the Canadian Health Libraries Association 26, 3 (2005): 85-89.
9. Chuck Hamaker and Brad Spry, "Key Issue: Google Scholar," Serials 18, 1 (2005): 70-72;
10. Péter Jacsó, "Pe ́ter's Picks & Pans: Cite Base Search, Institute of Physics Archive, and Google's Index to Scholarly Archive," Online 28, 5 (2004): 57-60.
11. Péter Jacsó, "Google Scholar: The Pros and the Cons," Online Information Review 29, 2 (2005): 208-214.
12. Péter Jacsó, "Visualizing Overlap and Rank Differences Among Web-Wide Search Engines: Some Free Tools and Services," Online Information Review 29, 5 (2005): 554-560.
13. Greg R. Notess, "Scholarly Web Searching: Google Scholar and Scirus," Online 29, 4 (2005): 39-41.
14. William H. Walters, "Google Scholar Coverage of a Multidisciplinary Field," Information Processing and Management 43, 4 (2007): 1121-1132.
15. William H. Walters, "Google Scholar Search Performance: Comparative Recall and Precision," portal: Libraries and the Academy 9, 1 (2009): 5-24.
16. Walters, "Google Scholar Search Performance."
17. Janice Adlington and Chris Benda, "Checking Under the Hood: Evaluating Google Scholar for Reference Use," Internet Reference Services Quarterly 10, 3/4 (2005): 135-148.
18. Rebecca Donlan and Rachel Cooke, "Running with the Devil: Accessing Library-Licensed Full-Text Holdings Through Google Scholar," Internet Reference Services Quarterly 10, 3/4 (2005): 149-157.
19. Péter Jacsó, "As We May Search: Comparison of Major Features of the Web of Science, Scopus, and Google Scholar Citation-Based and Citation-Enhanced Databases," Current Science 89, 9 (2005): 1537-1547.
20. Martin Myhill, "Google Scholar," The Charleston Advisor 6, 4 (2005): 49-52;
21. Marydee Ojala, "Scholarly Mistakes," Online 29, 3 (2005): 26.
22. Roy Tennant, "Google, the Naked Emperor," Library Journal 130, 13 (2005): 29.
23. Joann M. Wleklinski, "Studying Google Scholar: Wall to Wall Coverage?" Online 29, 3 (2005): 22-26;
24. Susan Gardner and Susanna Eng, "Gaga Over Google? Scholar in the Social Sciences," Library Hi Tech News 22, 8 (2005): 43.
25. Burton Callicott and Debbie Vaughn, "Google Scholar vs. Library Scholar: Testing the Performance of Schoogle," Internet Reference Services Quarterly 10, 3/4 (2005): 71-88.
26. Marilyn Christianson, "Ecology Articles in Google Scholar: Levels of Access to Articles in Core Journals," Issues in Science and Technology Librarianship 49 (2007), http://www.istl.org/07-winter/refereed.html (accessed May 25, 2011);
27. iguel A. García-Pérez, Accuracy and Completeness of Publication and Citation Records in the Web of Science, PsycINFO, and Google Scholar: A Case Study for the Computation of h Indices in Psychology," Journal of the American Society for Information Science and Technology 61, 10 (2010): 2070-2085.
28. Péter Jacsó, "Amazon, Google Book Search, and Google Scholar," Online 32, 2 (2008): 51-54.
29. Péter Jacsó, "Google Scholar Revisited," Online Information Review 32, 1 (2008): 102-114.
30. Péter Jacsó, "Metadata Mega Mess in Google Scholar," Online Information Review 34, 1 (2010): 175-191.
31. Margie Ruppel, "Google Scholar, Social Work Abstracts (EBSCO), and PsycINFO (EBSCO)," The Charleston Advisor 10, 3 (2009): 5-11.
32. Lydia Dixon et al, "Finding Articles and Journals via Google Scholar, Journal Portals, and Link Resolvers: Usability Study Results," Reference & User Services Quarterly 50, 2 (2010): 170-181.
33. Seikyung Jung et al, "LibraryFind: System Design and Usability Testing of Academic Metasearch System," Journal of the American Society for Information Science and Technology 59, 3 (2008): 375-389.
34. Chris Neuhaus et al, "The Depth and Breadth of Google Scholar: An Empirical Study," portal: Libraries and the Academy 6, 2 (2006): 127-141.
35. Christianson, "Ecology Articles in Google Scholar";
36. John J. Meier and Thomas W. Conkling, "Google Scholar's Coverage of the Engineering Literature: An Empirical Study," Journal of Academic Librarianship 34, 3 (2008): 196-201.
37. Philipp Mayr and Anne-Kathrin Walter, "An Exploratory Study of Google Scholar," Online Information Review 31, 6 (2007): 814-830. Reprinted as "Studying Journal Coverage in Google Scholar," Journal of Library Administration 47, 1/2 (2008): 81-99.
38. Walters, "Google Scholar Coverage."
39. Lyle Ford and Lisa Hanson O'Hara, "It's All Academic: Google Scholar, Scirus, and Windows Live Academic Search," Journal of Library Administration 46, 3/4 (2008): 43-52.
40. Michael Norris, Charles Oppenheim, Fytton Rowland, "Finding Open Access Articles Using Google, Google Scholar, OAIster and OpenDOAR," Online Information Review 32, 6 (2008): 709-715.
41. Jared L. Howland et al, "Google Scholar and the Continuing Education Literature," Journal of Continuing Higher Education 57, 1 (2009): 35-39.
42. Hannah Rozear, "Where Google Scholar Stands on Art: An Evaluation of Content Coverage in Online Databases," Art Libraries Journal 34, 2 (2009): 21-25.
43. Susanne Mikki, "Comparing Google Scholar and ISI Web of Science for Earth Sciences," Scientometrics 82, 2 (2010): 321-331.
44. Rebecca Griffiths, Michael Dawson, and Matthew Rascoff, Scholarly Communications in the History Discipline: A Report Commissioned by JSTOR (New York: Ithaka, 2006), http://www.ithaka.org/publications/History (accessed May 25, 2011).
45. Rena Helms-Park, Pavlina Radia, and Paul Stapleton, "A Preliminary Assessment of Google Scholar as a Source of EAP Students' Research Materials," Internet and Higher Education 10, 1 (2007): 65-76.
46. Jared L. Howland et al, "How Scholarly is Google Scholar? A Comparison to Library Databases," College & Research Libraries 70, 3 (2009): 227-234.
47. D. Yvonne Jones, "Biology Article Retrieval from Various Databases: Making Good Choices with Limited Resources," Issues in Science and Technology Librarianship 44 (2005), http://www.istl.org/05-fall/refereed.html (accessed May 25, 2011).
48. Mary L. Robinson and Judith Wusteman, "Putting Google Scholar to the Test: A Preliminary Study," Program: Electronic Library and Information Systems 41, 1 (2007): 71-80.
49. The study also included four non-scholarly searches for topics of general interest. In those cases, GS did not perform as well.
50. Glenn Haya, Else Nygren, and Wilhelm Widmark, "Metalib and Google Scholar: A User Study," Online Information Review 31, 3 (2007): 365-375.
51. Michael Levine-Clark and Joseph Kraus, "Finding Chemistry Information Using Google Scholar: A Comparison with Chemical Abstracts Service," Science & Technology Libraries 27, 4 (2007): 3-17.
52. Walters, "Google Scholar Search Performance."
53. Per O. Seglen, "The Skewness of Science," Journal of the American Society for Information Science 43, 9 (1992): 628-638.
54. William H. Walters, "Later-Life Migration in the United States: A Review of Recent Research," Journal of Planning Literature 17, 1 (2002): 37-66.
55. William H. Walters, "Types and Determinants of Later-Life Migration" (PhD diss., Brown University, 2002).
56. The 155 relevant documents include all the works cited in the bibliography except for those published before 1990 (forty items); those published after 2000 (one item); those published as books, book chapters, or dissertations (26 additional items); and those that are primarily bibliographic or editorial in nature (ten items).
57. Although the nine simple searches are very similar to those presented in an earlier investigation (Walters, "Google Scholar Search Performance"), this study evaluates a different set of databases. Specifically, it examines PubMed rather than MEDLINE, Sociological Abstracts rather than SocINDEX, and Social Sciences Full Text rather than Social Sciences Abstracts. AgeLine, ArticleFirst, and EBSCO Academic Search were included in the earlier study but are not examined here. Although the hit counts reported here for the simple searches in GS, EconLit, and POPLINE are slightly different from those reported in "Google Scholar Search Performance," the search phrase and the interface were the same in both studies. Any discrepancies are presumably due to changes in the databases that occurred between the two analyses-between February 2008 and June 2010.
58. Walters, "Google Scholar Coverage";
59. William H. Walters and Esther I. Wilder, "Bibliographic Index Coverage of a Multidisciplinary Field," Journal of the American Society for Information Science and Technology 54, 14 (2003): 1305-1312;
60. William H. Walters and Esther I. Wilder, "Disciplinary Perspectives on Later-Life Migration in the Core Journals of Social Gerontology," The Gerontologist 43, 5 (2003): 758-760.
61. APLIC, "Membership List" (2009), http://www.aplici.org/membership/membershiplist/(accessed May 25, 2011);
62. National Archive of Computerized Data on Aging, "NIA Funded Centers: Centers on the Demography of Aging" (2011), http://www.icpsr.umich.edu/icpsrweb/NACDA/niacenters/ (accessed May 25, 2011);
63. U.S. News & World Report, "Sociology Specialty Rankings: Sociology of Population" (2011), http://grad-schools.usnews.rankingsandreviews.com/best-graduate-schools/top-sociologyschools/sociologyofpopulation (accessed May 25, 2011).
64. As Table 3 suggests, this study assess the performance of entire information systems, each of which has several components: people, data, software, hardware, and network resources. While an investigation of database effectiveness might use a single search phrase across several databases in order to isolate the impact of database choice on search effectiveness, the goal of this study is to evaluate information systems rather than databases. In this context, the variation in search strategies is not a confounding factor, but a reflection of the fact that each information system has different capabilities and characteristics.
65. As noted earlier, each of the 155 relevant records can be found in the bibliography of a published literature review: Walters, "Later-Life Migration." Because some articles are indexed in GS solely due to their inclusion in the bibliographies of previously indexed papers, there is a possibility that the earlier paper (which was indexed in GS) might have itself led to an improvement in Google Scholar's coverage of the 155 relevant articles. Further investigation reveals that any such bias is likely to be minimal, however. GS includes a special notation in records that were generated from the bibliographies of previously indexed papers, and only five relevant records from the four GS searches carry that notation. Each of the five was cited in at least seven GS-indexed articles, and four of the five were cited in at least four GS-indexed articles that appeared prior to "Later-Life Migration." It is therefore unlikely that any of them were included in the database due to their appearance in that paper.
66. Several rules were established before the assessment process began. Records for working papers that were later published as relevant articles were not counted as relevant, since those records are missing publication information that may influence users' assessments of relevance. Likewise, records for collections of articles (such as special issues of journals) that mention a relevant article were not counted as relevant. If a relevant article appeared more than once in the results for a particular search, only the first occurrence-the lowestnumbered hit-was counted.
67. Most users of internet search engines examine just a single page of results, although users of scholarly databases are likely to be somewhat more thorough. See, for example,
68. Bernard J. Jansen and Amanda Spink, "How Are We Searching the World Wide Web? A Comparison of Nine Search Engine Transaction Logs," Information Processing and Management 42, 1 (2006): 248-263.
69. Walters, "Google Scholar Search Performance."
70. Walters, "Google Scholar Search Performance," 10.
71. Jacsó, "Google Scholar Revisited";
72. Jacsó, "Metadata Mega Mess";
73. Ruppel, "Google Scholar, Social Work Abstracts (EBSCO), and PsycINFO (EBSCO)."
74. As Table 8 reveals, expert searches in the Group 3 databases are also characterized by relatively long search phrases; relatively infrequent use of "journal articles only" or "peerreviewed only" limiters; and relatively frequent use of geographical area descriptors, title-field searching, and exact phrase searching. These differences do not help explain the relative effectiveness of simple searches in the Group 3 databases, however.
75. Christianson, "Ecology Articles in Google Scholar"; Jacsó, "Metadata Mega Mess."
76. Nicolas Vibert et al, "Effects of Domain Knowledge on Reference Search with the PubMed Database: An Experimental Study," Journal of the American Society for Information Science and Technology 60, 7 (2009): 1423-1447.