Research evaluation and citation analysis: Key issues and implications

The Electronic Library

Volumn 27, Issue 3, 2009, Pages 361-375

  Herther, Nancy K ^a  

^a UNIVERSITY OF MINNESOTA   (United States)

Author keywords

Higher education; Information management; Research; Statistics

Indexed keywords

EID: 67650179073     PISSN: 02640473     EISSN: None     Source Type: Journal    
DOI: 10.1108/02640470910966835     Document Type: Article

References (92)

1. Abbasi, K. (2004), “Let's dump impact factors”, British Medical Journal, Vol. 329, p. 867.
2. Adam, D. (2002), “The counting house”, Nature, Vol. 415, pp. 726-729, p. 727.
3. Allen consulting group (2005), Measuring the impact of publicly funded research: Report to the Australian Government Department of Education, Science and Training, available at: www.dest.gov.au/NR/rdonlyres/6FB9F35B-FADD-44C2-A15F-B145FF9416F6/1392/ report.pdf (accessed January 18, 2008).
4. Bensman, S.J. (2007), “Garfield and the impact factor”, Annual Review of Information Science and Technology, Vol. 41, pp. 93-155, p. 150.
5. Bollag, B. (2006), “Group endorses principles for ranking universities”, Chronicle of Higher Education, Vol. 52, p. A40, The full-text of the report is available online at: www.che.de/downloads/Berlin_Principles_IREG_534.pdf (accessed January 18, 2008).
6. Borgman, C. and Furner, J. (2002), “Scholarly communication and bibliometrics”, Annual Review of Information Science and Technology;, Vol. 36, pp. 3-72.
7. Cheek, J., Garnham, B. and Quan, J. (2006), “What's in a number? Issues in providing evidence of impact and quality of research(ers)”, Qualitative Health Research, Vol. 16 No. 3, pp. 23-435.
8. Cole, J.R. and Cole, S. (1973), Social Stratification in Science, University of Chicago Press, Chicago, IL, p. 6.
9. Colquhoun, D. (2003), “Challenging the tyranny of impact factors”, Nature, Vol. 423, p. 479.
10. Commission on the Future of Higher Education (2006), A Test of Leadership: Charting the Future of US Higher Education, Commission on the Future of Higher Education, Washington, DC, available at: www.ed.gov/about/bdscomm/list/hiedfuture/reports/final-report.pdf (accessed January 18, 2008).
11. Farrell, E.F. (2006), “The power and peril of admissions data”, Chronicle of Higher Education, Vol. 53 No. 8, p. A46.
12. Field, K. (2006), “New database would customize college rankings”, Chronicle of Higher Education, Vol. 52 No. 31, p. 34A.
13. Fogg, P. (2005), “A chronicle survey: what presidents think: presidents favor scrapping tenure”, Chronicle of Higher Education, Vol. 52 No. 11, p. A31.
14. Fogg, P. (2006), “All Eyes on Tenure: Amid public scrutiny, the U. of Colorado reviews its process for awarding the coveted status”, Chronicle of Higher Education, Vol. 52 No. 18, p. A22.
15. Garfield, E. (1955), “Citation indexes for science: a new dimension in documentation through association of ideas”, Science, Vol. 122 No. 3159, p. 108.
16. Garfield, E. (1998), “The impact factor and using it correctly”, Der Unfallchirurg, Vol. 48 No. 2, p. 413, available at: www.garfield.library.upenn.edu/papers/derunfallchirurg_v101(6) p413y1998english.html.
17. Garvey, W.D. (1979), Communication: The Essence of Science, Pergamon Press, Oxford.
18. Gelman, S.R. and Gibelman, M. (1999), “A quest for citations? An analysis of and commentary on the trend toward multiple authorship”, Journal of Social Work Education, Vol. 35 No. 2, pp. 203-13.
19. Goldin, R.F. (2006), “College rankings: F”, Chronicle of Higher Education, Vol. 53 No. 11, p. B24.
20. Gravos, J. (2006), “Tracking the invisible faculty: a new index confirms that, at thousands of colleges, most professors are off the tenure track”, Chronicle of Higher Education, Vol. 53 No. 17, p. A8.
21. Hargens, L.L. and Schuman, H. (1990), “Citation counts and social comparisons: scientists' use and evaluation of citation index data”, Social Science Research, Vol. 19 No. 3, pp. 205-12.
22. Hirsch, J.E. (2005), “An index to quantify and individual's scientific research”, Output. arXiv:physics/0508025v5. available at: www.pnas.org/cgi/content/short/102/46/16569 (accessed January 28, 2008).
23. Horan, J.J. and Erickson, C.D. (1991), “Fellowship behavior in division 17 and the MOMM cartel”, The Counseling Psychologist, Vol. 19 No. 2, pp. 253-9.
24. Kostoff, R.N. (2002), “Citation analysis of research performer quality”, Scientometics, Vol. 53, pp. 49-71.
25. McCready, K. (2006), “A multi-dimensional framework for academic support: data and response”, paper presented JSTOR Participating Publisher Meeting, May 16, available at: www.lib.umn.edu/about/mellon/docs.phtml (accessed January 18, 2008).
26. Meadows, A.J. (1974), Communication in Science, Butterworths, London, p. ix.
27. Merton, R.K. (1979), “Forward”, in Garfield, E. (Ed.), Citation Indexing: Its Theory and Application in Science, Technology and Humanities, Wiley, New York, NY, p. vi.
28. Merton, R.K. (1995), “The Thomas theorem and the Matthew effect”, Social Forces, Vol. 74 No. 2, pp. 379-424, p. 381.
29. Modern Language Association of America (2006), MLA Task Force on Evaluating Scholarship for Tenure and Promotion, MLA, New York, NY, available at: www.mla.org/pdf/task_force_tenure_promotio.pdf (accessed January 18, 2008).
30. Pfirman, S.L., Collins, J.P., Lowes, S. and Michaels, A.F. (2005), “Collaborative efforts: promoting interdisciplinary scholars”, Chronicle of Higher Education, Vol. 51 No. 23, p. B15.
31. Schuster, J.H. and Finkelstein, M.J. (2006), The American Faculty: The Restructuring of Academic Work and Careers, Johns Hopkins University Press, Baltimore, MD.
32. Smallwood, S. (2005), “Berkeley grants tenure to critic”, Chronicle of Higher Education, Vol. 51 No. 39, p. A8.
33. Thomson Scientific (2008a), “Essential science indicators”, http://scientific.thomson.com/products/esi/ (accessed January 18, 2008).
34. Thomson Scientific (2008b), Journal Citation Reports, http://scientific.thomson.com/products/jcr/ (accessed January 18, 2008).
35. Wallin, J.A. (2005), “Bibliometric methods: pitfalls and possibilities”, Basic & Clinical Pharmacology & Toxicology, Vol. 97, pp. 261-75.
36. Bataille, G.M. and Brown, B.E. (2006), Faculty Career Paths: Multiple Routes to Academic Success and Satisfaction, Praeger Publishers, Westport, CT.
37. Cameron, B.D. (2005), “Trends in the usage of ISI bibliometric data: uses, abuses and Implications”, Portal: Libraries and the Academy, Vol. 5 No. 1, pp. 105-25.
38. Choo, C.W., Detlor, B. and Turnbull, D. (2000), “Information Seeking on the Web: An Integrated Model of Browsing and Searching”, First Monday, Vol. 5 No. 2, available at: http://firstmonday.org/issues/issues5_2/choo/index.html.
39. Corby, K. (2001), “Method or Madness? Educational Research and Citation Prestige”, Portal: Libraries and the Academy, Vol. 1 No. 3, pp. 279-88.
40. Cronin, B., Snyder, H.W., Rosenbaum, H., Martinson, A. and Callahan, E. (1998), “Invoked on the Web”, Journal of the American Society for Information Science, Vol. 49 No. 14, pp. 1319-28.
41. Ding, Y. (1998), “Scholarly communication and bibliometrics: Part II. The scholarly communication process – Literature review”, International Forum on Information and Documentation, Vol. 23 No. 3, pp. 3-19.
42. Dracos, A. and Cognetti, G. (1995), “Scientific literature: Bibliometric and bibliographic indicators as integrative criteria for an objective evaluation of research activity”, Annali dell'Instituto Superiore di Sanita, Vol. 31, pp. 381-90.
43. Drake, M.A. (2007), “Scholarly communication in turmoil”, Information Today, Vol. 24 No. 2, pp. 1,19.
44. Duque, R., Ynalvez, M., Sooryamoorthy, R., Mbatia, P., Dzorgbo, D. and Shrum, W. (2005), “Collaboration paradox: scientific productivity, the internet and problems of research in developing areas”, Social Studies of Science, Vol. 35 No. 5, pp. 755-85.
45. Egghe, L. (2000), “New informetric aspects of the internet: some reflections – many problems”, Journal of Information Science, Vol. 26 No. 5, pp. 329-35.
46. Fellows, L. and Heberlein, A.S. et al. (2005), “Method matters: an empirical study of impact in cognitive neuroscience”, Journal of Cognitive Neuroscience, Vol. 17 No. 6, pp. 850-8.
47. Fine, M.A. and Kurdek, L.A. (1993), “Reflections on determining authorship credit and authorship order on faculty-student collaborations”, American Psychologist, Vol. 48, pp. 1141-7.
48. Garfield, E. (1996), “How can impact factors be improved?”, British Medical Journal, Vol. 313, pp. 411-3.
49. Garfield, E. (1998), “Random thoughts on citationology its theory and practice”, Scientometrics, Vol. 43, I, pp. 69-76, 74.
50. Griffth, B.C. (1980), Key Papers in Information Science, Knowledge Industry, White Plains, NY, p. 19.
51. Griffith, B.C. and Mullins, N. (1972), “Coherent social groups in scientific change”, Science, Vol. 177 No. 4053, pp. 959-64.
52. Griffith, B.C., Servi, P., Anker, A. and Drott, M.C. (1979), “The aging of scientific literature: a citation analysis”, Journal of Documentation, Vol. 35, pp. 179-96.
53. Griffith, B.C., Small, H.G., Stonehill, J.A. and Dey, S. (1974), “The structure of scientific literatures. II: toward a macro and microstructure for science”, Science Studies, Vol. 4, pp. 339-65.
54. Haggbloom, S.J. et al. (2002), “The 100 most eminent psychologists of the 20th century”, Review of General Psychology, Vol. 6, pp. 139-52.
55. Harter, S.P. (1996), “The impact of electronic journals on scholarly communication: a citation analysis. public-access computer systems review 7”, available at: http://info.lib.uh.edu/pr/v7/n5/hart7n5.html.
56. Harter, S.P. and Kim, H.J. (1996), “Accessing electronic journals and other e-publications”, College & Research Libraries, Vol. 57, pp. 440-56.
57. Holden, G., Rosenberg, G., Barker, K. and Onghena, P. (2006), “An assessment of the predictive validity of impact factor scores: implications for academic employment decisions in social work”, Research on Social Work Practice, Vol. 16 No. 6, pp. 613-24.
58. Hughes, C.A. (2006), “The case for scholars' management of author rights”, Portal: Libraries and the Academy, Vol. 6 No. 2, available online at: http://www.ucop.edu/lauc/opinions/author_rights.html (accessed January 18, 2008).
59. Keen, B., Horan, J.J., Hanish, C., Copperstone, J. and Tribbensee, N. (1998), “Publication frequency, citation frequency, and quality of counseling psychology research”, paper presented at the annual meeting of the American Psychological Association, San Francisco, CA, August, available at: http://horan.asu.edu/d-bk-apa.htm.
60. Kroc, R.J. (1984), “Using citation analysis to assess scholarly productivity”, Educational Researcher, Vol. 13 No. 6, pp. 17-22.
61. Laband, D.N. and Piette, M.J. (1994), “A citation analysis of the impact of blinded peer review”, Journal of the American Medical Association, Vol. 272, pp. 147-9.
62. Larson, R.R. (1996), “Bibliometrics of the world wide web: an exploratory analysis of the intellectual structure of cyberspace”, paper presented at ASIS 1996, available: http://sherlock.berkeley.edu/asis96/asis96.html (accessed December 15, 2007).
63. Lee, S. and Bozeman, B. (2005), “The impact of research collaboration on scientific productivity”, Social Studies of Science, Vol. 35 No. 5, pp. 673-702.
64. Lichtman, M.A. and Oakes, D. (2001), “The productivity and impact of the Leukemia & Lymphoma Society Scholar Program: the apparent positive effect of peer review”, Blood, Cells, Molecules and Diseases, Vol. 27, pp. 1020-7.
65. Lu, Z. (1999), “Scalable distributed architectures for information retrieval”, PhD thesis, University of Massachusetts, Amherst, MA.
66. McCain, K.W. (2000), “Sharing digitized research-related information on the world wide web”, Journal of the American Society for Information Science, Vol. 51 No. 14, pp. 1321-7.
67. Martin, B., Hazeri, A. and Sarrafzadeh, M. (2006), “Knowledge management and the LIS professions: investigating the implications for practice and for educational provision”, Australian Library Journal, Vol. 55 No. 1, pp. 12-29.
68. Mela, G.S., Martinoli, C., Poggi, E. and Derchi, L.E. (2003), “Radiological research in Europe: a bibliometric study”, European Radiology, Vol. 13 No. 4, pp. 657-62.
69. Merton, R.K. (1973), The Sociology of Science, University of Chicago Press, Chicago, IL, pp. vii-viii.
70. Metcalfe, A.S. (2006), Knowledge Management and Higher Education: A Critical Analysis, Information Science Publishing, Hershey, PA.
71. Moed, H.F. (2005), Citation Analysis in Research Evaluation, Springer, Dordrecht.
72. Montori, V.M., Wilczynski, N.L., Morgan, D. and Haynes, R.B. (2003), “Systematic reviews: a cross-sectional study of location and citation counts”, BMC Medicine, Vol. 1 No. 2.
73. Najman, J.M. and Hewitt, B. (2003), “The validity of publication and citation counts for sociology and other selected disciplines”, Journal of Sociology, Vol. 39 No. 1, pp. 62-80.
74. Ober, J., Candee, C. and French, B. (2004), “Reshaping scholarly communication”, Against the Grain, Vol. 16 No. 3, available at: http://osc.universityofcalifornia.edu/responses/materials/against_the_grain_june2004.pdf (accessed December 15, 2007).
75. Perry, B. (2006), “Science, society and the university: a paradox of values”, Social Epistemology, Vol. 20 Nos 3/4, pp. 201-19.
76. Rousseau, R. (1997), “Sitations: an exploratory study”, Cybermetrics, Vol. 1 No. 1, p. 1, available at: www.cindoc.csic.es/cybermetrics/articles/v1i1p1.html.
77. Seglen, P. (1997), “Why the impact factor of journals should not be used for evaluating research”, BMJ, Vol. 314, p. 497.
78. Shaw, V. (2005), “Life course of academic professionals: substantive tasks, false assumptions, institutional accommodations, and personal adjustments”, Advances in Life Course Research, Vol. 9, pp. 331-47.
79. Shore, C. and Wright, S. (2004), “Whose Accountability? Governmentality and the Auditing of Universities”, Parallax, Vol. 10 No. 2, pp. 110-6.
80. Small, H. (1973), “Co-citation in the scientific literature: a new measure of the relationship between two documents”, Journal of the American Society of Information Science, Vol. 24, pp. 265-9.
81. Small, H. and Griffith, B. (1974), “The structure of scientific literature: identifying and graphing specialities”, Science Studies, Vol. 4, pp. 17-40.
82. Sonnenwald, D.H. (2007), “Scientific collaboration”, Annual Review Of Information Science and Technology, Vol. 41, pp. 643-81.
83. Swan, A. (2007), “Open access and the progress of science”, American Scientist, Vol. 95 No. 3, pp. 197-9.
84. Taubes, G. (1993), “Measure for measure in science”, Science, Vol. 260, pp. 884-6.
85. Walsh, J.P. and Maloney, N.G. (2003), “Problems in scientific collaboration: does email help or hinder?”, unpublished paper, University of Illinois, Chicago, IL, available at: http://tigger.uic.edu/∼jwalsh/WalshMaloneyAAAS.pdf.
86. West, R. and McIlwaine, A. (2002), “What do citation counts count for in the field of addiction? An empirical evaluation of citation counts and their link with peer ratings of quality”, Addiction, Vol. 97, pp. 501-4.
87. White, H.D. and Griffith, B.C. (1981), “Author cocitation: a literature measure of intellectual structure”, Journal of the American Society for Information Science, Vol. 32, pp. 163-71.
88. White, H.D. and McCain, K.W. (1997), “Visualization of literatures”, In Annual Review of Information Science and Technology (ARIST), Vol. 32, pp. 99-168.
89. Wilson, C.S. (2001), “Informetrics”, In Annual Review of Information Science and Technology, Vol. 34, pp. 3-143.
90. Wilson, P. (1996), “The future of research in our field”, in Olaisen, J., Munch-Petersen, E. and Wilson, P. (Eds), Information science: From the development of the discipline to social interaction, Scandinavian University Press, Oslo, pp. 319-23.
91. Youngen, G.K. (1998), “Citation patterns to traditional and electronic preprints in the published literature”, College & Research Libraries, Vol. 59 No. 5, pp. 448-56.
92. Zhao, D. and Logan, E. (2002), “Citation analysis of scientific publications on the web: a case study in XML research area”, Scientometrics, Vol. 54, pp. 449-72.