An assessment of the predictive validity of impact factor scores: Implications for academic employment decisions in social work

Research on Social Work Practice

Volumn 16, Issue 6, 2006, Pages 613-624

  Holden, Gary ^a   Rosenberg, Gary ^b   Barker, Kathleen ^c   Onghena, Patrick ^d  

^a NEW YORK UNIVERSITY   (United States)

^b MOUNT SINAI SCHOOL OF MEDICINE   (United States)

^c CITY UNIVERSITY OF NEW YORK   (United States)

^d UNIVERSITY OF LEUVEN   (Belgium)

Author keywords

Career development; Citation analysis; Decision making; Educational personnel; Faculty; Higher education; Hiring; Impact factor scores; Occupational tenure; Personnel promotion; Personnel selection; Reappointment; Scientific communication; Teacher tenure; Validity

Indexed keywords

EID: 33749845887     PISSN: 10497315     EISSN: 15527581     Source Type: Journal    
DOI: 10.1177/1049731506292570     Document Type: Article

References (102)

1. Abbasi, K. (2004). Let's dump impact factors. British Medical Journal, 329-329. Retrieved October 16, 2004, from http://bmj.bmjjournals.com/cgi/ content/full/329/7471/0-h
2. Adam, D. (2002). Citation analysis: The counting house. Nature, 415, 726-729.
3. Aronowitz, S. (2001). The knowledge factory: Dismantling the corporate university and creating true higher learning. Boston: Beacon.
4. Baker, D. R. (1990). Citation analysis: A methodological review. Social Work Research & Abstracts, 26, 3-10.
5. Baker, D. R. (1991). On-line bibliometric analysis for researchers and educators. Journal of Social Work Education, 27, 41-47.
6. Balaram, P. (2000). The impact of publication lists. Current Science, 78, 1177-1178.
7. Barker, K. (1995). Contingent work: Research issues and the lens of moral exclusion. In L. Tetrick & J. Barling (Eds.), Changing employment relations: Behavioral and social perspectives (pp. 31-60). Washington, DC: American Psychological Association.
8. Barker, K. (1998). Toiling for piece-rates and accumulating deficits: Contingent work in higher education. In K. Barker & K. Christensen (Eds.), Contingent work: American employment relations in transition (pp. 195-220). Cornell, NY: Cornell University Press.
9. Barker, K. (2001). Standard and nonstandard employment in higher education: Findings from the 1990's. In Commission on Professionals in Science and Technology (Ed.), Changing career paths in science and engineering: Report of a CPST workshop (pp. 12-30). Washington, DC: AAAS, Commission on Professionals in Science and Technology. Retrieved April 15, 2006, from http://www.cpst.org/Changing.pdf
10. Barker, K. (2003). Contingent work in the United States. In Supplement to the 19th Edition of the Encyclopedia of Social Work (pp. 9-19). Washington, DC: National Association of Social Workers.
11. Barker, K., & Christensen, K. (1998a). Introduction: Controversy and challenges raised by contingent work arrangements. In K. Barker & K. Christensen (Eds.), Contingent work: American employment relations in transition (pp. 1-18). Cornell, NY: Cornell University Press.
12. Barker, K., & Christensen, K. (1998b). Charting future research. In K. Barker & K. Christensen (Eds.), Contingent work: American employment relations in transition (pp. 306-320). Cornell, NY: Cornell University Press.
13. Barker, K., & Kohout, J. (2003). Contemporary employment in psychology and future trends. In M. J. Prinstein & M. Patterson (Eds.), The portable mentor: Expert guide to a successful career in psychology (pp. 309-332). New York: Kluwer Academic/Plenum.
14. Beck, U. (2000). The brave new world of work: The political economy of insecurity. Cambridge, UK: Cambridge University Press.
15. Borgman, C. L., & Furner, J. (2002). Scholarly communication and bibliometrics. Annual Review of Information Science and Technology, 36, 3-72.
16. Braxton, J. M., & Del Favero, M. (2002). Evaluating scholarship performance: Traditional and emergent assessment templates. New Directions for Institutional Research, 114, 19-31.
17. Burgan, M. (1998, March/April). What we talk about when we talk about tenure. Academe, 84 (2). 2-2.
18. Campling, J. (1992). Social-policy digest 81. Journal of Social Policy, 21, 99-118.
19. Campling, J. (1993). Social-policy digest 85. Journal of Social Policy, 22, 99-118.
20. Christenson, J. A., & Sigelman, L. (1985). Accrediting knowledge: Journal stature and citation impact in social science. Social Science Quarterly, 66, 964-975.
21. Cnaan, R. A., Caputo, R. K., & Shmuely, Y. (1994). Senior faculty perceptions of social work journals. Journal of Social Work Education, 30, 185-199.
22. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
23. Cole, J., & Cole, S. (1971). Measuring the quality of sociological research: Problems in the use of the Science Citation Index. American Sociologist, 6, 23-29.
24. Cole, J. R. (2000). A short history of the use of citations as a measure of the impact of scientific and scholarly work. In B. Cronin & H. B. Atkins (Eds.), The web of knowledge: A festschrift in honor of Eugene Garfield (pp. 281-300). Medford, NY: Information Today.
25. Colquhoun, D. (2003). Challenging the tyranny of impact factors. Nature, 423, 479-479.
26. Dahwan, S. M., & Gupta, B. M. (2004). Comparative evaluation of Indian physics research: Impact factor vs. citations frequency. Current Science, 86, 1194-1195.
27. Decker, O., Beutel, M. E., & Brahler, E. (2004). Deep impact - Evaluation in the sciences. Sozial Und Praventivmedizin, 49, 10-14.
28. Fassoulaki,A., Sarantopoulos, C., Papilas, K., Patris, K., & Melemeni,A. (2001). Academic anesthesiologists' views on the importance of the impact factor of scientific journals: A North American and European survey. Canadian Journal of Anesthesia, 48, 953-957.
29. Fava, G. A., & Ottolini, F. (2000). Impact factors versus actual citations. Psychotherapy and Psychosomatics, 69, 285-286.
30. Finkelstein, M. J., Seal, R. K., & Schuster, J. H. (1998). The new academic generation: A profession in transformation. Baltimore, MD: Johns Hopkins University Press.
31. Frank, M. (2003). Impact factors: Arbiter of excellence. Journal of the American Medical Library Association, 91, 4-6.
32. Furr, A. L. (1995). The relative influence of social work journals: Impact factors vs. core influence. Journal of Social Work Education, 31, 38-45.
33. Garfield, E. (1983). How to use citation analysis for faculty evaluations, and when is it relevant? Part 2. Current Comments, 45, 5-14.
34. Garfield, E. (1996). How can impact factors be improved? British Medical Journal, 313, 411-413.
35. Garfield, E. (1997). Validation of citation analysis. Journal of the American Society for Information Science, 48, 962-964.
36. Garfield, E. (1999). Journal impact factor: A brief review. Canadian Medical Association Journal, 161, 979-980.
37. Garfield, E. (2003). The meaning of the impact factor. International Journal of Clinical and Health Psychology, 3, 363-369.
38. Gastel, B. (2001). Assessing the impact of investigators' work: Beyond "impact" factors. Canadian Journal of Anesthesia, 48, 941-945.
39. Glanzel, W., & Moed, H. F. (2002). Journal impact measures in bibliometric research. Scientometrics, 53, 171-193.
40. Hansen, H. B., & Henriksen, J. H. (1997). How well does journal "impact" work in the assessment of papers on clinical physiology and nuclear medicine. Clinical Physiology, 17, 409-418.
41. Holden, G., Rosenberg, G., & Barker, K. (2005a). Tracing thought through time and space: A selective review of bibliometrics in social work. Social Work in Health Care, 41 (3/4). 1-34.
42. Holden, G., Rosenberg, G., & Barker, K. (2005b). Shallow science or meta-cognitive insights: A few thoughts on bibliometrics. Social Work in Health Care, 41 (3/4). 129-148.
43. Holden, G., Rosenberg, G., & Barker, K. (2005c). Bibliometrics: A potential decision making aid in hiring, reappointment, tenure and promotion decisions. Social Work in Health Care, 41 (3/4). 67-92.
44. Howard, M. O., & Howard, D. A. (1992). Citation analysis of 541 articles published in drug and alcohol journals: 1984-1988. Journal of Studies on Alcohol, 53, 427-434.
45. Hutcheson, P. (1997). The corrosion of tenure: A bibliography. NEA Higher Education Journal, Fall, 89-106.
46. ISI. (2003). Journal Citation Reports. Retrieved November 12, 2004, from http://www.isinet.com/
47. ISI. (2004). Journal Citation Reports. Retrieved November 12, 2004, from http://www.isinet.com/
48. Katz, J. S. (1999). Bibliometric indicators and the social sciences. Retrieved July 27, 2003, from http://www.sussex.ac.uk/Users/sylvank/pubs/ESRC. pdf
49. Kirk, S. A. (1984). Methodological issues in the comparative study of schools of social work. Journal of Social Service Research, 7 (3). 59-73.
50. Kostoff, R. N. (1998). The use and misuse of citation analysis in research evaluation. Scientometrics, 43 (1). 27-43.
51. Kostoff, R. N. (2002). Citation analysis of research performer quality. Scientometrics, 53, 49-71.
52. Kreuger, L. W. (1993). Should there be a moratorium on articles that rank schools of social work based on faculty publications? Yes! Journal of Social Work Education, 29, 240-246.
53. Kurmis, A. P. (2003). Understanding the limitations of the journal impact factor. Journal of Bone and Joint Surgery, 85A, 2449-2454.
54. Linardi, P. M., Coelho, P. M. Z., & Costa, H. M. A. (1996). The "impact factor" as a criterion for the quality of scientific production is a relative, not absolute, measure. Brazilian Journal of Medical and Biological Research, 29, 555-561.
55. Lindsey, D. (1976). Distinction, achievement and editorial board membership. American Psychologist, 31, 799-804.
56. Lindsey, D. (1978). The scientific publication system in social science. San Francisco: Jossey-Bass.
57. Lindsey, D. (1980). Production and citation measures in the sociology of science: The problem of multiple authorship. Social Studies of Science, 10, 145-162.
58. Lindsey, D. (1982). Further evidence for adjusting for multiple authorship. Scientometrics, 4, 389-395.
59. Lindsey, D. (1989). Using citation counts as a measure of quality in science: Measuring what's measurable rather than what's valid. Scientometrics, 15, 189-203.
60. Lindsey, D., & Kirk, S. A. (1992). The role of social work journals in the development of a knowledge base for the profession. Social Service Review, 66, 295-310.
61. MacRoberts, M. H., & MacRoberts, B. R. (1989). Problems of citation analysis: A critical review. Journal of the American Society for Information Science, 40, 342-349.
62. MacRoberts, M. H., & MacRoberts, B. R. (1997). Citation content analysis of a botany journal. Journal of the American Society for Information Science, 48, 274-275.
63. Meittunen, J., & Nieminen, P. (2003). The effect of statistical methods and study reporting characteristics on the number of citations: A study of four general psychiatric journals. Scientometrics, 57, 377-388.
64. Moed, H. F. (2005). Citation analysis in research evaluation. Dordrecht, Netherlands: Springer.
65. Monastersky, R. (2005a). Impact factors run into competition. Chronicle of Higher Education, 52 (8). A17 - A17.
66. Monastersky, R. (2005b). The number that's devouring science. Chronicle of Higher Education, 52 (8). A12 - A12.
67. Oromaner, M. (1974). Collaboration and impact: The career of multiauthored publications. Social Science Information, 14, 147-155.
68. Ouwehand, A. (2002). Power calculator. Retrieved March 31, 2005, from http://calculators.stat.ucla.edu/powercalc/
69. Pardeck, J. T. (1992). Are social work journal editorial boards competent? Some disquieting data with implications for research on social work practice. Research on Social Work Practice, 2, 487-496.
70. Pardeck, J. T., Arndt, B. J., Light, D. B., Mosley, G. F., Thomas, S. D., Werner, M. A., et al. (1991). Distinction and achievement levels of editorial board members of psychology and social work journals. Psychological Reports, 68, 523-527.
71. Pardeck, J. T., Chung, W. S., & Murphy, J. W. (1995). An examination of the scholarly productivity of social work journal editorial board members and guest reviewers. Research on Social Work Practice, 5, 223-234.
72. Pardeck, J. T., & Meinart, R. G. (1999). Scholarly achievements of the Social Work editorial board and consulting editors: A commentary. Research on Social Work Practice, 9, 86-91.
73. Phelan, T. J. (1999). A compendium of issues for citation analysis. Scientometrics, 45, 117-136.
74. Plomp, R. (1990). The significance of the number of highly cited papers as an indicator of scientific prolificacy. Scientometrics, 19, 185-197.
75. Porter, A. L. (1977). Citation analysis: Queries and caveats. Social Studies of Science, 7, 257-267.
76. Rhoades, G. (1998). Managed professionals: Unionized faculty and restructuring academic labor. Albany: State University of New York Press.
77. Rhoads, R. A., & Rhoades, G. (2005). Graduate employee unionization as symbol of and challenge to the corporatization of U.S. research universities. Journal of Higher Education, 76 (3). 243-275.
78. Rosenberg, G., Holden, G., & Barker, K. (2005). What happens to our ideas? A bibliometric analysis of articles in Social Work in Health Care in the 1990s. Social Work in Health Care, 41, 35-66.
79. Saha, S., Saint, S., & Christakis, D. A. (2003). Impact factor: A valid measure of journal quality? Journal of the Medical Library Association, 91, 42-46.
80. SAS Institute Inc. (2003). SAS 9.1 for Windows [Computer program]. Cary, NC: SAS Institute.
81. Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461-464.
82. Seglen, P. O. (1992). The skewness of science. Journal of the American Society for Information Science, 43 (9). 628-638.
83. Seglen, P. O. (1994). Causal relationship between article citedness and journal impact. Journal of the American Society for Information Science, 45 (1). 1-11.
84. Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. British Medical Journal, 314, 498-502. Retrieved February 17, 2003, from http://bmj.com/cgi/content/full/314/7079/497
85. Seipel, M. M. O. (2003). Assessing publication for tenure. Journal of Social Work Education, 39, 79-88.
86. Sellers, S. L., Mathiesen, S. G., Perry, R., & Smith, T. (2004). Evaluation of social work journal quality: Citation versus reputation approaches. Journal of Social Work Education, 40, 143-160.
87. Sevinc, A. (2004). Manipulating impact factor: An unethical issue or an editor's choice. Swiss Medical Weekly, 134, 410-410.
88. Slaughter, S., & Leslie, L. L. (1997). Academic capitalism: Politics, policies, and the entrepreneurial university. Baltimore: Johns Hopkins University Press.
89. Smith, R. (1997). Journal accused of manipulating impact factor. British Medical Journal, 314, 461-461.
90. Smith, R. (1998). Unscientific practice flourishes in science. British Medical Journal, 316, 1036-1040.
91. Snijders, T. A. B., & Bosker, R. J. (1994). Modeled variance in two-level models. Sociological Methods & Research, 22, 342-363.
92. Snijders, T. A. B., & Bosker, R. J. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. London: Sage.
93. Staff. (1999). Citations reveal concentrated influence: Some fields have it, but what does it mean? Science Watch, 10 (1). 1-2.
94. Staff. (2004b). The undue importance of the impact factor. European Journal of Cancer, 40, 919-919.
95. Statistical Package for the Social Sciences. (2004). SPSS for Windows, 13.0. Chicago: Author.
96. Tenure Modifications. (2005, September). Copy editor. New York University: Faculty Senators Council, p. 3-3.
97. Thyer, B. A. (2004). Evaluating social work journal publication systems. Journal of Social Work Education, 40, 361-363.
98. Urbaniak, G. C., & Plous, S. (2005). Research randomizer. Retrieved July 31, 2005, from http://www.randomizer.org/
99. Weale, A. R., Bailey, M., & Lear, P. A. (2004). The level of non-citation of articles within a journal as a measure of quality: A comparison to the impact factor. BMC Medical Research Methodology, p. 4-4. Retrieved July 15, 2005, from http://www.biomedcentral.com/1471-2288/4/14
100. Weigel, R. J., Dracon, G., Radhakrishnan, R., Rho, Y., Sevgen, F., & Dafoe, D. C. (2004). Incentive systems for academic productivity in a depart of surgery. Journal of the American College of Surgeons, 199, 300-307.
101. Whitehouse, G. H. (2001). Citation rates and impact factors: Should they matter? British Journal of Radiology, 74, 1-3.
102. Zwahlen, M., Junker, C., & Egger, M. (2004). Commentary II. The journal impact factor in the evaluation of research quality: Villain, scapegoat or innocent bystander? Sozial Und Praventivmedizin, 49, 1, 19-22.