Multidimensional assessment of scholarly research impact

Journal of the Association for Information Science and Technology

Volumn 66, Issue 10, 2015, Pages 1988-2002

  Moed, Henk F ^a   Halevi, Gali ^a  

^a ELSEVIER MASSON   (France)

Author keywords

evaluation

Indexed keywords

ASSESSMENT MATRIXES; ASSESSMENT PROCESS; EVALUATION; MULTIDIMENSIONAL RESEARCH; RESEARCH IMPACTS; RESEARCH PERFORMANCE; SCHOLARLY RESEARCH; SCIENTIFIC OUTPUT;

EID: 84948156759     PISSN: 23301635     EISSN: 23301643     Source Type: Journal    
DOI: 10.1002/asi.23314     Document Type: Article

References (88)

1. Abbott, A., Cyranoski, D., Jones, N., Maher, B., Schiermeier, Q., &, Van Noorden, R., (2010). Metrics: Do metrics matter? Nature, 465 (7300), 860-862.
2. Ackers, L., (2005). Moving people and knowledge: Scientific mobility in the European Union1. International Migration, 43 (5), 99-131.
3. Adie, E., &, Roe, W., (2013). Altmetric: Enriching scholarly content with article-level discussion and metrics. Learned Publishing, 26 (1), 11-17.
4. Alberts, B., (2013). " Impact Factor Distortions." Science (American Association for the Advancement of Science), 340 (6134), 787.
5. Arthur, M.W., &, Blitz, C., (2000). Bridging the gap between science and practice in drug abuse prevention through needs assessment and strategic community planning. Journal of Community Psychology, 28 (3), 241-255.
6. AUBR Expert Group (2010). Expert Group on the Assessment of University-Based Research. Assessing Europe's university-based research. European Commission-DG Research. EUR 24187 EN. Retrieved from http://ec.europa.eu/research/science-society/document-library/pdf-06/assessing-europe-university-based-research-en.pdf
7. Ball, P., (2007). Achievement index climbs the ranks. Nature, 448 (7155), 737.
8. Barjak, F., Li, X., &, Thelwall, M., (2007). Which Factors explain the web impact of scientists' personal homepages? Journal of the American Society for Information Science and Technology, 58 (2), 200-211.
9. Billaut, J.C., Bouyssou, D., &, Vincke, P., (2010). Should you believe in the Shanghai ranking? Scientometrics, 84 (1), 237-263.
10. Björk, B.C., &, Paetau, P., (2012). Open access to the scientific journal literature-status and challenges for the information systems community. Bulletin of the American Society for Information Science and Technology, 38 (5), 39-44.
11. Bollen, J., &, Van De Sompel, H., (2008). Usage impact factor: The effects of sample characteristics on usage-based impact metrics. Journal of the American Society for Information Science and Technology, 59 (1), 136-149.
12. Bonzi, S., &, Snyder, H.W., (1991). Motivations for citation: A comparison of self citation and citation to others. Scientometrics, 21 (2), 245-252.
13. Bornmann, L., &, Marx, W., (2014). How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations. Scientometrics, 98, 487-509.
14. Bornmann, L., Mutz, R., &, Daniel, H., (2013). Multilevel-statistical reformulation of citation-based university rankings: The Leiden ranking 2011/2012. Journal of the American Society for Information Science and Technology [Epub ahead of print].
15. Bozeman, B., Dietz, J.S., &, Gaughan, M., (2001). Scientific and technical human capital: An alternative model for research evaluation. International Journal of Technology Management, 22 (7), 716-740.
16. Burns, C.S., Lana, A., &, Budd, J.M., (2013). Institutional repositories: Exploration of costs and value. D-Lib Magazine, 19, 1-2.
17. Calero-Medina, C., Lõpez-Illescas, C., Visser, M.S., &, Moed, H.F., (2008). Important factors in the interpretation of bibliometric rankings of world universities. Research Evaluation, 17, 71-81.
18. Chamberlain, S., (2013). Consuming article-level metrics: Observations and lessons from comparing aggregator provider data. Information Standards Quality, 25, (4-13).
19. Chen, H., Roco, M.C., &, Son, J., (2013). Nanotechnology public funding and impact analysis: A tale of two decades (1991-2010). IEEE Nanotechnology Magazine, 7 (1), 9-14.
20. Cole, S., Cole, J.R., &, Simon, G.A., (1981). Chance and consensus in peer review. Science, 214 (4523), 881-886.
21. EUROCRIS. (2013). Website of the European Organization for International Research Information, Retrieved from http://www.eurocris.org/
22. European Commission. (2010b). Assessing Europe's university-based research. Report by the expert group on assessment of university-based research.
23. Ewell, P., (2009). Assessment, Accountability, and Improvement: Revisiting the Tension. Retrieved from: http://www.learningoutcomeassessment.org/occasionalpaperone.htm
24. F1000. Faculty of 1000. http://f1000.com/prime
25. Fetterman, D.M., (1994). Empowerment evaluation. Evaluation Practice, 15 (1), 1-15.
26. Fetterman, D., &, Wandersman, A., (2007). Empowerment evaluation: Yesterday, today, and tomorrow. American Journal of Evaluation, 28 (2), 179-198.
27. Fralinger, L., &, Bull, J., (2013). Measuring the international usage of US institutional repositories. OCLC Systems and Services, 29, 134-150.
28. Galligan, F., &, Dyas-Correia, S., (2013). Altmetrics: Rethinking the way we measure. Serials Review, 39 (1), 56-61.
29. Garfield, E., (1964). The citation index-A new dimension in indexing. Science, 144, 649-654.
30. Garfield, E., (1994). The application of citation indexing to journals management. Current Contents, 33, 3-5.
31. González-Pereira, B., Guerrero-Bote, V.P., &, Moya-Anegón, F., (2011). The SJR indicator: A new indicator of journals' scientific prestige. 2009. arXiv preprint arXiv:0912.4141.
32. Google Blog. Google Scholar Citations Open to All, Google, 16 November 2011. Retrieved from http://googlescholar.blogspot.com/2011/11/google-scholar-citations-open-to-all.html
33. Hazelkorn, E., (2011). Rankings and the reshaping of higher education. The battle for world-class excellence. London: Palgrave Macmillan.
34. Hicks, D., (2009). Evolving regimes of multi-university research evaluation. Higher Education, 57 (4), 393-404.
35. Hicks, D., (2012). Performance-based university research funding systems. Research Policy, 41 (2), 251-261.
36. Hirsch, J.E., (2005). An index to quantify an individual's scientific research output. Proceedings of the National Academy of the Sciences U S A, 102 (46), 16569-16572.
37. Huang, M., Chen, S., Lin, C., &, Chen, D., (2013). Exploring temporal relationships between scientific and technical fronts: A case of biotechnology field. Scientometrics, 1-16.
38. Imbens, G.M., &, Wooldridge, J.M., (2008). Recent developments in the econometrics of program evaluation (No. w14251). National Bureau of Economic Research.
39. Katz, J.S., &, Hicks, D., (1997). Desktop scientometrics. Scientometrics, 38 (1), 141-153.
40. Kurtz, M.J., &, Bollen, J., (2010). Usage bibliometrics. Annual Review of Information Science and Technology, 44 (1), 1-64.
41. Lane, J.P., &, Flagg, J.L., (2010). Translating three states of knowledge: discovery, invention and innovation. Implementation Science, 5 (9), 1-14.
42. Lee, L.S., Pusek, S.N., McCormack, W.T., Helitzer, D.L., Martina, C.A., Dozier, A.M.,... (2012). Clinical and translational scientist career success: Metrics for evaluation. Clinical and Translational Science, 5 (5), 400-407.
43. Lin, C., Huang, M., &, Chen, D., (2013). The influences of counting methods on university rankings based on paper count and citation count. Journal of Informetrics, 7 (3), 611-621.
44. Liu, N.C., &, Cheng, Y., (2005). The academic ranking of world universities-Methodologies and problems. Higher Education in Europe, 30 (2), 127-136.
45. Luther, J., (2002). White paper on electronic usage statistics. Serial Librarian, 41 (2), 119-148.
46. Martin, B.R., (1996). The use of multiple indicators in the assessment of basic research. Scientometrics, 36 (3), 343-362.
47. Martin, B.R., &, Irvine, J., (1983). Assessing basic research: Some partial indicators of scientific progress in radio astronomy, Research Policy, 12 (2), 61-90.
48. Martinez, A., Dimitriadis, Y., Rubia, B., Gõmez, E., &, De La Fuente, P., (2003). Combining qualitative evaluation and social network analysis for the study of classroom social interactions. Computers & Education, 41 (4), 353-368.
49. McCain W.K., (2011). Eponymy and obliteration by incorporation: The case of the "Nash Equilibrium." Journal of the American Society for Information Science and Technology, 62 (7), 1412-1424.
50. Michel, J., &, Bettels, B., (2001). Patent citation analysis. A closer look at the basic input data from patent search reports. Scientometrics, 51, 185-201.
51. Miller, P., &, O'Leary, T., (2007). Mediating instruments and making markets: Capital budgeting, science and the economy. Accounting, Organizations and Society, 32 (7-8), 701-734.
52. Moed, H.F., (2005). Citation analysis in research evaluation. Dordrecht, Netherlands: Springer.
53. Moed, H.F., (2010). Measuring contextual citation impact of scientific journals. Journal of Informetrics, 4 (3), 265-277.
54. Moed, H.F., (2012). The use of big datasets in bibliometric research. Research Trends, 30. Retrieved from http://www.researchtrends.com/issue-30-september-2012/the-use-of-big-datasets-in-bibliometric-research/
55. Moed, H.F., Nederhof, A.J, &, Luwel, M., (2002). Towards performance in the humanities. Library Trends, 50, 498-520.
56. Narin, F., (1994). Patent bibliometrics. Scientometrics, 30, 147-155.
57. Narin, F., &, Olivastro, D., (1998). Linkage between patents and papers: An interim EPO/US comparison. Scientometrics, 41 (1), 51-59.
58. Nederhof, A.J., Luwel, M., &, Moed, H.F., (2001). Assessing the quality of scholarly journals in linquistics: An alternative to citation-based journal impact factors. Scientometrics 51, 241-265.
59. O'Connell, C., (2013). Research discourses surrounding global university rankings: Exploring the relationship with policy and practice recommendations. Higher Education, 65 (6), 709-723.
60. OECD (2010). Performance-based funding for public research in tertiary education institutions: workshop proceedings. Paris: OECD Publishing. http://dx.doi.org/10.1787/9789264094611-en
61. Osterrieder, A., (2013). The value and use of social media as communication tool in the plant sciences. Plant methods, 9 (1), 26.
62. Paiva, C.E., Lima, J.P.S.N., &, Paiva, B.S.R., (2012). Articles with short titles describing the results are cited more often. Clinics, 67 (5), 509-513.
63. Pusser, B., &, Marginson, S., (2013). University rankings in critical perspective. Journal of Higher Education, 84 (4), 544-568.
64. Reedijk, J., &, Moed, H.F., (2008). Is the impact of journal impact factors decreasing? Journal of Documentation, 64, 183-192.
65. REF (Research Excellence Framework). (2012). Panel Criteria and Working Methods. http://www.ref.ac.uk/media/ref/content/pub/panelcriteriaandworkingmethods/01-12.pdf
66. Rohn, J., (2012). Business as usual in judging the worth of a researcher? Retrieved from http://www.theguardian.com/science/occams-corner/2012/nov/30/1
67. Rowan-Szal, G.A., Greener, J.M., Joe, G.W., &, Simpson, D.D., (2007). Assessing program needs and planning change. Journal of Substance Abuse Treatment, 33 (2), 121-129.
68. Salter, A.J., &, Martin, B.R., (2001). The economic benefits of publicly funded basic research: a critical review. Research policy, 30 (3), 509-532.
69. Sarli, C., &, Holmes, K., (2013). The Becker Medical Library Model for Assessment of Research Impact model. Retrieved from https://becker.wustl.edu/impact-assessment
70. Schloegl, C., &, Gorraiz, J., (2010). Comparison of citation and usage indicators: The case of oncology journals. Scientometrics, 82 (3), 567-580.
71. Schmoch, U., (2004). The technological output of scientific institutions. In Handbook of quantitative science and technology research (pp. 717-731). Amsterdam: Springer.
72. Simons, K., (2008). The misused impact factor. Science, 322 (5899), 165.
73. Simpson, D.D., (2002). A conceptual framework for transferring research to practice. Journal of Substance Abuse Treatment, 22 (4), 171-182.
74. Sugimoto, C.R., Larivière, V., Ni, C., &, Cronin, B., (2013). Journal acceptance rates: A cross-disciplinary analysis of variability and relationships with journal measures. Journal of Informetrics, 7 (4), 897-906.
75. Taylor, M., (2013). Exploring the boundaries: How altmetrics can expand our vision of scholarly communication and social impact. Information Standards Quarterly, 25, 27-32.
76. Vale, R.D., (2012). Evaluating how we evaluate. Molecular Biology of the Cell, 23 (17), 3285-3289.
77. Van Noorden, R., (2013). Scientists join journal editors to fight impact-factor abuse. Nature News Blog (Nature Publishing Group).
78. Van Raan, A.F., (1996). Advanced bibliometric methods as quantitative core of peer review based evaluation and foresight exercises. Scientometrics, 36 (3), 397-420.
79. Van Raan, A.F.J., (2004). Measuring science. In, H.F. Moed, W. Glänzel, &, U. Schmoch, (Eds.), Handbook of quantitative science and technology research. The use of publication and patent statistics in studies of S&T systems (pp. 19-50). Dordrecht, the Netherlands: Kluwer Academic Publishers.
80. Van Raan, A.F.J., (2005). Fatal attraction-Conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics, 62 (1), 133-143.
81. Waltman, L., van Eck, N.J., &, Wouters, P., (2013). Counting publications and citations: Is more always better? Journal of Informetrics, 7 (3), 635-641.
82. Wandersman, A., Keener, D.C., Snell-Johns, J., Miller, R.L., Flaspohler, P., Livet-Dye, M.,... (2004). Empowerment evaluation: Principles and action. Participatory community research: Theories and methods in action. Washington, DC: American Psychological Association.
83. Wandersman, A., &, Snell-Johns, J., (2005). Empowerment evaluation: Clarity, dialoque, and growth. American Journal of Evaluation, 26 (3), 421-428.
84. Wang, X., Wang, Z., &, Xu, S., (2013). Tracing scientist's research trends realtimely. Scientometrics, 1-13.
85. Wilson, V., (2013). Research methods: Altmetrics. Evidence Based Library and Information Practice, 8 (1), 126-128.
86. Zare, R.N., (2012). Editorial: Assessing academic researchers. Angewandte Chemie, International Edition, 51 (30), 7338-7339.
87. Zellner, C., (2003). The economic effects of basic research: Evidence for embodied knowledge transfer via scientists' migration. Research Policy, 32 (10), 1881-1895.
88. Zuckerman, H., (1987). Citation analysis and the complex problem of intellectual influence. Scientometrics, 12 (5), 329-338.