Do rankings reflect research quality?

Journal of Applied Economics

Volumn 13, Issue 1, 2010, Pages 1-38

  Frey, Bruno S ^a   Rost, Katja ^a  

^a UNIVERSITY OF ZURICH   (Switzerland)

Author keywords

Evaluations; Rankings; Research quality; Scholars; Universities

Indexed keywords

EID: 77955585621     PISSN: 15140326     EISSN: 16676726     Source Type: Journal    
DOI: 10.1016/S1514-0326(10)60002-5     Document Type: Article

References (86)

1. Adam David The counting house. Nature 2002, 415:726-729.
2. Adler Robert, Ewing John, Taylor Peter Citation statistics, A report from the Joint Committee on Quantitative Assessment of Research 2008, IMU, ICIAM, IMS.
3. Banerjee Abhjit V. A simple model of herd behavior. Quarterly Journal of Economics 1992, 107:797-817.
4. Barabási Albert-László, Albert Réka Emergence of scaling in random networks. Science 1999, 286:509-512.
5. Bauwens, Luc (1998), A new method to rank university research and researchers in economics in Belgium, working paper, CORE, Université Catholique de Louvain.
6. Beed Clive, Beed Cara Measuring the quality of academic journals: The case of economics. Journal of Post Keynesian Economics 1996, 18:369-396.
7. Bikhchandani Sushil, Hirshleifer David, Welch Ivo A theory of fads, fashion, custom, and cultural change as informational cascades. Journal of Political Economy 1992, 100:992-1026.
8. Bommer Rolf, Ursprung Heinrich Spieglein, Spieglein an der Wand: Eine Publikationsanalytische Erfassung der Forschleistungen volkswirtschaftlicher Fachbereiche in Deutschland, Osterreich und der Schweiz. Zeitschrift fur Wirtschafts- und Sozialwissenschaften 1998, 118:1-28.
9. Bonitz Manfred, Bruckner Eberhard, Scharnhorst Andrea The Matthew Index - Concentration patterns and Matthew core journals. Scientometrics 1999, 44:361-378.
10. Borgman Christine L. Scholarly communication and bibliometrics 1990, Sage, Newbury Park.
11. Borgman Christine L., Furner Jonathan Scholarly communication and bibliometrics. Annual Review of Information Science and Technology 2002, 36.
12. Bräuninger Michael, Haucap Justus Reputation and relevance of economics journals. Kyklos 2003, 56:175-198.
13. Butler Linda Modifying publication practices in response to funding formulars. Research Evaluation 2003, 12:39-46.
14. Butler Linda Assessing university research: A plea for a balanced approach. Science and Public Policy 2007, 34:565-574.
15. Cheng Chun Hung, Holsapple Clyde W., Lee Anita Citation-based journal rankings for expert system research. Expert Systems 1995, 12:313-322.
16. Cherchye Laurens, Abeele Piet Vanden On research efficiency - A micro-analysis of Dutch university research in economics and business management. Research Policy 2005, 34:495-516.
17. Coats Alfred William The role of scholarly journals in the history of economics: An essay. Journal of Economic Literature 1971, 9:29-44.
18. Council for the Humanities Arts and Social Sciences Measures of quality and impact of publicly funded research in the humanities, arts and social sciences, Occasional Paper 2 http://www.chass.org.au/op2.pdf.
19. Coupé Tom Revealed performances: Worldwide rankings of economists and economics departements, 1990-2000. Journal of the European Economic Association 2003, 1:1309-1345.
20. Cronin Blaise, Meho Lokman I. The shifting balance of intellectual trade in information studies. Journal of the American Society for Information Science and Technology 2008, 59:551-564.
21. Dasgupta Partha, David Paul A. Towards a new economics of science. Research Policy 1994, 23:487-521.
22. De Ruyter Van Steveninck M.A. Opmars der Econometristen. Economische en Sociale Berichten 1998, 4180:965.
23. Diamond Arthur M. What is a citation worth?. Journal of Human Resources 1986, 21:200-215.
24. Diamond Arthur M. The core journals in economics. Current Contents 1989, 1:4-11.
25. Donovan Claire The chequered career of a cryptic concept. The Political Quarterly 2006, 77:61-72.
26. Donovan Claire, Butler Linda Testing novel quantitative indicators of research 'quality', esteem and 'user engagement': An economics pilot study. Research Evaluation 2007, 16:231-242.
27. Frey Bruno S. Publishing as prostitution? - Choosing between one's own ideas and academic success. Public Choice 2003, 116:205-223.
28. Frey, Bruno S., and Margit Osterloh (2006), Evaluations: Hidden costs, questionable benefits, and superior alternatives, Working Paper Series ISSN 1424-0459, Vol. 302, Social Science Research Network (SSRN).
29. Garfield Eugene Citation indexing: Its theory and application in science, technology, and humanities 1979, Wiley, New York.
30. Garfield Eugene Editors are justified in asking authors to cite equivalent references from same journal. British Medical Journal 1997, 314:1765.
31. Gibbons Jean D. United-States Institutional Representations on editorial-boards of United-States statistics journals. American Statistician 1990, 44:210-213.
32. Gibbons Jean D., Fish Mary Ranking of economics faculties and representation on editorial-boards of top-journals. Journal of Economic Education 1991, 22:361-372.
33. Groot Tom, Garcia-Valderrama Teresa Research quality and efficiency - An analysis of assessments and management issues in Dutch economics and business research programs. Research Policy 2006, 35:1362-1376.
34. Handelsblatt Economist Ranking http://www.handelsblatt.com/politik/vwl-ranking/
35. Henrekson, Magnus, and Daniel Waldenstrom (2007), How should research performance be measured?, Working Paper 712, IFN.
36. Hodgson Geoffrey M., Rothman Harry The editors and authors of economics journals: A case of institutional oligopoly?. Economic Journal 1999, 109:F165-F186.
37. Holmström Bengt R., Milgrom Paul Multitask principal-agent analyses: Incentice contracts, assert ownership and job design. Journal of Law, Economics and Organization 1991, 7:24-52.
38. Horrobin David F. The philosophical basis of peer review and the suppression of innovation. Journal of the American Medical Association 1990, 263:1438-1441.
39. Ideas, Repec IDEAS Ranking http://ideas.repec.org/top/.
40. Johnes Geraint Determinants of research output in economics departments in British universities. Research Policy 1988, 17:171-178.
41. Judge Institute of Management Studies Quality rankings of journals in management and social science - The Judge Institute of Management Studies list, mimeo 1994, University of Cambridge, Cambridge.
42. Katz J.Sylvan Bibliometric indicators and the social sciences. Report to the UK Economic and Social Research Council, SPRU, University of Sussex http://www.sussex.ac.uk/Users/sylvank/pubs/ESRC.pdf.
43. Kaufman George G. Rankings of finance departments by faculty representation on editorial boards of professional journals - A note. Journal of Finance 1984, 39:1189-1197.
44. Kurtz David L., Boone Louis E. Rating marketing faculties on the basis of editorial review board memberships. Journal of Marketing Education 1988, 10:64-68.
45. Laband David N., Piette Michael J. The relative impacts of economics journals - 1970-1990. Journal of Economic Literature 1994, 32:640-666.
46. Lee Frederic S. The ranking game, class and scholarship in American mainstream economics. Australasian Journal of Economics Education 2006, 3:1-41.
47. Lee Frederic S. The Research Assessment Exercise, the state and the dominance of mainstream economics in British universities. Cambridge Journal of Economics 2007, 31:309-325.
48. Lee Frederic S. A case for ranking heterodox journals and departments. On the Horizon 2008, 16:241-251.
49. Liebowitz Stan J., Palmer John P. Assessing the relative impacts of economics journals. Journal of Economic Literature 1984, 22:77-88.
50. Lo Melody, Sunny Wong M.C., Mixon Franklin G. Ranking economics journals, economics departments, and economists using teaching-focused research productivity. Southern Economic Journal 2008, 74:894-906.
51. Luwel Marc, Moed Henk F., Nederhof Anton J., De Samblanx V., Verbrugghen K., Van Der Wurff L.J. Towards indicators of research performance in the social sciences and humanities 1999, CWTS, Leiden.
52. Meho Lokman I. The rise and rise of citation analysis. Physics World 2007, 20:32-36.
53. Meho Lokman I., Rogers Yvonne Citation counting, citation ranking, and h-index of human-computer interaction researchers: A comparison of Scopus and Web of Science. Journal of the American Society for Information Science and Technology 2008, 59:1711-1726.
54. Merton Robert K. The Matthew Effect in science. Science 1968, 159:56-63.
55. Merton Robert K. The sociology of science: theoretical and empirical investigation 1973, University of Chicago Press, Chicago, IL.
56. Metcalf Charles E. Can government-sponsered evaluations be independent?. Journal of Policy Analysis and Management 2008, 27:926-942.
57. Moed Henk F. Citation analysis in research evaluation 2005, Springer, Dordrecht.
58. Moed Henk F. UK research assessment exercises: Informed judgments on research quality or quantity?. Scientometrics 2008, 74:153-161.
59. Moed Henk F., Burger W.J.M., Frankfort J.G., Van Raan Anthony F.J. The use of bibliometric data for the measurement of university research performance. Research Policy 1985, 14:131-149.
60. Moxham H., Anderson J. Peer review. A view from the inside. Science and Technology Policy 1992, 5:7-15.
61. Nederhof Anton J., Van Raan Anthony F.J. A bibliometric analysis of six economics research groups: A comparison with peer review. Research Policy 1993, 22:353-368.
62. Nightingale Paul, Scott Alister Peer review and the relevance gap: Ten suggestions for policy-makers. Science and Public Policy 2007, 34:543-553.
63. Osterloh, Margit, and Bruno S. Frey (2009), Are more and better indicators the solution? Comment to William Starbuck, Scandinavian Journal of Management, forthcoming.
64. Podsakoff Philip M., Mackenzie Scott B., Bachrach Daniel G., Podsakoff Nathan P. The influence of management journals in the 1980s and 1990s. Strategic Management Journal 2005, 26:473-488.
65. Polanyi Michael The Republic of Science: Its political and economic theory. Minerva 1962, 1:54-73.
66. Reedijk Jan Sense and nonsense of science citation analyses: Comments on the monopoly position of ISI and citation inaccuracies. Risks of possible misuse and biased citation and impact data. New Journal of Chemistry 1998, 22:767-770.
67. Reingold David A. David A. Reingold's response to Charles E. Metcalfs. Journal of Policy Analysis and Management 2008, 27:942-944.
68. Rigby John, Edler Jakob Peering inside research networks: Some observations on the effect of the intensity of collaboration on the variability of research quality. Research Policy 2005, 34:784-794.
69. Rinia Ed J., Van Leeuwen Thed N., Van Vuren Hendrik G., Van Raan Anthony F.J. Comparitive analysis of a set of bibliometric indicators and central peer reviewcriteria. Evaluation of condensed matter physics in the Netherlands. Research Policy 1998, 27:95-107.
70. Roessler, Christian (2004), rankings 2004 (c). http://econphd.net.
71. Seglen Per O. Causal relationship between article citedness and journal impact. Journal of the American Society for Information Science 1994, 45:1-11.
72. Seglen Per O. Why the impact factor of journals should not be used for evaluating research. British Medical Journal 1997, 314:498-502.
73. Shanghai Jiao Tong University Academic ranking of world universities http://www.arwu.org/rank/2007/ranking2007.
74. Simkin Mikhail V., Roychowdhury Vwany P. Stochastic modeling of citation slips. Scientometrics 2005, 62:367-384.
75. Simkin Mikhail V., Roychowdhury Vwany P. Do you sincerely want to be cited? Or: read before you cite. Significance 2006, 12:179-181.
76. Simkin Mikhail V., Roychowdhury Vwany P. A mathematical theory of citing. Journal of the American Society for Information Science and Technology 2007, 58:1661-1673.
77. Sivertsen, Gunnar (2006), A bibliometric model for performance based budgeting of research institutions, Proceedings of the 9th International Conference on Science & Technology Indicators, Leuven, Belgium.
78. Stigler George J., Stigler Stephen M., Friedland Claire The journals of economics. Journal of Political Economy 1995, 103:331-359.
79. The Thomson Corporation ISI Web of Knowledge Essential Science Indicators http://esi.isiknowledge.com/home.cgi.
80. The Thomson Corporation ISI Web of Knowledge Journal Citation Report http://admin-apps.isiknowledge.com/JCR/.
81. Tijssen Robert J.W., Van Wijk Erik In search of the European paradox: An international comparison of Europe's scientific performance and knowledge flows in information and communication technologies research. Research Policy 1999, 28:519-543.
82. Van Raan Anthony F.J. The use of bibliometric analysis in research performance assessment and monitoring of interdisciplinary scientific developments. Technikfolgenabschätzung 2003, 1:20-29.
83. Ventura Oscar N., Mombru Alvaro W. Use of bibliometric information to assist research policy making. A comparison of publication and citation profiles of Full and Associate Professors at a School of Chemistry in Uruguay. Scientometrics 2006, 69:287-313.
84. Vieira Pedro Cosme Costa Statistical variability of top ranking economics journals impact. Applied Economics Letters 2004, 11:945-948.
85. Warner Julian A critical review of the application of citation studies to the Research Assessment Exercises. Journal of Information Science 2000, 26:453-459.
86. Weingart Peter Impact of bibliometrics upon the science system: Inadvertent consequences?. Scientometrics 2005, 62:117-131.