A multilevel modelling approach to investigating the predictive validity of editorial decisions: Do the editors of a high profile journal select manuscripts that are highly cited after publication?

Journal of the Royal Statistical Society. Series A: Statistics in Society

Volumn 174, Issue 4, 2011, Pages 857-879

  Bornmann, Lutz ^a   Mutz, Rüdiger ^b   Marx, Werner ^c   Schier, Hermann ^c   Daniel, Hans Dieter ^d  

^a MAX PLANCK INSTITUTE   (Germany)

^b ETH ZURICH   (Switzerland)

^c MAX PLANCK INSTITUT FÜR FESTKÖRPERFORSCHUNG   (Germany)

^d UNIVERSITY OF ZURICH   (Switzerland)

Author keywords

Chemistry; Editorial decisions; Multilevel modelling; Peer review; Predictive validity; Reference standard

Indexed keywords

EID: 79959377015     PISSN: 09641998     EISSN: 1467985X     Source Type: Journal    
DOI: 10.1111/j.1467-985X.2011.00689.x     Document Type: Article

References (85)

1. Adler, R., Ewing, J., Taylor, P. and Hall, P. G. (2009) A report from the International Mathematical Union (IMU) in cooperation with the International Council of Industrial and Applied Mathematics (ICIAM) and the Institute of Mathematical Statistics (IMS). Statist. Sci., 24, 1-28.
2. Aksnes, D. W. (2003) Characteristics of highly cited papers. Res. Eval., 12, 159-170.
3. Archambault, É. and Larivière, V. (2009) History of the journal impact factor: contingencies and consequences. Scientometrics, 79, 635-649.
4. Bauer, D. J. (2009) A note on comparing the estimates of models for cluster-correlated or longitudinal data with binary or ordinal outcomes. Psychometrika, 74, 97-105.
5. van den Besselaar, P. and Leydesdorff, L. (2007) Past Performance as Predictor of Successful Grant Applications: a Case Study. Den Haag: Rathenau Instituut.
6. Bordons, M. and Barrigon, S. (1992) Bibliometric analysis of publications of Spanish pharmacologists in the SCI (1984-89): part II, Contribution to subfields other than pharmacology and pharmacy (ISI). Scientometrics, 25, 425-446.
7. Bornmann, L. (2010) Towards an ideal method of measuring research performance: some comments to the Opthof and Leydesdorff (2010) paper. J. Inf., 4, 441-443.
8. Bornmann, L. (2011) Scientific peer review. A. Rev. Inform. Sci. Technol., 45, 199-245.
9. Bornmann, L. and Daniel, H.-D. (2005) Selection of research fellowship recipients by committee peer review: analysis of reliability, fairness and predictive validity of Board of Trustees' decisions. Scientometrics, 63, 297-320.
10. Bornmann, L. and Daniel, H.-D. (2006) Selecting scientific excellence through committee peer review-a citation analysis of publications previously published to approval or rejection of post-doctoral research fellowship applicants. Scientometrics, 68, 427-440.
11. Bornmann, L. and Daniel, H.-D. (2007) Multiple publication on a single research study: does it pay?: the influence of number of research articles on total citation counts in biomedicine. J. Am. Soc. Inf. Sci. Technol., 58, 1100-1107.
12. Bornmann, L. and Daniel, H.-D. (2008a) The effectiveness of the peer review process: interreferee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. Angew. Chem. Int. Edn, 47, 7173-7178.
13. Bornmann, L. and Daniel, H.-D. (2008b) Selecting manuscripts for a high impact journal through peer review: a citation analysis of Communications that were accepted by Angewandte Chemie International Edition, or rejected but published elsewhere. J. Am. Soc. Inf. Sci. Technol., 59, 1841-1852.
14. Bornmann, L. and Daniel, H.-D. (2008c) What do citation counts measure?: a review of studies on citing behavior. J. Doc., 64, 45-80.
15. Bornmann, L. and Daniel, H.-D. (2009a) The luck of the referee draw: the effect of exchanging reviews. Learn. Publ., 22, 117-125.
16. 0 at the micro level using data from chemistry. J. Am. Soc. Inf. Sci. Technol., 60, 1664-1670.
17. Bornmann, L. and Daniel, H.-D. (2010) The manuscript reviewing process-empirical research on review requests, review sequences and decision rules in peer review. Libr. Inf. Sci. Res., 32, 5-12.
18. Bornmann, L., Leydesdorff, L. and van den Besselaar, P. (2010) A meta-evaluation of scientific research proposals: different ways of comparing rejected to awarded applications. J. Inf., 4, 211-220.
19. Bornmann, L., Leydesdorff, L. and Marx, W. (2007) Citation environment of Angewandte Chemie. CHIMIA, 61, 104-109.
20. Bornmann, L., Marx, W., Schier, H., Thor, A. and Daniel, H.-D. (2010) From black box to white box at open access journals: predictive validity of manuscript reviewing and editorial decisions at Atmospheric Chemistry and Physics. Res. Eval., 19, 81-156.
21. Bornmann, L., Mutz, R. and Daniel, H.-D. (2007) The b index as a measure of scientific excellence: a promising supplement to the h index. Cybermetrics, 11, no.1, paper 6.
22. Bornmann, L., Mutz, R., Neuhaus, C. and Daniel, H.-D. (2008a) Use of citation counts for research evaluation: standards of good practice for analyzing bibliometric data and presenting and interpreting results. Eth. Sci. Environ. Polit., 8, 93-102.
23. Bornmann, L., Wallon, G. and Ledin, A. (2008b) Does the committee peer review select the best applicants for funding?: an investigation of the selection process for two European Molecular Biology Organization programmes. PLOS One, 3, no. 10, e3480.
24. Braun, T., Dióspatonyi, I., Zsindely, S. and Zádor, E. (2007) Gatekeeper index versus impact factor of science journals. Scientometrics, 71, 541-543.
25. Browne, W. J., Subramanian, S. V., Jones, K. and Goldstein, H. (2005) Variance partitioning in multilevel logistic models that exhibit overdisperson. J. R. Statist, Soc. A, 168, 599-613.
26. Craig, I. D., Plume, A. M., McVeigh, M. E., Pringle, J. and Amin, M. (2007) Do open access articles have greater citation impact?: a critical review of the literature. J. Inf., 1, 239-248.
27. Daniel, H.-D. (1993) Guardians of Science: Fairness and Reliability of Peer Review. Weinheim: Wiley.
28. Evidence Ltd. (2007) The Use of Bibliometrics to Measure Research Quality in UK Higher Education Institutions. London: Universities UK.
29. Fielding, A. (2004) Scaling for residual variance components of ordered category response in generalized linear mixed multilevel models. Qual. Quant., 38, 425-433.
30. Fielding, A., Yang, M. and Goldstein, H. (2003) Multilevel ordinal models for examination grades. Statist. Mod., 3, 127-153.
31. Giske, J. (2008) Benefitting from bibliometry. Eth. Sci. Environ. Polit., 8, 79-81.
32. Glänzel, W. and Schubert, A. (1992) Some facts and figures on highly cited papers in the sciences, 1981-1985. Scientometrics, 25, 373-380.
33. Glänzel, W., Schubert, A. and Czerwon, H. J. (1999) An item-by-item subject classification of papers published in multidisciplinary and general journals using reference analysis. Scientometrics, 44, 427-439.
34. Glänzel, W., Thijs, B., Schubert, A. and Debackere, K. (2009) Subfield-specific normalized relative indicators and a new generation of relational charts: methodological foundations illustrated on the assessment of institutional research performance. Scientometrics, 78, 165-188.
35. Godin, B. (2007) From eugenics to scientometrics: Galton, Cattell, and men of science. Socl Stud. Sci., 37, 691-728.
36. Goldstein, H. and Spiegelhalter, D. J. (1996) League tables and their limitations: statistical issues in comparisons of institutional performance. J. R. Statist. Soc. A, 159, 385-409.
37. Gölitz, P. (2005) Who is going to read all this? Angew. Chem. Int. Edn, 44, 5538-5541.
38. Gosden, H. (2003) Why not give us the full story?: functions of referees comments in peer reviews of scientific research papers. J. Engl. Acad. Purp., 2, 87-101.
39. Hames, I. (2007) Peer Review and Manuscript Management of Scientific Journals: Guidelines for Good Practice. Oxford: Blackwell.
40. Harnad, S. (2008) Validating research performance metrics against peer rankings. Eth. Sci. Environ. Polit., 8, 103-107.
41. Hedeker, D. (2008) Multilevel models for ordinal and nominal variablesInHandbook of Multilevel Analysis (eds J. De Leeuw and E. Meijer), pp. 237-274. New York: Springer.
42. Hedeker, D. and Gibbons, R. D. (2006) Longitudinal Data Analysis. Hoboken: Wiley-Interscience.
43. Hornbostel, S., Böhmer, S., Klingsporn, B., Neufeld, J. and von Ins, M. (2009) Funding of young scientist and scientific excellence. Scientometrics, 79, 171-190.
44. Hox, J. J. (2010) Multilevel Analysis: Techniques and Applications. New York: Routledge.
45. Jackson, C. (1996) Understanding Psychological Testing. Leicester: British Psychological Society.
46. Järvelin, K. and Persson, O. (2008) The DCI index: discounted cumulated impact-based research evaluation. J. Am. Soc. Inf. Sci. Technol., 59, 1433-1440.
47. Jayasinghe, U. W., Marsh, H. W. and Bond, N. (2003) A multilevel cross-classified modelling approach to peer review of grant proposals: the effects of assessor and researcher attributes on assessor ratings. J. R. Statist. Soc. A, 166, 279-300.
48. Jennings, C. G. (2006) Quality and value: the true purpose of peer review. What you can't measure, you can't manage: the need for quantitative indicators in peer review, doi 10.1038/nature05032.
49. Kostoff, R. N. (2002) Citation analysis of research performer quality. Scientometrics, 53, 49-71.
50. Kurtz, M. J. and Henneken, E. A. (2007) E-prints and journal articles in astronomy: a productive co-existence. Learn. Publshng, 20, 16-22.
51. Larivière, V. and Gingras, Y. (2010) The impact factor's Matthew Effect: a natural experiment in bibliometrics. J. Am. Soc. Inf. Sci. Technol., 61, 424-427.
52. van Leeuwen, T. N. (2007) Modelling of bibliometric approaches and importance of output verification in research performance assessment. Res. Eval., 16, 93-105.
53. Lewison, G., Thornicroft, G., Szmukler, G. and Tansella, M. (2007) Fair assessment of the merits of psychiatric research. Br. J. Psychiatr., 190, 314-318.
54. Liang, K.-Y. and Zeger, S. L. (1986) Longitudinal data analysis using generalized linear models. Biometrika, 73, 13-22.
55. Littell, R. C., Milliken, G. A., Stroup, W. W., Wolfinger, R. D. and Schabenberger, O. (2006) SAS for Mixed Models. Cary: SAS Institute.
56. Lock, S. (1985) A Difficult Balance: Editorial Peer Review in Medicine. Philadelphia: Institute for Scientific Information.
57. Marsh, H. W. and Ball, S. (1991) Reflections on the peer review process. Behav. Brain Sci., 14, 157-158.
58. Marsh, H. W., Jayasinghe, U. W. and Bond, N. W. (2008) Improving the peer-review process for grant applications-reliability, validity, bias, and generalizability. Am. Psychol., 63, 160-168.
59. McDonald, R. J., Cloft, H. J. and Kallmes, D. F. (2009) Fate of manuscripts previously rejected by the American Journal of Neuroradiology: a follow-up analysis. Am. J. Neuroradiol., 30, 253-256.
60. Melin, G. and Danell, R. (2006) The top eight percent: development of approved and rejected applicants for a prestigious grant in Sweden. Sci. Publ. Poly, 33, 702-712.
61. Mutz, R., Guilley, E., Sauter, U. H. and Nepveu, G. (2004) Modelling juvenile-mature wood transition in Scots pine (Pinus sylvestris L.) using nonlinear mixed-effects models. Ann. For. Sci., 61, 831-841.
62. Neuhaus, C. and Daniel, H.-D. (2009) A new reference standard for citation analysis in chemistry and related fields based on the sections of Chemical Abstracts. Scientometrics, 78, 219-229.
63. Neuhaus, C., Marx, W. and Daniel, H.-D. (2009) The publication and citation impact profiles of Angewandte Chemie and the Journal of the American Chemical Society based on the sections of Chemical Abstracts: a case study on the limitations of the Journal Impact Factor. J. Am. Soc. Inf. Sci. Technol., 60, 176-183.
64. Opthof, T., Furstner, F., van Geer, M. and Coronel, R. (2000) Regrets or no regrets?: no regrets! The fate of rejected manuscripts. Cardvasc. Res., 45, 255-258.
65. Opthof, T. and Leydesdorff, L. (2010) Caveats for the journal and field normalizations in the CWTS (''Leiden'') evaluations of research performance. J. Inf., 4, 423-430.
66. Pendlebury, D. A. (2008) Using Bibliometrics in Evaluating Research. Philadelphia: Thomson Scientific.
67. Plomp, R. (1990) The significance of the number of highly cited papers as an indicator of scientific prolificacy. Scientometrics, 19, 185-197.
68. van Raan, A. F. J. (2005) For your citations only?: hot topics in bibliometric analysis. Measurement, 3, 50-62.
69. Radicchi, F., Fortunato, S. and Castellano, C. (2008) Universality of citation distributions: toward an objective measure of scientific impact. Proc. Natn. Acad. Sci. USA, 105, 17268-17272.
70. Raudenbush, S. W., Bryk, A. S., Cheong, Y. F. and Congdon, Jr, R. T. (2004) HLM 6: Hierarchical Linear and Nonlinear Modeling. Lincolnwood: Scientific Software International.
71. Reinhart, M. (2009) Peer review of grant applications in biology and medicine: reliability, fairness, and validity. Scientometrics, 81, 789-809.
72. Rons, N. and Amez, L. (2008) Impact Vitality-a measure for excellent scientists. In Excellence and Emergence: a New Challenge for the Combination of Quantitative and Qualitative Approaches (eds J. Gorraiz and E. Schiebel), pp. 211-213. Vienna: Austrian Research Centers.
73. Rosenbaum, P. R. (2002) Observational Studies. New York: Springer.
74. Rousseau, R. (2005) Median and percentile impact factors: a set of new indicators. Scientometrics, 63, 431-441.
75. Schubert, A. and Braun, T. (1996) Cross-field normalization of scientometric indicators. Scientometrics, 36, 311-324.
76. Seglen, P. O. (1997) Why the impact factor of journals should not be used for evaluating research. Br. Med. J., 314, 498-502.
77. Shatz, D. (2004) Peer Review: a Critical Inquiry. Lanham: Rowman and Littlefield.
78. Thomson Reuters (2008) Using Bibliometrics: a Guide to Evaluating Research Performance with Citation Data. Philadelphia: Thomson Reuters.
79. Tijssen, R. and van Leeuwen, T. (2006) Centres of research excellence and science indicators: can 'excellence' be captured in numbers? In Proc. 9th Int. Conf. Science and Technology Indicators, Leuven (ed. W. Glänzel), pp. 146-147. Leuven: Katholieke Universiteit Leuven.
80. Tijssen, R., Visser, M. and van Leeuwen, T. (2002) Benchmarking international scientific excellence: are highly cited research papers an appropriate frame of reference? Scientometrics, 54, 381-397.
81. Vinkler, P. (1997) Relations of relative scientometric impact indicators: the relative publication strategy index. Scientometrics, 40, 163-169.
82. Wilson, J. D. (1978) Peer review and publication. J. Clin. Invest., 61, 1697-1701.
83. Winship, C. and Mare, R. D. (1983) Structural equations and path analysis for discrete data. Am. J. Sociol., 89, 54-110.
84. Winship, C. and Mare, R. D. (1984) Regression models with ordinal variables. Am. Sociol. Rev., 49, 512-525.
85. Wuchty, S., Jones, B. F. and Uzzi, B. (2007) The increasing dominance of teams in production of knowledge. Science, 316, 1036-1039.