Interrater reliability and convergent validity of F1000Prime peer review

Journal of the Association for Information Science and Technology

Volumn 66, Issue 12, 2015, Pages 2415-2426

  Bornmann, Lutz ^a  

^a MAX PLANCK INSTITUTE   (Germany)

Author keywords

bibliometrics; citation analysis; informetrics

Indexed keywords

INFORMATION SCIENCE; PAPER; REGRESSION ANALYSIS;

BIBLIOMETRICS; BIOMEDICAL LITERATURE; CITATION ANALYSIS; CONVERGENT VALIDITY; HIGHLY-CITED PAPERS; INFORMETRICS; INTER-RATER RELIABILITIES; LOGISTIC REGRESSION MODELS;

RELIABILITY;

CONVERGENT VALIDITY; INTERRATER RELIABILITY; LOGISTIC REGRESSION ANALYSIS; MODEL; PEER REVIEW; UNIVERSITY;

EID: 84925512962     PISSN: 23301635     EISSN: 23301643     Source Type: Journal    
DOI: 10.1002/asi.23334     Document Type: Review

References (70)

1. Allen, L., Jones, C., Dolby, K., Lynn, D., &, Walport, M., (2009). Looking for landmarks: The role of expert review and bibliometric analysis in evaluating scientific publication outputs. PLoS One, 4 (6). doi: 10.1371/journal.pone.0005910
2. Anon,. (2006). Three cheers for peers. Nature, 439 (7073), 118.
3. Benda, W.G.G., &, Engels, T.C.E., (2011). The predictive validity of peer review: A selective review of the judgmental forecasting qualities of peers, and implications for innovation in science. International Journal of Forecasting, 27 (1), 166-182. doi: 10.1016/j.ijforecast.2010.03.003
4. Berry, W.D., &, Feldman, S., (1985). Multiple regression in practice. Thousand Oaks, CA: Sage Publications.
5. Bock, O., (2002). The peer-review process: Accept, revise, or reject? Cortex: A Journal Devoted to the Study of the Nervous System and Behavior, 38 (3), 419-420.
6. Bornmann, L., (2011). Scientific peer review. Annual Review of Information Science and Technology, 45, 199-245.
7. Bornmann, L., (2014). How are excellent (highly cited) papers defined in bibliometrics? A quantitative analysis of the literature. Research Evaluation, 23 (2), 166-173.
8. Bornmann, L., &, Daniel, H.-D., (2008a). Selecting manuscripts for a high impact journal through peer review: A citation analysis of communications that were accepted by Angewandte Chemie International Edition, or rejected but published elsewhere. Journal of the American Society for Information Science and Technology, 59 (11), 1841-1852. doi: 10.1002/asi.20901
9. Bornmann, L., &, Daniel, H.-D., (2008b). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64 (1), 45-80. doi: 10.1108/00220410810844150
10. Bornmann, L., &, de Moya Anegõn, F., (2014). What proportion of excellent papers makes an institution to one of the best worldwide? Specifying thresholds for the interpretation of the results of the SCImago Institutions Ranking and the Leiden Ranking. Journal of the American Society for Information Science and Technology, 65 (4), 732-736.
11. Bornmann, L., &, Leydesdorff, L., (2013). The validation of (advanced) bibliometric indicators through peer assessments: A comparative study using data from InCites and F1000. Journal of Informetrics, 7 (2), 286-291. doi: 10.1016/j.joi.2012.12.003
12. Bornmann, L., &, Marx, W., (2013). How good is research really? EMBO Reports, 14 (3), 226-230. doi: 10.1038/embor.2013.9
13. Bornmann, L., &, Mutz, R., (2011). Further steps towards an ideal method of measuring citation performance: The avoidance of citation (ratio) averages in field-normalization. Journal of Informetrics, 5 (1), 228-230.
14. Bornmann, L., &, Williams, R., (2013). How to calculate the practical significance of citation impact differences? An empirical example from evaluative institutional bibliometrics using adjusted predictions and marginal effects. Journal of Informetrics, 7 (2), 562-574. doi: 10.1016/j.joi.2013.02.005
15. Bornmann, L., Mutz, R., &, Daniel, H.-D., (2011). A reliability-generalization study of journal peer reviews-A multilevel meta-analysis of inter-rater reliability and its determinants. PLoS One, 5 (12), e14331.
16. Bornmann, L., Mutz, R., Marx, W., Schier, H., &, Daniel, H.-D., (2011). A multilevel modelling approach to investigating the predictive validity of editorial decisions: Do the editors of a high-profile journal select manuscripts that are highly cited after publication? Journal of the Royal Statistical Society. Series A, (Statistics in Society), 174 (4), 857-879. doi: 10.1111/j.1467-985X.2011.00689.x
17. Bornmann, L., de Moya Anegõn, F., &, Leydesdorff, L., (2012). The new excellence indicator in the World Report of the SCImago Institutions rankings 2011. Journal of Informetrics, 6 (2), 333-335. doi: 10.1016/j.joi.2011.11.006
18. Bornmann, L., Leydesdorff, L., &, Mutz, R., (2013). The use of percentiles and percentile rank classes in the analysis of bibliometric data: Opportunities and limits. Journal of Informetrics, 7 (1), 158-165.
19. Buela-Casal, G., &, Zych, I., (2010). Analysis of the relationship between the number of citations and the quality evaluated by experts in psychology journals. Psicothema, 22 (2), 270-275.
20. Callaham, M., &, McCulloch, C., (2011). Longitudinal trends in the performance of scientific peer reviewers. Annals of Emergency Medicine, 57 (2), 141-148. doi: 10.1016/j.annemergmed.2010.07.027
21. Cicchetti, D.V., (1991). The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation. The Behavioral and Brain Sciences, 14 (1), 119-135.
22. Cicchetti, D.V., &, Sparrow, S.A., (1981). Developing criteria for establishing interrater reliability of specific items-Applications to assessment of adaptive behavior. American Journal of Mental Deficiency, 86 (2), 127-137.
23. Cohen, J., (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates, Publishers.
24. Cole, J.R., (2000). The role of journals in the growth of scientific knowledge. In, B. Cronin, &, H.B. Atkins, (Eds.), The Web of Knowledge. A Festschrift in Honor of Eugene Garfield (pp. 109-142). Medford, NJ: Information Today.
25. de Vries, D.R., Marschall, E.A., &, Stein, R.A., (2009). Exploring the peer review process: What is it, does it work, and can it be improved? Fisheries, 34 (6), 270-279.
26. Eckberg, D.L., (1991). When non-reliability of reviews indicates solid science. The Behavioral and Brain Sciences, 14 (1), 145-146.
27. Eisenhart, M., (2002). The paradox of peer review: Admitting too much or allowing too little? Research in Science Education, 32 (2), 241-255.
28. Evered, D., Hamett, S., &, Narin, F., (1989). The impact of different modes of research funding. In, D. Evered, &, S. Hamett, (Eds.), The Evaluation of Scientific Research (pp. 120-140). Chichester, UK: John Wiley & Sons.
29. F1000. (2012). What is F1000? Retrieved from http://f1000.com/about/whatis
30. Fleiss, J., (1981). Statistical Methods for Rates and Proportions. New York: Wiley VCH.
31. Garfield, E., (1991). In truth, the flood of scientific literature is only a myth. The Scientist, 5 (17), 11.
32. Gelman, A., (2009). How does statistical analysis differ when analyzing the entire population rather than a sample? Retrieved from http://andrewgelman.com/2009/07/03/how-does-statis/
33. Glass, G.V., (1976). Primary, secondary, and meta-analysis. Educational Researcher, 5, 3-8.
34. Hardin, J., &, Hilbe, J., (2012). Generalized Linear Models and Extensions (3rd ed.). College Station, TX: Stata Corporation.
35. Harnad, S., (2008). Validating research performance metrics against peer rankings. Ethics in Science and Environmental Politics, 8 (1), 103-107.
36. Hosmer, D.W., &, Lemeshow, S., (2000). Applied Logistic Regression (2nd ed.). Chichester, UK: John Wiley & Sons, Inc.
37. Jennings, C.G., (2006). Quality and value: the true purpose of peer review. What you can't measure, you can't manage: The need for quantitative indicators in peer review. Retrieved from http://www.nature.com/nature/peerreview/debate/nature05032.html
38. Kreiman, G., &, Maunsell, J.H.R., (2011). Nine criteria for a measure of scientific output. Frontiers in Computational Neuroscience, 5 (48), doi: 10.3389/fncom.2011.00048
39. Li, X., &, Thelwall, M., (2012). F1000, Mendeley and traditional bibliometric indicators. In, E. Archambault, Y. Gingras, &, V. Lariviere, (Eds.), The 17th International Conference on Science and Technology Indicators (pp. 541-551). Montréal: Repro-UQAM.
40. Lozano, G.A., Larivière, V., &, Gingras, Y., (2012). The weakening relationship between the impact factor and papers' citations in the digital age. Journal of the American Society for Information Science and Technology, 63 (11), 2140-2145. doi: 10.1002/asi.22731
41. Marsh, H.W., Bond, N.W., &, Jayasinghe, U.W., (2007). Peer review process: assesSments by applicant-nominated referees are biased, inflated, unreliable and invalid. Australian Psychologist, 42 (1), 33-38.
42. Martin, B.R., &, Irvine, J., (1983). Assessing basic research-Some partial indicators of scientific progress in radio astronomy. Research Policy, 12 (2), 61-90.
43. Medical Research Council. (2009). Publications. Retrieved from http://www.mrc.ac.uk/Achievementsimpact/Outputsoutcomes/MRCe-Val2009/Publications/index.htm
44. Mitchell, M.N., (2012). Interpreting and Visualizing Regression Models Using Stata. College Station, TX: Stata Corporation.
45. Mohammadi, E., &, Thelwall, M., (2013). Assessing non-standard article impact using F1000 labels. Scientometrics, 97 (2), 383-395. doi: 10.1007/s11192-013-0993-9
46. Mutz, R., Bornmann, L., &, Daniel, H.-D., (2012). Heterogeneity of inter-rater reliabilities of grant peer reviews and its determinants: A general estimating equations approach. PLoS One, 7 (10), e48509. doi: 10.1371/journal.pone.0048509
47. Rabe-Hesketh, S., &, Everitt, B., (2004). A Handbook of Statistical Analyses Using Stata. Boca Raton, FL: Chapman & Hall/CRC.
48. Reichenheim, M.E., (2004). Confidence intervals for the kappa statistic. The Stata Journal, 4 (4), 421-428.
49. Schreiber, M., (2013). How much do different ways of calculating percentiles influence the derived performance indicators? A case study. Scientometrics, 97 (3), 821-829.
50. Schubert, A., &, Braun, T., (1986). Relative indicators and relational charts for comparative assessment of publication output and citation impact. Scientometrics, 9 (5-6), 281-291.
51. Sheskin, D., (2007). Handbook of Parametric and Nonparametric Statistical Procedures (4th ed.). Boca Raton, FL: Chapman & Hall/CRC.
52. Siegelman, S.S., (1991). Assassins and zealots-Variations in peer review-Special report. Radiology, 178 (3), 637-642.
53. Smith, L.C., (1981). Citation analysis. Library Trends, 30 (1), 83-106.
54. Starbuck, W.H., (2005). How much better are the most-prestigious journals? The statistics of academic publication. Organization Science, 16 (2), 180-200.
55. StataCorp. (2013). Stata statistical software: Release 13. College Station, TX: Stata Corporation.
56. Thorngate, W., Dawes, R.M., &, Foddy, M., (2009). Judging Merit. New York: Psychology Press.
57. van Raan, A., (2012). Properties of journal impact in relation to bibliometric research group performance indicators. Scientometrics, 92 (2), 457-469. doi: 10.1007/s11192-012-0747-0
58. von Eye, A., &, Mun, E.Y., (2005). Analyzing Rater Agreement. Manifest Variable Methods. Mahwah, NJ: Lawrence Erlbaum Associates.
59. Waltman, L., &, Costas, R., (2014). F1000 recommendations as a potential new data source for research evaluation: a comparison with citations. Journal of the Association for Information Science and Technology, 65 (3), 433-445.
60. Waltman, L., Calero-Medina, C., Kosten, J., Noyons, E.C.M., Tijssen, R.J.W., van Eck, N.J., &, Wouters, P., (2012). The Leiden Ranking 2011/2012: Data collection, indicators, and interpretation. Journal of the American Society for Information Science and Technology, 63 (12), 2419-2432.
61. Waltman, L., van Eck, N.J., van Leeuwen, T.N., &, Visser, M.S., (2013). Some modifications to the SNIP journal impact indicator. Journal of Informetrics, 7 (2), 272-285. doi: 10.1016/j.joi.2012.11.011
62. Wardle, D.A., (2010). Do "Faculty of 1000" (F1000) ratings of ecological publications serve as reasonable predictors of their future impact? Ideas in Ecology and Evolution, 3, 11-15.
63. Weller, A.C., (2002). Editorial Peer Review: Its Strengths and Weaknesses. Medford, NJ: Information Today, Inc.
64. Wets, K., Weedon, D., &, Velterop, J., (2003). Post-publication filtering and evaluation: Faculty of 1000. Learned Publishing, 16 (4), 249-258.
65. Wiley, S., (2008). Peer review isn't perfect... but it's not a conspiracy designed to maintain the status quo. The Scientist, 22 (11), 31.
66. Williams, R., (2012). Using the margins command to estimate and interpret adjusted predictions and marginal effects. The Stata Journal, 12 (2), 308-331.
67. Williams, R., &, Bornmann, L., (2014a). Sampling issues in bibliometric analysis. Retrieved from http://arxiv.org/abs/1401.2254
68. Williams, R., &, Bornmann, L., (2014b). The substantive and practical significance of citation impact differences between institutions: Guidelines for the analysis of percentiles using effect sizes and confidence intervals. Retrieved from http://arxiv.org/abs/1404.3720
69. Wouters, P., &, Costas, R., (2012). Users, Narcissism and Control-Tracking the Impact of Scholarly Publications in the 21st Century. Utrecht, The Netherlands: SURFfoundation.
70. Ziman, J., (2000). Real Science. What It Is, and What It Means. Cambridge, UK: Cambridge University Press.