Is interactive open access publishing able to identify high-impact submissions? A study on the predictive validity of Atmospheric Chemistry and Physics by using percentile rank classes

Journal of the American Society for Information Science and Technology

Volumn 62, Issue 1, 2011, Pages 61-71

  Bornmann, Lutz ^a   Schier, Hermann ^b   Marx, Werner ^b   Daniel, Hans Dieter ^c,d  

^a MAX PLANCK INSTITUTE   (Germany)

^b MAX PLANCK INSTITUT FÜR FESTKÖRPERFORSCHUNG   (Germany)

^c ETH ZURICH   (Switzerland)

^d UNIVERSITY OF ZURICH   (Switzerland)

Author keywords

[No Author keywords available]

Indexed keywords

CITATION IMPACT; COMPREHENSIVE RESEARCH; JOURNAL PUBLISHING; OPEN ACCESS; PEER REVIEW SYSTEM; PHYSICAL PROCESS; POPULATION DISTRIBUTION; RESEARCH AREAS; SCIENTIFIC JOURNALS; SELECTION DECISIONS;

PUBLISHING; RESEARCH;

ATMOSPHERIC CHEMISTRY;

EID: 78650077602     PISSN: 15322882     EISSN: 15322890     Source Type: Journal    
DOI: 10.1002/asi.21418     Document Type: Article

References (74)

1. Adler, R., Ewing, J., Taylor, P., &, Hall, P.G., (2009). A report from the International Mathematical Union (IMU) in cooperation with the International Council of Industrial and Applied Mathematics (ICIAM) and the Institute of Mathematical Statistics (IMS). Statistical Science, 24 (1), 1-28.
2. Agresti, A., (2002). Categorical data analysis. Hoboken, NJ: John Wiley & Sons.
3. Aksnes, D.W., (2003). Characteristics of highly cited papers. Research Evaluation, 12 (3), 159-170. (Pubitemid 38355026)
4. Bloom, T., (2006). Systems: Online frontiers of the peer-reviewed literature. The Internet is allowing much more interactive science publishing. Retrieved July 6, 2006, from
5. Bornmann, L., &, Daniel, H.-D., (2008a). The effectiveness of the peer review process: Inter-referee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. Angewandte Chemie International Edition, 47 (38), 7173-7178.
6. Bornmann, L., &, Daniel, H.-D., (2008b). Selecting manuscripts for a high impact journal through peer review: A citation analysis of Communications that were accepted by Angewandte Chemie International Edition, or rejected but published elsewhere. Journal of the American Society for Information Science and Technology, 59 (11), 1841-1852.
7. 0 at the micro level using data from chemistry. Journal of the American Society for Information Science and Technology, 60 (8), 1664-1670.
8. Bornmann, L., &, Daniel, H.-D., (2010a). Citation speed as a measure to predict the attention an article receives: An investigation of the validity of editorial decisions at Angewandte Chemie International Edition. Journal of Informetrics, 4 (1), 83-88.
9. Bornmann, L., &, Daniel, H.-D., (2010b). Reliability of reviewers' ratings at an interactive open access journal using public peer review: A case study on Atmospheric Chemistry and Physics. Learned Publishing, 23 (2), 124-131.
10. Bornmann, L., Marx, W., Schier, H., Thor, A., &, Daniel, H.-D., (2010). From black box to white box at open access journals: predictive validity of manuscript reviewing and editorial decisions at Atmospheric Chemistry and Physics. Research Evaluation, 19 (2), 105-118.
11. Bornmann, L., Mutz, R., &, Daniel, H.-D., (2007). The b-index as a measure of scientific excellence. A promising supplement to the h index. Cybermetrics, 11 (1), paper 6.
12. Bornmann, L., Mutz, R., Marx, W., Schier, H., &, Daniel, H.-D., (conditionally accepted for publication). A multilevel modelling approach to investigating the predictive validity of editorial decisions: Do the editors of a high-impact journal select manuscripts that are highly cited after publication ? Journal of the Royal Statistical Society.
13. Bornmann, L., Mutz, R., Neuhaus, C., &, Daniel, H.-D., (2008). Use of citation counts for research evaluation: Standards of good practice for analyzing bibliometric data and presenting and interpreting results. Ethics in Science and Environmental Politics, 8, 93-102. (Pubitemid 351836496)
14. Bornmann, L., Neuhaus, C., &, Daniel, H.-D., (in press). The effect of a two-stage publication process on the Journal Impact Factor: A case study on the interactive open access journal Atmospheric Chemistry and Physics. Scientometrics.
15. British Academy. (2007). Peer review: The challenges for the humanities and social sciences. London: British Academy.
16. Chemical Abstracts Service. (1997). Subject coverage and arrangement of abstracts by sections in Chemical Abstracts. Columbus, OH: Chemical Abstracts Service (CAS).
17. Cicchetti, D.V., (1991). The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation. Behavioral and Brain Sciences, 14 (1), 119-135.
18. 2 tests. Biometrics, 10 (4), 417-451.
19. Conroy, R.M., (2002). Choosing an appropriate real-life measure of effect size: The case of a continuous predictor and a binary outcome. The Stata Journal, 2 (3), 290-295.
20. Craig, I.D., Plume, A.M., McVeigh, M.E., Pringle, J., &, Amin, M., (2007). Do open access articles have greater citation impact? A critical review of the literature. Journal of Informetrics, 1 (3), 239-248.
21. Cramér, H., (1980). Mathematical methods of statistics. Princeton, NJ: Princeton University Press.
22. Cytel Software Corporation. (2010). StatXact: Version 9. Cambridge, MA: Cytel Software Corporation.
23. Daniel, H.-D., (1993). Guardians of science. Fairness and reliability of peer review. Weinheim, Germany: Wiley-VCH.
24. de Vries, D.R., Marschall, E.A., &, Stein, R.A., (2009). Exploring the peer review process: What is it, does it work, and can it be improved ? Fisheries, 34 (6), 270-279.
25. Evidence Ltd. (2007). The use of bibliometrics to measure research quality in UK higher education institutions. London: Universities UK.
26. Figueredo, E., (2006). The numerical equivalence between the impact factor of journals and the quality of the articles. Journal of the American Society for Information Science and Technology, 57 (11), 1561. (Pubitemid 44351050)
27. Geisler, E., (2000). The metrics of science and technology. Westport, CT: Quorum Books.
28. Giglia, E., (2007). Open access in the biomedical field: A unique opportunity for researchers (and research itself). Europa Medicophyisica, 43 (2), 203-213.
29. Glänzel, W., Schubert, A., &, Czerwon, H.J., (1999). An item-by-item subject classification of papers published in multidisciplinary and general journals using reference analysis. Scientometrics, 44 (3), 427-439.
30. Glänzel, W., Thijs, B., Schubert, A., &, Debackere, K., (2009). Subfield-specific normalized relative indicators and a new generation of relational charts: Methodological foundations illustrated on the assessment of institutional research performance. Scientometrics, 78 (1), 165-188.
31. Gölitz, P., (2010). Twitter, Facebook, and Open Access. Angewandte Chemie International Edition, 49 (1), 4-6.
32. Grant, B., (2009). Merck published fake journal. Retrieved March 25, 2010, from
33. Hames, I., (2007). Peer review and manuscript management of scientific journals: Guidelines for good practice. Oxford, United Kingdom: Blackwell.
34. Hansson, F., (2006). Research evaluation and organisational learning in the university: A possible coexistence ? International Journal of Learning and Change, 1 (3), 267-284.
35. Jackson, C., (1996). Understanding psychological testing. Leicester, United Kingdom: British Psychological Society.
36. Joint Committee on Quantitative Assessment of Research. (2008). Citation statistics. A report from the International Mathematical Union (IMU) in cooperation with the International Council of Industrial and Applied Mathematics (ICIAM) and the Institute of Mathematical Statistics (IMS). Berlin, Germany: International Mathematical Union (IMU).
37. Joint Information Systems Committee. (2004). Journal authors survey report. Truro, United Kingdom: Key Perspectives Ltd.
38. Kline, R.B., (2004). Beyond significance testing: Reforming data analysis methods in behavioral research. Washington, DC: American Psychological Association.
39. Koop, T., &, Pöschl, U., (2006). Systems: An open, two-stage peer-review journal. The editors of Atmospheric Chemistry and Physics explain their journal's approach. Retrieved 26 June 2006, from
40. Kostoff, R.N., (2002). Citation analysis of research performer quality. Scientometrics, 53 (1), 49-71.
41. Kurtz, M.J., &, Henneken, E.A., (2007). Open Access does not increase citations for research articles from The Astrophysical Journal. Retrieved September 10, 2007, from
42. Lewison, G., Thornicroft, G., Szmukler, G., &, Tansella, M., (2007). Fair assessment of the merits of psychiatric research. British Journal of Psychiatry, 190, 314-318. (Pubitemid 46580680)
43. Lock, S., (1985). A difficult balance: Editorial peer review in medicine. Philadelphia: ISI Press.
44. McCormack, N., (2009). Peer review and legal publishing: What law librarians need to know about open, single-blind, and double-blind reviewing. [Review]. Law Library Journal, 101 (1), 59-70.
45. McDonald, R.J., Cloft, H.J., &, Kallmes, D.F., (2009). Fate of manuscripts previously rejected by the American Journal of Neuroradiology: A follow-up analysis. American Journal of Neuroradiology, 30 (2), 253-256.
46. National Science Board. (2008). Science and Engineering Indicators 2008 (NSB 08-01A) (Vol. 2). Arlington, VA: National Science Foundation (NSF).
47. Neuhaus, C., &, Daniel, H.-D., (2009). A new reference standard for citation analysis in chemistry and related fields based on the sections of Chemical Abstracts. Scientometrics, 78 (2), 219-229.
48. Neuhaus, C., &, Daniel, H.D., (2010). How good is research ? Chemistry World, 7 (3), 42.
49. Neuhaus, C., Marx, W., &, Daniel, H.-D., (2009). The publication and citation impact profiles of Angewandte Chemie and the Journal of the American Chemical Society based on the sections of Chemical Abstracts: Aâcase study on the limitations of the Journal Impact Factor. Journal of the American Society for Information Science and Technology, 60 (1), 176-183.
50. Opthof, T., Furstner, F., van Geer, M., &, Coronel, R., (2000). Regrets or no regrets? No regrets! The fate of rejected manuscripts. Cardiovascular Research, 45 (1), 255-258.
51. Pendlebury, D.A., (2008). Using bibliometrics in evaluating research. Philadelphia: Research Department, Thomson Scientific.
52. Plomp, R., (1990). The significance of the number of highly cited papers as an indicator of scientific prolificacy. Scientometrics, 19 (3-4), 185-197.
53. Pöschl, U., (2004). Interactive journal concept for improved scientific publishing and quality assurance. Learned Publishing, 17 (2), 105-113. (Pubitemid 38516671)
54. Pöschl, U., (2010). Interactive open access publishing and peer review: The effectiveness and perspectives of transparency and self-regulation in scientific communication and evaluation. Liber Quarterly, 19 (3/4), 293-314.
55. Publishing Research Consortium. (2008). Peer review in scholarly journals: perspective of the scholarly communityâan international study. Bristol, United Kingdom: Publishing Research Consortium.
56. Radicchi, F., Fortunato, S., &, Castellano, C., (2008). Universality of citation distributions: Toward an objective measure of scientific impact. Proceedings of the National Academy of Sciences of the United States of America, 105 (45), 17268-17272.
57. Research Evaluation and Policy Project. (2005). Quantitative indicators for research assessment-A literature review (REPP discussion paper 05/1). Canberra, Australia: Research Evaluation and Policy Project, Research School of Social Sciences, The Australian National University.
58. Rousseau, R., (2005). Median and percentile impact factors: A set of new indicators. Scientometrics, 63 (3), 431-441. (Pubitemid 41201773)
59. Rowland, F., (2002). The peer-review process. Learned Publishing, 15 (4), 247-258.
60. Schubert, A., &, Braun, T., (1996). Cross-field normalization of scientometric indicators. Scientometrics, 36 (3), 311-324. (Pubitemid 126449886)
61. Schultz, D.M., (2010). Rejection rates for journals publishing atmospheric science. Bulletin of the American Meteorological Society, 91 (2), 231-243.
62. Shatz, D., (2004). Peer review: A critical inquiry. Lanham, MD: Rowman & Littlefield.
63. Sheskin, D., (2007). Handbook of parametric and nonparametric statistical procedures (4th ed.). Boca Raton, FL: Chapman & Hall/CRC.
64. Taylor, M., Perakakis, P., &, Trachana, V., (2008). The siege of science. Ethics in Science and Environmental Politics, 8, 17-40. (Pubitemid 351796588)
65. Thorngate, W., Dawes, R.M., &, Foddy, M., (2009). Judging merit. New York: Psychology Press.
66. Tijssen, R., &, van Leeuwen, T., (2006). Centres of research excellence and science indicators. Can 'excellence' be captured in numbers ? In, W. Glänzel, (Ed.), Ninth International Conference on Science and Technology Indicators (pp. 146-147). Leuven, Belgium: Katholieke Universiteit Leuven.
67. Tijssen, R., Visser, M., &, van Leeuwen, T., (2002). Benchmarking international scientific excellence: Are highly cited research papers an appropriate frame of reference ? Scientometrics, 54 (3), 381-397.
68. van Leeuwen, T.N., (2007). Modelling of bibliometric approaches and importance of output verification in research performance assessment. Research Evaluation, 16 (2), 93-105.
69. van Raan, A.F.J., (1999). Advanced bibliometric methods for the evaluation of universities. Scientometrics, 45 (3), 417-423.
70. van Raan, A.F.J., (2004). Measuring science. Capita selecta of current main issues. In, H.F. Moed, W. Glänzel, &, U. Schmoch, (Eds.), Handbook of quantitative science and technology research. The use of publication and patent statistics in studies of S&T systems (pp. 19-50). Dordrecht, The Netherlands: Kluwer Academic Publishers.
71. van Rooyen, S., (2001). The evaluation of peer-review quality. Learned Publishing, 14 (2), 85-91. (Pubitemid 33692299)
72. Vinkler, P., (1986). Evaluation of some methods for the relative assessment of scientific publications. Scientometrics, 10 (3-4), 157-177.
73. Vinkler, P., (1997). Relations of relative scientometric impact indicators. The relative publication strategy index. Scientometrics, 40 (1), 163-169. (Pubitemid 127457607)
74. Wilson, J.D., (1978). Peer review and publication. Journal of Clinical Investigation, 61 (4), 1697-1701.