A multi-disciplinary perspective on emergent and future innovations in peer review

F1000Research

Volumn 6, Issue , 2017, Pages

  Tennant, Jonathan P ^a,b   Dugan, Jonathan M ^c   Graziotin, Daniel ^d   Jacques, Damien C ^e   Waldner, François ^e   Mietchen, Daniel ^f   Elkhatib, Yehia ^g   B Collister, Lauren ^h   Pikas, Christina K ⁱ   Crick, Tom ^j   Masuzzo, Paola ^k   Caravaggi, Anthony ^l   Berg, Devin R ^m   Niemeyer, Kyle E ⁿ   Ross Hellauer, Tony ^o   Mannheimer, Sara ^p   Rigling, Lillian ^q   Katz, Daniel S ^r,s,t   Greshake Tzovaras, Bastian ^u   Pacheco Mendoza, Josmel ^v   Fatima, Nazeefa ^w   Poblet, Marta ^x   Isaakidis, Marios ^y   Irawan, Dasapta Erwin ^z   Renaut, Sébastien ^aa   Madan, Christopher R ^ab   Matthias, Lisa ^o   Nørgaard Kjær, Jesper ^ac   O'Donnell, Daniel Paul ^ad   Neylon, Cameron ^ae   Kearns, Sarah ^af   Selvaraju, Manojkumar ^ag,ah   Colomb, Julien ^ai   more..

^a IMPERIAL COLLEGE LONDON   (United Kingdom)

^b ScienceOpen   (Germany)

^c UNIVERSITY OF CALIFORNIA   (United States)

^d UNIVERSITY OF STUTTGART   (Germany)

^e UNIVERSITÉ CATHOLIQUE DE LOUVAIN   (Belgium)

^f UNIVERSITY OF VIRGINIA   (United States)

^g LANCASTER UNIVERSITY   (United Kingdom)

^h UNIVERSITY OF PITTSBURGH   (United States)

ⁱ JOHNS HOPKINS UNIVERSITY APPLIED PHYSICS LABORATORY   (United States)

^j CARDIFF METROPOLITAN UNIVERSITY   (United Kingdom)

^k GHENT UNIVERSITY   (Belgium)

^l UNIVERSITY COLLEGE CORK   (Ireland)

^m UNIVERSITY OF WISCONSIN STOUT   (United States)

ⁿ Oregon State University   (United States)

^o UNIVERSITY OF GÖTTINGEN   (Germany)

^p MONTANA STATE UNIVERSITY   (United States)

^q UNIVERSITY OF WESTERN ONTARIO   (Canada)

^r UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

^s University of Illinois at Urbana Champaign ^*  (United States)

^t University of Illinois at Urbana Champaign   (United States)

^u GOETHE UNIVERSITY   (Germany)

^v UNIVERSIDAD SAN IGNACIO DE LOYOLA   (Peru)

^w LUND UNIVERSITY   (Sweden)

^x RMIT UNIVERSITY   (Australia)

^y UNIVERSITY COLLEGE LONDON   (United Kingdom)

^z BANDUNG INSTITUTE OF TECHNOLOGY   (Indonesia)

^aa UNIVERSITÉ DE MONTRÉAL   (Canada)

^ab UNIVERSITY OF NOTTINGHAM   (United Kingdom)

^ac AARHUS UNIVERSITY HOSPITAL   (Denmark)

^ad UNIVERSITY OF LETHBRIDGE   (Canada)

^ae CURTIN UNIVERSITY   (Australia)

^af UNIVERSITY OF MICHIGAN   (United States)

^ag KING ABDULAZIZ CITY FOR SCIENCE AND TECHNOLOGY   (Saudi Arabia)

^ah INTEGRATED GULF BIOSYSTEMS   (Saudi Arabia)

^ai Independent Researcher   (Germany)

more..

Author keywords

Incentives; Open Peer Review; Open Science; Quality Control; Scholarly Publishing; Social Media; Web 2.0

Indexed keywords

CERTIFICATION; CONTROLLED STUDY; HUMAN; PEER REVIEW; PUBLISHING; QUALITY CONTROL; SOCIAL MEDIA;

EID: 85028984570     PISSN: 20461402     EISSN: 1759796X     Source Type: Journal    
DOI: 10.12688/f1000research.12037.3     Document Type: Review

References (294)

1. Adam D: Climate scientists hit out at 'sloppy' melting glaciers error. The Guardian. 2010; Accessed: 2017-06-12.
2. Al-Rahawi IbA: Practical Ethics of the Physician (Adab al-Tabib). c900.
3. Albert AY, Gow JL, Cobra A, et al.: Is it becoming harder to secure reviewers for peer review? a test with data from five ecology journals. Research Integrity and Peer Review. 2016; 1(1): 14.
4. Almquist M, von Allmen RS, Carradice D, et al.: A prospective study on an innovative online forum for peer reviewing of surgical science. PLoS One. 2017; 12(6): e0179031.
5. Alvesson M, Sandberg J: Habitat and habitus: Boxed-in versus box-breaking research. Organization Studies. 2014; 35(7): 967-987.
6. Anderson S, McPherson T: Engaging digital scholarship: Thoughts on evaluating multimedia scholarship. Profession. 2011; 136-151.
7. Antonopoulos AM: Mastering Bitcoin: unlocking digital cryptocurrencies. "O'Reilly Media, Inc". 2014.
8. Armstrong JS: Peer review for journals: Evidence on quality control, fairness, and innovation. Sci Eng Ethics. 1997; 3(1): 63-84.
9. arXiv: arXiv monthly submission rates. 2017; Date accessed: 2017-06-12.
10. Baggs JG, Broome ME, Dougherty MC, et al.: Blinding in peer review: the preferences of reviewers for nursing journals. J Adv Nurs. 2008; 64(2): 131-138.
11. Baldwin M: Credibility, peer review, and Nature, 1945-1990. Notes Rec R Soc Lond. 2015; 69(3): 337-352.
12. Baldwin M: In referees we trust? Phys Today. 2017a; 70(2): 44-49.
13. Baldwin M: What it was like to be peer reviewed in the 1860s. Phys Today. 2017b.
14. Ball CE, Lamanna CA, Saper C, et al.: Annotated bibliography on evaluating digital scholarship for tenure and promotion. Stasis: A Kairos Archive. 2016.
15. Bartling S, Fecher B: Blockchain for science and knowledge creation. Zenodo. 2016.
16. Baxt WG, Waeckerle JF, Berlin JA, et al.: Who reviews the reviewers? Feasibility of using a fictitious manuscript to evaluate peer reviewer performance. Ann Emerg Med. 1998; 32(3 Pt 1): 310-317.
17. Bedeian AG: The manuscript review process the proper roles of authors, referees, and editors. J Manage Inquiry. 2003; 12(4): 331-338.
18. Begel A, Bosch J, Storey MA: Social networking meets software development: Perspectives from GitHub, MSDN, Stack Exchange, and TopCoder. IEEE Software. 2013; 30(1): 52-66.
19. Belojevic N, Sayers J, INKE and MVP Research Teams: Peer review personas. Journal of Electronic Publishing. 2014; 17(3).
20. Benda WG, Engels TC: The predictive validity of peer review: A selective review of the judgmental forecasting qualities of peers, and implications for innovation in science. Int J Forecasting. 2011; 27(1): 166-182.
21. Bernstein R: Updated: Sexist peer review elicits furious twitter response, PLOS apology. Science. 2015.
22. Berthaud C, Capelli L, Gustedt J, et al.: EPISCIENCES-an overlay publication platform. Information Services Use. 2014; 34(3-4): 269-277.
23. Biagioli M: From book censorship to academic peer review. Emergences: Journal for the Study of Media & Composite Cultures. 2002; 12(1): 11-45.
24. BioMed Central: What might peer review look like in 2030? figshare. 2017.
25. Black EW: Wikipedia and academic peer review: Wikipedia as a recognised medium for scholarly publication? Online Inform Rev. 2008; 32(1): 73-88.
26. Blank RM: The effects of double-blind versus single-blind reviewing: Experimental evidence from the american economic review. Am Econ Rev. 1991; 81(5): 1041-1067.
27. Blatt MR: Vigilante Science. Plant Physiol. 2015; 169(2): 907-909.
28. Boldt A: Extending ArXiv.org to achieve open peer review and publishing. J Scholarly Publ. 2011; 42(2): 238-242.
29. Bon M, Taylor M, McDowell GS: Novel processes and metrics for a scientific evaluation rooted in the principles of science-Version 1. arXiv: 1701.08008 [cs. DL]. 2017.
30. Bornmann L, Daniel HD: How long is the peer review process for journal manuscripts? A case study on Angewandte Chemie International Edition. Chimia (Aarau). 2010a; 64(1-2): 72-77.
31. Bornmann L, Daniel HD: Reliability of reviewers' ratings when using public peer review: a case study. Learn Publ. 2010b; 23(2): 124-131.
32. Bornmann L, Mutz R: Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references. J Assoc Inf Sci Technol. 2015; 66(11): 2215-2222.
33. Bornmann L, Wolf M, Daniel HD: Closed versus open reviewing of journal manuscripts: how far do comments differ in language use? Scientometrics. 2012; 91(3): 843-856.
34. Brembs B: The cost of the rejection-resubmission cycle. The Winnower. 2015.
35. Brembs B, Button K, Munafò M: Deep impact: unintended consequences of journal rank. Front Hum Neurosci. 2013; 7: 291.
36. Breuning M, Backstrom J, Brannon J, et al.: Reviewer fatigue? why scholars decline to review their peers' work. PS: Polit Sci Polit. 2015; 48(4): 595-600.
37. Bruce R, Chauvin A, Trinquart L, et al.: Impact of interventions to improve the quality of peer review of biomedical journals: a systematic review and meta-analysis. BMC Med. 2016; 14(1): 85.
38. Budden AE, Tregenza T, Aarssen LW, et al.: Double-blind review favours increased representation of female authors. Trends Ecol Evol. 2008; 23(1): 4-6.
39. Burghardt K, Alsina EF, Girvan M, et al.: The myopia of crowds: Cognitive load and collective evaluation of answers on stack exchange. PLoS One. 2017; 12(3): e0173610.
40. Burnham JC: The evolution of editorial peer review. JAMA. 1990; 263(10): 1323-1329.
41. Burris V: The academic caste system: Prestige hierarchies in PhD exchange networks. Am Sociol Rev. 2004; 69(2): 239-264.
42. Campanario JM: Peer review for journals as it stands today-part 1. Sci Commun. 1998a; 19(3): 181-211.
43. Campanario JM: Peer review for journals as it stands today-part 2. Sci Commun. 1998b; 19(4): 277-306.
44. Carlisle BG: Proof of prespecified endpoints in medical research with the bitcoin blockchain. The Grey Literature. 2014; Date accessed: 2017-06-12.
45. Casnici N, Grimaldo F, Gilbert N, et al.: Attitudes of referees in a multidisciplinary journal: An empirical analysis. J Assoc Inf Sci Technol. 2017; 68(7): 1763-1771.
46. Chambers CD, Feredoes E, Muthukumaraswamy SD, et al.: Instead of "playing the game" it is time to change the rules: Registered Reports at AIMS Neuroscience and beyond. AIMS Neurosci. 2014; 1(1): 4-17.
47. Chambers CD, Forstmann B, Pruszynski JA: Registered reports at the European Journal of Neuroscience: consolidating and extending peer-reviewed study pre-registration. Eur J Neurosci. 2017; 45(5): 627-628.
48. Chauvin A, Ravaud P, Baron G, et al.: The most important tasks for peer reviewers evaluating a randomized controlled trial are not congruent with the tasks most often requested by journal editors. BMC Med. 2015; 13(1): 158.
49. Chevalier JA, Mayzlin D: The effect of word of mouth on sales: Online book reviews. J Mark Res. 2006; 43(3): 345-354.
50. Cohen BA: How should novelty be valued in science? eLife. 2017; 6: pii: e28699.
51. Cole S: The role of journals in the growth of scientific knowledge. In Cronin B, and Atkins HB, editors, The Web of Knowledge: A Festschrift in Honor of Eugene Garfield. ASIS Monograph Series, chapter 6, Information Today, Inc., Medford, NJ. 2000; 109-142.
52. Cope B, Kalantzis M: Signs of epistemic disruption: Transformations in the knowledge system of the academic journal. The Future of the Academic Journal. Oxford: Chandos Publishing, 2009; 14(2): 13-61.
53. Crotty D: How meaningful are user ratings? (this article = 4.5 stars!). The Scholarly Kitchen. 2009; Date accessed: 2017-06-12.
54. Csiszar A: Peer review: Troubled from the start. Nature. 2016; 532(7599): 306-8.
55. Dall'Aglio P: Peer review and journal models. arXiv:physics/0608307. [physics. soc-ph]. 2006.
56. D'Andrea R, O'Dwyer JP: Can editors protect peer review from bad reviewers? PeerJ Preprints. 2017; 5: e3005v1.
57. Daniel HD: Guardians of science: fairness and reliability of peer review. VCH, Weinheim, Germany. 1993.
58. Dappert A, Farquhar A, Kotarski R, et al.: Connecting the persistent identifier ecosystem: Building the technical and human infrastructure for open research. Data Sci J. 2017; 16: 28.
59. Darling ES: Use of double-blind peer review to increase author diversity. Conserv Biol. 2015; 29(1): 297-299.
60. Davis P: Wither portable peer review. The Scholarly Kitchen. 2017; Accessed: 2017-06-12.
61. Davis P, Fromerth M: Does the arXiv lead to higher citations and reduced publisher downloads for mathematics articles? Scientometrics. 2007; 71(2): 203-215.
62. Dhillon V: From bench to bedside: Enabling reproducible commercial science via blockchain. Bitcoin Magazine. 2016; Date accessed: 2017-06-12.
63. Eckberg DL: When nonreliability of reviews indicates solid science. Behav Brain Sci. 1991; 14(1): 145-146.
64. Edgar B, Willinsky J: A survey of scholarly journals using open journal systems. Scholarly Res Commun. 2010; 1(2).
65. Eisen M: Peer review is f***ed up-let's fix it. 2011; Accessed: 2017-03-15.
66. Elkhatib Y, Tyson G, Sathiaseelan A: Does the Internet deserve everybody? In Proceedings of the 2015 ACM SIGCOMM Workshop on Ethics in Networked Systems Research. ACM, 2015; 5-8. ISBN: 978-1-4503-3541-6.
67. Emilsson R: The influence of the Internet on identity creation and extreme groups. Bachelor Thesis, Blekinge Institute of Technology, Department of Technology and Aesthetics. 2015.
68. Ernst E, Kienbacher T: Chauvinism. Nature. 1991; 352: 560.
69. Farley T: Hypothes.is reaches funding goal. James Randi Educational Foundation Swift Blog. 2011; Date accessed: 2017-06-12.
70. Fitzpatrick K: Peer-to-peer review and the future of scholarly authority. Soc Epistemol. 2010; 24(3): 161-179.
71. Fitzpatrick K: Peer review, judgment, and reading. Profession. 2011a; 196-201.
72. Fitzpatrick K: Planned Obsolescence. New York University Press, New York, 2011b. ISBN: 978-0-8147-2788-1-0-8147-2788-3.
73. Ford E: Defining and characterizing open peer review: A review of the literature. J Scholarly Publ. 2013; 44(4): 311-326.
74. Fox CW, Albert AY, Vines TH: Recruitment of reviewers is becoming harder at some journals: a test of the influence of reviewer fatigue at six journals in ecology and evolution. Res Integr Peer Rev. 2017; 2(1): 3.
75. Fox MF: Scientific misconduct and editorial and peer review processes. J Higher Educ. 1994; 65(3): 298-309.
76. Frishauf P: Reputation systems: a new vision for publishing and peer review. J Participat Med. 2009; 1(1): e13a.
77. Fukuzawa N: Characteristics of papers published in journals: an analysis of open access journals, country of publication, and languages used. Scientometrics. 2017; 112(2): 1007-1023. ISSN: 1588-2861.
78. Fyfe A, Coate K, Curry S, et al.: Untangling Academic Publishing: A history of the relationship between commercial interests, academic prestige and the circulation of research. Zenodo. 2017.
79. Galipeau J, Moher D, Campbell C, et al.: A systematic review highlights a knowledge gap regarding the effectiveness of health-related training programs in journalology. J Clin Epidemiol. 2015; 68(3): 257-65.
80. Gashler M: GPeerReview-a tool for making digital-signatures using data mining. KDnuggets. 2008; Date accessed: 2017-06-12.
81. Ghosh SS, Klein A, Avants B, et al.: Learning from open source software projects to improve scientific review. Front Comput Neurosci. 2012; 6: 18.
82. Gibney E: Toolbox: Low-cost journals piggyback on arXiv. Nature. 2016; 530(7588): 117-118.
83. Gibson M, Spong CY, Simonsen SE, et al.: Author perception of peer review. Obstet Gynecol. 2008; 112(3): 646-652.
84. Ginsparg P: Winners and losers in the global research village. Ser Libr. 1997; 30(3-4): 83-95.
85. Godlee F: Making reviewers visible: openness, accountability, and credit. JAMA. 2002; 287(21): 2762-2765.
86. Godlee F, Gale CR, Martyn CN: Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. JAMA. 1998; 280(3): 237-240.
87. Goodman SN, Berlin J, Fletcher SW, et al.: Manuscript quality before and after peer review and editing at Annals of Internal Medicine. Ann Intern Med. 1994; 121(1): 11-21.
88. Gøtzsche PC: Methodology and overt and hidden bias in reports of 196 doubleblind trials of nonsteroidal antiinflammatory drugs in rheumatoid arthritis. Control Clin Trials. 1989; 10(1): 31-56.
89. Goues CL, Brun Y, Apel S, et al.: Effectiveness of anonymization in double-blind review. 2017.
90. Graf K: Fetisch peer review. Archivalia. In German. 2014. Accessed: 2017-03-15.
91. Greaves S, Scott J, Clarke M, et al.: Overview: Nature's peer review trial. Nature. 2006.
92. Greenberg SA: How citation distortions create unfounded authority: analysis of a citation network. BMJ. 2009; 339: b2680.
93. Grivell L: Through a glass darkly: The present and the future of editorial peer review. EMBO Rep. 2006; 7(6): 567-570.
94. Gropp RE, Glisson S, Gallo S, et al.: Peer review: A system under stress. BioScience. 2017; 67(5): 407-410.
95. Gupta S: How has publishing changed in the last twenty years? Notes Rec R Soc J Hist Sci. 2016; 70(4): 391-392.
96. Haider J, Åström F: Dimensions of trust in scholarly communication: Problematizing peer review in the aftermath of John Bohannon's "sting" in science. J Assoc Inf Sci Technol. 2017; 68(2): 450-467.
97. Halavais A, Kwon KH, Havener S, et al.: Badges of friendship: Social influence and badge acquisition on stack overflow. In System Sciences (HICSS), 2014 47th Hawaii International Conference on. IEEE. 2014; 1607-1615.
98. Harmon AH, Metaxas PT: How to create a smart mob: Understanding a social network capital. In Krishnamurthy S, Singh G, and McPherson M, editors, Proceedings of the IADIS International Conference on e-Democracy, Equity and Social Justice. International Association for Development of the Information Society, 2010. ISBN: 978-972-8939-24-3.
99. Hasty RT, Garvalosa RC, Barbato VA, et al.: Wikipedia vs peer-reviewed medical literature for information about the 10 most costly medical conditions. J Am Osteopath Assoc. 2014; 114(5): 368-373.
100. Haug CJ: Peer-Review Fraud--Hacking the Scientific Publication Process. N Engl J Med. 2015; 373(25): 2393-2395.
101. Heaberlin B, DeDeo S: The evolution of wikipedia's norm network. Future Internet. 2016; 8(2): 14.
102. Heller L, The R, Bartling S: Dynamic Publication Formats and Collaborative Authoring. Opening Science. Springer International Publishing, Cham, 2014; 191-211. ISBN: 978-3-319-00026-8.
103. Helmer M, Schottdorf M, Neef A, et al.: Gender bias in scholarly peer review. eLife. 2017; 6: pii: e21718.
104. Hettyey A, Griggio M, Mann M, et al.: Peerage of Science: will it work? Trends Ecol Evol. 2012; 27(4): 189-190.
105. Horrobin DF: The philosophical basis of peer review and the suppression of innovation. JAMA. 1990; 263(10): 1438-1441.
106. Hu M, Lim EP, Sun A, et al.: Measuring article quality in Wikipedia: models and evaluation. In Proceedings of the Sixteenth ACM Conference on Information and Knowledge Management. ACM. 2007; 243-252.
107. Hukkinen JI: Peer review has its shortcomings, but AI is a risky fix. Wired. 2017. Accessed: 2017-01-31.
108. Ioannidis JP: Why most published research findings are false. PLoS Med. 2005; 2(8): e124.
109. Isenberg SJ, Sanchez E, Zafran KC: The effect of masking manuscripts for the peer-review process of an ophthalmic journal. Br J Ophthalmol. 2009; 93(7): 881-884.
110. Jadad AR, Moore RA, Carroll D, et al.: Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials. 1996; 17(1): 1-12.
111. Janowicz K, Hitzler P: Open and transparent: the review process of the semantic web journal. Learn Publ. 2012; 25(1): 48-55.
112. Jefferson T, Rudin M, Brodney Folse S, et al.: Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database Syst Rev. 2007; (2): MR000016.
113. Jefferson T, Wager E, Davidoff F: Measuring the quality of editorial peer review. JAMA. 2002; 287(21): 2786-2790.
114. Jubb M: Peer review: The current landscape and future trends. Learn Publ. 2016; 29(1): 13-21.
115. Justice AC, Cho MK, Winker MA, et al.: Does masking author identity improve peer review quality? A randomized controlled trial. PEER Investigators. JAMA. 1998; 280(3): 240-242.
116. Katsh E, Rule C: What we know and need to know about online dispute resolution. SCL Rev. 2015; 67: 329.
117. Kelty CM, Burrus CS, Baraniuk RG: Peer review anew: Three principles and a case study in postpublication quality assurance. Proc IEEE. 2008; 96(6): 1000-1011.
118. Khan MS: Exploring citations for conflict of interest detection in peer review system. IJCISIM. 2012; 4: 283-299.
119. Klyne G: Peer review #2 of "software citation principles (v0.1)". PeerJ Comput Sci. 2016.
120. Kosner AW: GitHub is the next big social network, powered by what you do, not who you know. Forbes. 2012.
121. Kostoff RN: Federal research impact assessment: Axioms, approaches, applications. Scientometrics. 1995; 34(2): 163-206.
122. Kovanis M, Porcher R, Ravaud P, et al.: The Global Burden of Journal Peer Review in the Biomedical Literature: Strong Imbalance in the Collective Enterprise. PLoS One. 2016; 11(11): e0166387.
123. Kriegeskorte N, Walther A, Deca D: An emerging consensus for open evaluation: 18 visions for the future of scientific publishing. Front Comput Neurosci. 2012; 6: 94, ISSN: 1662-5188.
124. Kronick DA: Peer review in 18th-century scientific journalism. JAMA. 1990; 263(10): 1321-1322.
125. Kubátová J, et al.: Growth of collective intelligence by linking knowledge workers through social media. Lex ET Scientia International Journal (LESIJ). 2012; (XIX-1): 135-145.
126. Kuehn BM: Rooting out bias. eLife. 2017; 6: pii: e32014.
127. Kuhn T: Peer review #1 of "software citation principles (v0.1)". PeerJ Comput Sci. 2016a.
128. Kuhn T: Peer review #1 of "software citation principles (v0.2)". PeerJ Comput Sci. 2016b.
129. Larivière V, Haustein S, Mongeon P: The Oligopoly of Academic Publishers in the Digital Era. PLoS One. 2015; 10(6): e0127502.
130. Larivière V, Sugimoto CR, Macaluso B, et al.: arxiv e-prints and the journal of record: An analysis of roles and relationships. J Assoc Inf Sci Technol. 2014; 65(6): 1157-1169.
131. Larsen PO, von Ins M: The rate of growth in scientific publication and the decline in coverage provided by Science Citation index. Scientometrics. 2010; 84(3): 575-603.
132. Lee CJ, Moher D: Promote scientific integrity via journal peer review data. Science. 2017; 357(6348): 256-257.
133. Lee CJ, Sugimoto CR, Zhang G, et al.: Bias in peer review. J Am Soc Inf Sci Technol. 2013; 64(1): 2-17.
134. Lee D: The new Reddit journal of science. IMM-press Magazine. 2015. Date accessed: 2016-06-12.
135. Leek JT, Taub MA, Pineda FJ: Cooperation between referees and authors increases peer review accuracy. PLoS One. 2011; 6(11): e26895.
136. Lerback J, Hanson B: Journals invite too few women to referee. Nature. 2017; 541(7638): 455-457.
137. Li L, Steckelberg AL, Srinivasan S: Utilizing peer interactions to promote learning through a web-based peer assessment system. Can J Learn Technol. 2009; 34(2).
138. Link AM: US and non-US submissions: an analysis of reviewer bias. JAMA. 1998; 280(3): 246-247.
139. Lipworth W, Kerridge I: Shifting power relations and the ethics of journal peer review. Soc Epistemol. 2011; 25(1): 97-121.
140. Lipworth W, Kerridge IH, Carter SM, et al.: Should biomedical publishing be "opened up"? toward a values-based peer-review process. J Bioeth Inq. 2011; 8(3): 267-280.
141. List B: Crowd-based peer review can be good and fast. Nature. 2017; 546(7656): 9.
142. Lloyd ME: Gender factors in reviewer recommendations for manuscript publication. J Appl Behav Anal. 1990; 23(4): 539-543.
143. Lui KM, Chan KC: Pair programming productivity: Novice-novice vs. expertexpert. Int J Hum Comput Stud. 2006; 64(9): 915-925.
144. Luzi D: Trends and evolution in the development of grey literature: a review. Int J Grey Lit. 2000; 1(3): 106-117.
145. Lyman RL: A three-decade history of the duration of peer review. J Scholarly Publ. 2013; 44(3): 211-220.
146. Magee JC, Galinsky AD: 8 social hierarchy: The self-reinforcing nature of power and status. Acad Manag Ann. 2008; 2(1): 351-398.
147. Maharg P, Duncan N: Black box, pandora's box or virtual toolbox? an experiment in a journal's transparent peer review on the web. Int Rev Law Comp Technol. 2007; 21(2): 109-128.
148. Mahoney MJ: Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognit Ther Res. 1977; 1(2): 161-175.
149. Manten AA: Development of european scientific journal publishing before 1850. Development of science publishing in Europe. 1980; 1-22.
150. Margalida A, Colomer Mà: Improving the peer-review process and editorial quality: key errors escaping the review and editorial process in top scientific journals. PeerJ. 2016; 4: e1670.
151. Marra M: Arxiv-based commenting resources by and for astrophysicists and physicists: An initial survey. Expanding Perspectives on Open Science: Communities, Cultures and Diversity in Concepts and Practices. 2017; 100-117.
152. Mayden KD: Peer Review: Publication's Gold Standard. J Adv Pract Oncol. 2012; 3(2): 117-22.
153. McCormack N: Peer review and legal publishing: What law librarians need to know about open, single-blind, and double-blind reviewing. Law Libr J. 2009; 101: 59.
154. McKiernan EC, Bourne PE, Brown CT, et al.: How open science helps researchers succeed. eLife. 2016; 5: pii: e16800.
155. McKiernan G: Alternative peer review: Quality management for 21st century scholarship. Talk presented at the Workshop on Peer Review in the Age of Open Archives, in Trieste, Italy, 2003.
156. McNutt RA, Evans AT, Fletcher RH, et al.: The effects of blinding on the quality of peer review. A randomized trial. JAMA. 1990; 263(10): 1371-1376.
157. Melero R, Lopez-Santovena F: Referees' attitudes toward open peer review and electronic transmission of papers. Food Sci Technol Int. 2001; 7(6): 521-527.
158. Merton RK: The Matthew Effect in Science: The reward and communication systems of science are considered. Science. 1968; 159(3810): 56-63.
159. Merton RK: The sociology of science: Theoretical and empirical investigations. University of Chicago Press, Chicago. 1973.
160. Mhurchú AN, McLeod L, Collins S, et al.: The present and the future of the research excellence framework impact agenda in the UK academy: A reflection from politics and international studies. Political Stud Rev. 2017; 15(1): 60-72.
161. Mietchen D: Referee report for: Sharing individual patient and parasite-level data through the worldwide antimalarial resistance network platform: A qualitative case study [version 1; referees: 2 approved]. Wellcome Open Res. 2017.
162. Moed HF: The effect of "open access" on citation impact: An analysis of ArXiv's condensed matter section. J Am Soc Inf Sci Technol. 2007; 58(13): 2047-2054.
163. Moher D, Galipeau J, Alam S, et al.: Core competencies for scientific editors of biomedical journals: consensus statement. BMC Med. 2017; 15(1): 167. ISSN: 1741-7015.
164. Moore S, Neylon C, Eve MP, et al.: "excellence R Us": university research and the fetishisation of excellence. Palgrave Commun. 2017; 3: 16105.
165. Morrison J: The case for open peer review. Med Educ. 2006; 40(9): 830-831.
166. Moxham N, Fyfe A: The royal society and the prehistory of peer review, 1665-1965. Hist J. 2017.
167. Mudambi SM, Schuff D: What makes a helpful review? A study of customer reviews on Amazon.com. MIS Quarterly. 2010; 34(1): 185-200.
168. Mulligan A, Akerman R, Granier B, et al.: Quality, certification and peer review. Inf Serv Use. 2008; 28(3-4): 197-214.
169. Mulligan A, Hall L, Raphael E: Peer review in a changing world: An international study measuring the attitudes of researchers. J Am Soc Inf Sci Technol. 2013; 64(1): 132-161.
170. Munafò MR, Nosek BA, Bishop DV, et al.: A manifesto for reproducible science. Nat Hum Behav. 2017; 1: 0021.
171. Murphy T, Sage D: Perceptions of the UK's research excellence framework 2014: a media analysis. J Higher Educ Pol Manage. 2014; 36(6): 603-615.
172. Nakamoto S: Bitcoin: A peer-to-peer electronic cash system. 2008.
173. Nature: Response required. Nature. 2010; 468(7326): 867.
174. Nature Human Behaviour: Promoting reproducibility with registered reports. Nat Hum Behav. 2017; 1: 0034.
175. Neylon C, Wu S: Article-level metrics and the evolution of scientific impact. PLoS Biol. 2009; 7(11): e1000242.
176. Nicholson J, Alperin JP: A brief survey on peer review in scholarly communication. The Winnower. 2016.
177. Nobarany S, Booth KS: Understanding and supporting anonymity policies in peer review. J Assoc Inf Sci Technol. 2016; 68(4): 957-971.
178. Nosek BA, Lakens D: Registered reports: A method to increase the credibility of published results. Soc Psychol. 2014; 45(3): 137-141.
179. Okike K, Hug KT, Kocher MS, et al.: Single-blind vs Double-blind Peer Review in the Setting of Author Prestige. JAMA. 2016; 316(12): 1315-6.
180. Oldenburg H: Epistle dedicatory. Phil Trans. 1665; 1(1-22).
181. Overbeke J: The state of the evidence: what we know and what we don't know about journal peer review. Peer review in health sciences. BMJ Books, London. 1999; 32-44.
182. Owens S: The world's largest 2-way dialogue between scientists and the public. Sci Am. 2014. Date accessed: 2017-06-12.
183. Paglione LD, Lawrence RN: Data exchange standards to support and acknowledge peer-review activity. Learn Publ. 2015; 28(4): 309-316.
184. Pallavi Sudhir A, Knöpfel R: PhysicsOverflow: A postgraduate-level physics Q&A site and open peer review system. Asia Pac Phys Newslett. 2015; 4(1): 53-55.
185. Palus S: Is double-blind review better? 2015.
186. Parnell LD, Lindenbaum P, Shameer K, et al.: BioStar: An online question & answer resource for the bioinformatics community. PLoS Comput Biol. 2011; 7(10): e1002216.
187. Patel J: Why training and specialization is needed for peer review: a case study of peer review for randomized controlled trials. BMC Med. 2014; 12: 128.
188. Penev L, Mietchen D, Chavan V, et al.: Strategies and guidelines for scholarly publishing of biodiversity data. Research Ideas and Outcomes. 2017; 3: e12431.
189. Perakakis P, Taylor M, Mazza M, et al.: Natural selection of academic papers. Scientometrics. 2010; 85(2): 553-559.
190. Perkel JM: Annotating the scholarly web. Nature. 2015; 528(7580): 153-4.
191. Peters DP, Ceci SJ: Peer-review practices of psychological journals: The fate of published articles, submitted again. Behav Brain Sci. 1982; 5(2): 187-195.
192. Pierie JP, Walvoort HC, Overbeke AJ: Readers' evaluation of effect of peer review and editing on quality of articles in the nederlands tijdschrift voor geneeskunde. Lancet. 1996; 348(9040): 1480-1483.
193. Pinfield S: Mega-journals: the future, a stepping stone to it or a leap into the abyss? Times Higher Education. 2016; Date accessed: 2017-06-12.
194. Plume A, van Weijen D: Publish or perish? The rise of the fractional author. Research Trends. 2014; 38(3).
195. Pocock SJ, Hughes MD, Lee RJ: Statistical problems in the reporting of clinical trials. A survey of three medical journals. N Engl J Med. 1987; 317(7): 426-432.
196. Pontille D, Torny D: The blind shall see! the question of anonymity in journal peer review. Ada: A Journal of Gender, New Media, and Technology. 2014; (4).
197. Pontille D, Torny D: From manuscript evaluation to article valuation: the changing technologies of journal peer review. Hum Stud. 2015; 38(1): 57-79.
198. Pöschl U: Multi-stage open peer review: Scientific evaluation integrating the strengths of traditional peer review with the virtues of transparency and selfregulation. Front Comput Neurosci. 2012; 6: 33. ISSN: 1662-5188.
199. Prechelt L, Graziotin D, Fernández DM: On the status and future of peer review in software engineering. arXiv:1706.07196 [cs.SE]. 2017.
200. Priem J: Scholarship: Beyond the paper. Nature. 2013; 495(7442): 437-440.
201. Priem J, Hemminger BH: Scientometrics 2.0: New metrics of scholarly impact on the social web. First Monday. 2010; 15(7).
202. Priem J, Hemminger BM: Decoupling the scholarly journal. Front Comput Neurosci. 2012; 6: 19.
203. Procter R, Williams R, Stewart J, et al.: Adoption and use of web 2.0 in scholarly communications. Philos Trans A Math Phys Eng Sci. 2010a; 368(1926): 4039-4056.
204. Procter RN, Williams R, Stewart J, et al.: If you build it, will they come? How researchers perceive and use Web 2.0. Technical report, Research Network Information, 2010b.
205. Public Knowledge Project: OJS stats. 2016. Date accessed: 2017-06-12.
206. Pullum GK: Stalking the perfect journal. Nat Lang Linguist Th. 1984; 2(2): 261-267. ISSN: 0167-806X.
207. Putman TE, Burgstaller-Muehlbacher S, Waagmeester A, et al.: Centralizing content and distributing labor: a community model for curating the very long tail of microbial genomes. Database (Oxford). 2016; 2016: pii: baw028.
208. Raaper R: Academic perceptions of higher education assessment processes in neoliberal academia. Crit Stud Educ. 2016; 57(2): 175-190.
209. Rennie D: Let's make peer review scientific. Nature. 2016; 535(7610): 31-3.
210. Rennie D: Misconduct and journal peer review. 2003.
211. Research Information Network: Activities, costs and funding flows in the scholarly communications system in the UK: Report commissioned by the Research Information Network (RIN). 2008.
212. Review Meta: Analysis of 7 million Amazon reviews: customers who receive free or discounted item much more likely to write positive review. 2016. Date accessed: 2017-06-12.
213. Riggs JE: Priority, rivalry, and peer review. J Child Neurol. 1995; 10(3): 255-256.
214. Roberts SG, Verhoef T: Double-blind reviewing at evolang 11 reveals gender bias. Journal of Language Evolution. 2016; 1(2): 163-167.
215. Rodgers P: Decisions, decisions. eLife. 2017; 6: pii: e32011.
216. Rodriguez MA, Bollen J: An algorithm to determine peer-reviewers. In Proceedings of the 17th ACM conference on Information and Knowledge Management. ACM. 2008; 319-328.
217. Rodriguez MA, Bollen J, Van de Sompel H: The convergence of digital libraries and the peer-review process. J Inform Sci. 2006; 32(2): 149-159.
218. Rodríguez-Bravo B, Nicholas D, Herman E, et al.: Peer review: The experience and views of early career researchers. Learn Publ. 2017; 30(4): 269-277. in press.
219. Ross JS, Gross CP, Desai MM, et al.: Effect of blinded peer review on abstract acceptance. JAMA. 2006; 295(14): 1675-1680.
220. Ross N, Boettiger C, Bryan J, et al.: Onboarding at rOpenSci: A year in reviews. rOpenSci Blog. 2016. Date accessed: 2017-06-12.
221. Ross-Hellauer T: What is open peer review? A systematic review [version 1; referees: 1 approved, 3 approved with reservations]. F1000Res. 2017; 6: 588.
222. Rughinis R, Matei S: Digital badges: Signposts and claims of achievement. In Stephanidis C, editor, HCI International 2013-Posters' Extended Abstracts. HCI 2013. Communications in Computer and Information Science. Springer, Berlin, Heidelberg, 2013; 374: 84-88. ISBN: 978-3-642-39476-8.
223. Salager-Meyer F: Scientific publishing in developing countries: Challenges for the future. J Engl Acad Purp. 2008; 7(2): 121-132.
224. Salager-Meyer F: Writing and publishing in peripheral scholarly journals: How to enhance the global influence of multilingual scholars? Journal of English for Academic Purposes. 2014; 13: 78-82.
225. Sanger L: The early history of Nupedia and Wikipedia: a memoir. In DiBona C, Stone M, and Cooper D, editors, Open Sources 2.0: The Continuing Evolution. chapter 20, O'Reilly Media, Sebastopol, CA, 2005; 307-338. ISBN: 978-0-596-00802-4.
226. Schiermeier Q: 'You never said my peer review was confidential'-scientist challenges publisher. Nature. 2017; 541(7638): 446.
227. Schmidt B, Görögh E: New toolkits on the block: Peer review alternatives in scholarly communication. In Chan L and Loizides F, editors, Expanding Perspectives on Open Science: Communities, Cultures and Diversity in Concepts and Practices. IOS Press, 2017; 62-74.
228. Schroter S, Black N, Evans S, et al.: Effects of training on quality of peer review: randomised controlled trial. BMJ. 2004; 328(7441): 673.
229. Schroter S, Tite L, Hutchings A, et al.: Differences in review quality and recommendations for publication between peer reviewers suggested by authors or by editors. JAMA. 2006; 295(3): 314-317.
230. Seeber M, Bacchelli A: Does single blind peer review hinder newcomers? Scientometrics. 2017; 113(1): 567-585.
231. Shotton D: The five stars of online journal articles: A framework for article evaluation. D-Lib Magazine. 2012; 18(1): 1.
232. Shuttleworth S, Charnley B: Science periodicals in the nineteenth and twentyfirst centuries. Notes Rec R Soc Lond. 2016; 70(4): 297-304.
233. Siler K, Lee K, Bero L: Measuring the effectiveness of scientific gatekeeping. Proc Natl Acad Sci U S A. 2015a; 112(2): 360-365.
234. Siler K, Lee K, Bero L: Measuring the effectiveness of scientific gatekeeping. Proc Natl Acad Sci U S A. 2015b; 112(2): 360-365.
235. Siler K, Strang D: Peer review and scholarly originality: Let 1,000 flowers bloom, but don't step on any. Science, Technology, & Human Values. 2017; 42(1): 29-61.
236. Singh Chawla D: Here's why more than 50,000 psychology studies are about to have PubPeer entries. Retraction Watch. 2016; Date accessed: 2017-06-12.
237. Smith AM, Katz DS, Niemeyer KE, et al.: Software citation principles. PeerJ Computer Science. 2016; 2: e86.
238. Smith JW: The deconstructed journal - a new model for academic publishing. Learn Publ. 1999; 12(2): 79-91.
239. Smith R: Peer review: a flawed process at the heart of science and journals. J R Soc Med. 2006; 99(4): 178-182.
240. Smith R: Classical peer review: an empty gun. Breast Cancer Res. 2010; 12(Suppl 4): S13.
241. Snell L, Spencer J: Reviewers' perceptions of the peer review process for a medical education journal. Med Educ. 2005; 39(1): 90-97.
242. Snodgrass RT: Editorial: Single-versus double-blind reviewing. ACM Trans Database Syst. 2007; 32(1): 1.
243. Sobkowicz P: Peer-review in the internet age. arXiv: 0810.0486 [physics.soc-ph]. 2008.
244. Spier R: The history of the peer-review process. Trends Biotechnol. 2002; 20(8): 357-358.
245. Squazzoni F, Brezis E, Marušic A: Scientometrics of peer review. Scientometrics. 2017a; 113(1): 501-502. ISSN: 1588-2861.
246. Squazzoni F, Grimaldo F, Marušic A: Publishing: Journals could share peerreview data. Nature. 2017b; 546(7658): 352.
247. Stanton DC, Bérubé M, Cassuto L, et al.: Report of the MLA task force on evaluating scholarship for tenure and promotion. Profession. 2007; 9-71.
248. Steen RG, Casadevall A, Fang FC: Why has the number of scientific retractions increased? PLoS One. 2013; 8(7): e68397.
249. Stemmle L, Collier K: RUBRIQ: tools, services, and software to improve peer review. Learn Publ. 2013; 26(4): 265-268.
250. Sutton C, Gong L: Popularity of arxiv.org within computer science. 2017.
251. Swan M: Blockchain: Blueprint for a new economy. O'Reilly Media, Sebastopol, CA; 2015; ISBN: 978-1-4919-2044-2.
252. Szegedy C, Zaremba W, Sutskever I, et al.: Intriguing properties of neural networks. In International Conference on Learning Representations. 2014; arXiv:1312.6199 [cs.CV].
253. Tausczik YR, Kittur A, Kraut RE: Collaborative problem solving: A study of MathOverflow. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW' 14), ACM. 2014; 355-367.
254. Tennant JP: The dark side of peer review. Editorial Office News. 2017; 10(8): 2.
255. Tennant JP, Waldner F, Jacques DC, et al.: The academic, economic and societal impacts of Open Access: an evidence-based review [version 3; referees: 4 approved, 1 approved with reservations]. F1000Res. 2016; 5: 632.
256. Teytelman L, Stoliartchouk A, Kindler L, et al.: Protocols.io: Virtual Communities for Protocol Development and Discussion. PLoS Biol. 2016; 14(8): e1002538.
257. Thompson N, Hanley D: Science is shaped by wikipedia: Evidence from a randomized control trial. 2017.
258. Thung F, Bissyande TF, Lo D, et al.: Network structure of social coding in GitHub. In 17th European Conference on Software Maintenance and Reengineering (CSMR), IEEE, 2013; 323-326.
259. Tomkins A, Zhang M, Heavlin WD: Single versus double blind reviewing at WSDM 2017. arXiv:1702.00502 [cs.DL]. 2017.
260. Torpey K: Astroblocks puts proofs of scientific discoveries on the bitcoin blockchain. Inside Bitcoins. 2015; Date accessed: 2017-06-12.
261. Tregenza T: Gender bias in the refereeing process? Trends Ecol Evol. 2002; 17(8): 349-350.
262. Ubois J: Online reputation systems. In: Dyson E, editor, Release 1.0, EDventure Holdings Inc., New York, NY, 2003; 21: 1-35.
263. van Assen MA, van Aert RC, Nuijten MB, et al.: Why publishing everything is more effective than selective publishing of statistically significant results. PLoS One. 2014; 9(1): e84896.
264. Van Noorden R: Web of Science owner buys up booming peer-review platform. Nature News. 2017.
265. van Rooyen S, Delamothe T, Evans SJ: Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial. BMJ. 2010; 341: c5729.
266. van Rooyen S, Godlee F, Evans S, et al.: Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomised trial. BMJ. 1999; 318(7175): 23-27.
267. van Rooyen S, Godlee F, Evans S, et al.: Effect of blinding and unmasking on the quality of peer review: a randomized trial. JAMA. 1998; 280(3): 234-237.
268. Vines T: Molecular Ecology's best reviewers 2015. The Molecular Ecologist. 2015a; Accessed: 2017-06-12.
269. Vines TH: The core inefficiency of peer review and a potential solution. Limnology and Oceanography Bulletin. 2015b; 24(2): 36-38.
270. Vitolo C, Elkhatib Y, Reusser D, et al.: Web technologies for environmental big data. Environ Model Softw. 2015; 63: 185-198. ISSN: 1364-8152.
271. von Muhlen M: We need a Github of science. 2011; Date accessed: 2017-06-12.
272. W3C: Three recommendations to enable annotations on the web. 2017; Date accessed: 2017-06-12.
273. Wakeling S, Willett P, Creaser C, et al.: Open-Access Mega-Journals: A Bibliometric Profile. PLoS One. 2016; 11(11): e0165359.
274. Walker R, Rocha da Silva P: Emerging trends in peer review-a survey. Front Neurosci. 2015; 9: 169.
275. Walsh E, Rooney M, Appleby L, et al.: Open peer review: a randomised controlled trial. Br J Psychiatry. 2000; 176(1): 47-51.
276. Wang W, Kong X, Zhang J, et al.: Editorial behaviors in peer review. Springerplus. 2016; 5(1): 903.
277. Wang WT, Wei ZH: Knowledge sharing in wiki communities: an empirical study. Online Inform Rev. 2011; 35(5): 799-820.
278. Ware M: Peer review in scholarly journals: Perspective of the scholarly community-results from an international study. Information Services and Use. 2008; 28(2): 109-112.
279. Ware M: Peer review: Recent experience and future directions. New Review of Information Networking. 2011; 16(1): 23-53.
280. Warne V: Rewarding reviewers-sense or sensibility? a Wiley study explained. Learn Publ. 2016; 29(1): 41-50.
281. Webb TJ, O'Hara B, Freckleton RP: Does double-blind review benefit female authors? Trends Ecol Evol. 2008; 23(7): 351-353; author reply 353-4.
282. Weicher M: Peer review and secrecy in the "information age". Proc Am Soc Inform Sci Tech. 2008; 45(1): 1-12.
283. Whaley D: Annotation is now a web standard. hypothes.is Blog. 2017; Date accessed: 2017-06-12.
284. Whittaker RJ: Journal review and gender equality: a critical comment on Budden et al. Trends Ecol Evol. 2008; 23(9): 478-479; author reply 480.
285. Wicherts JM: Peer Review Quality and Transparency of the Peer-Review Process in Open Access and Subscription Journals. PLoS One. 2016; 11(1): e0147913.
286. Wikipedia contributors: Digital medievalist. 2017; Accessed: 2017-6-16.
287. Wodak SJ, Mietchen D, Collings AM, et al.: Topic pages: PLoS Computational Biology meets Wikipedia. PLoS Comput Biol. 2012; 8(3): e1002446.
288. Xiao L, Askin N: Wikipedia for academic publishing: advantages and challenges. Online Inform Rev. 2012; 36(3): 359-373.
289. Xiao L, Askin N: Academic opinions of Wikipedia and open access publishing. Online Inform Rev. 2014; 38(3): 332-347.
290. Yarkoni T: Designing next-generation platforms for evaluating scientific output: what scientists can learn from the social web. Front Comput Neurosci. 2012; 6: 72.
291. Yeo SK, Liang X, Brossard D, et al.: The case of #arseniclife: Blogs and Twitter in informal peer review. Front Comput Neurosci. 2017; 26(8): 937-952.
292. Yli-Huumo J, Ko D, Choi S, et al.: Where Is Current Research on Blockchain Technology?-A Systematic Review. PLoS One. 2016; 11(10): e0163477.
293. Zamiska N: Nature cancels public reviews of scientific papers. Wall Str J. 2006; Date accessed: 2017-06-12.
294. Zuckerman H, Merton RK: Patterns of evaluation in science: Institutionalization, structure and functions of the referee system. Minerva. 1971; 9(1): 66-100.