Alternatives to peer review: Novel approaches for research evaluation

Frontiers in Computational Neuroscience

Volumn 5, Issue , 2011, Pages

  Birukou, Aliaksandr ^a,b   Wakeling, Joseph Rushton ^b   Bartolini, Claudio ^c   Casati, Fabio ^a   Marchese, Maurizio ^a   Mirylenka, Katsiaryna ^a   Osman, Nardine ^d   Ragone, Azzurra ^a,e   Sierra, Carles ^d   Wassef, Aalam ^f  

^a UNIVERSITY OF TRENTO   (Italy)

^b European Alliance for Innovation   (Belgium)

^c HEWLETT PACKARD LABORATORIES   (United States)

^d ARTIFICIAL INTELLIGENCE RESEARCH INSTITUTE IIIA CSIC   (Spain)

^e Exprivia Spa   (Italy)

^f PeerevaluationOrg   (France)

Author keywords

Bidding; LiquidPub; Metrics; Opinions; Peer review; Research evaluation; UCount

Indexed keywords

NEUROLOGY;

BIDDING; LIQUIDPUB; METRICS; OPINIONS; PEER REVIEW; RESEARCH EVALUATION; UCOUNT;

MEDICAL COMPUTING;

PEER REVIEW; VISION;

EID: 84889009337     PISSN: None     EISSN: 16625188     Source Type: Journal    
DOI: 10.3389/fncom.2011.00056     Document Type: Review

References (67)

1. Adomavicius, G., and Tuzhilin, A. (2005). Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans. Knowl. Data Eng. 17, 734-749.
2. Akst, J. (2010). I hate your paper. Scientist 24, 36.
3. Best, M. L. (2004). Can the internet be a human right? Hum. Rights Hum. Welf. 4, 23-31.
4. Black, N., van Rooyen, S., Godlee, F., Smith, R., and Evans, S. (1998). What makes a good reviewer and a good review for a general medical journal? J. Am. Med. Assoc. 280, 231-233
5. Bornmann, L. (2007). Bias cut. women, it seems, often get a raw deal in science so how can discrimination be tackled? Nature 445, 566.
6. Bornmann, L., and Daniel, H.-D. (2005a). Committee peer review at an international research foundation: predictive validity and fairness of selection decisions on postgraduate fellowship applications. Res. Eval. 14, 15-20.
7. Bornmann, L., and Daniel, H.-D. (2005b). Selection of research fellowship recipients by committee peer review. reliability, fairness and predictive validity of board of trustees’ decisions. Scientometrics 63, 297-320.
8. Bornmann, L., and Daniel, H.-D. (2010a). The validity of staff editors’ initial evaluations of manuscripts: a case study of Angewandte Chemie International Edition. Scientometrics 85, 681-687.
9. Bornmann, L., and Daniel, H.-D. (2010b). The usefulness of peer review for selecting manuscripts for publication: a utility analysis taking as an example a high-impact journal. PLoS ONE 5, e11344. doi:10.1371/journal.pone.0011344
10. Bornmann, L., Wallon, G., and Ledin, A. (2008). Does the committee peer review select the best applicants for funding? An investigation of the selection process for two European molecular biology organization programmes. PLoS ONE 3, e3480. doi:10.1371/journal.pone.0003480
11. Brody, T., Harnad, S., and Carr, L. (2006). Earlier web usage statistics as predictors of later citation impact. JASIST 58, 1060-1072.
12. Burnham, J. C. (1990). The evolution of editorial peer review. J. Am. Med. Assoc. 263, 1323-1329.
13. Casati, F., Marchese, M., Mirylenka, K., and Ragone, A. (2010). Reviewing Peer Review: A Quantitative Analysis of Peer Review. Technical Report 1813. University of Trento. Available at: http://eprints.biblio.unitn.it/archive/00001813/
14. Ceci, S. J., and Peters, D. P. (1982). Peer review: a study of reliability. Change 14, 44-48.
15. Ceci, S. J., and Williams, W. M. (2011). Understanding current causes of women’s underrepresentation in science. Proc. Natl. Acad. Sci. U.S.A. 108, 3157-3162.
16. Cho, M. K., Justice, A. C., Winker, M. A., Berlin, J. A., Waeckerle, J. F., Callaham, M. L., and Rennie, D. (1998). Masking author identity in peer review: what factors influence masking success? PEER Investigators. JAMA 280, 243-245.
17. de Solla Price, D. (1976). A general theory of bibliometric and other cumulative advantage processes. J. Am. Soc. Inf. Sci. 27, 292-306.
18. Evans, J. A. (2008). Electronic publication and the narrowing of science and scholarship. Science 321, 395-399.
19. Fisher, M., Friedman, S. B., and Strauss, B. (1994). The effects of blinding on acceptance of research papers by peer review. J. Am. Med. Assoc. 272, 143-146.
20. Gichora, N. N., Fatumo, S. A., Ngara, M. V., Chelbat, N., Ramdayal, K., Opap, K. B., Siwo, G. H., Adebiyi, M. O., El Gonnouni, A., Zofou, D., Maurady, A. A. M., Adebiyi, E. F., de Villiers, E. P., Masiga, D. K., Bizzaro, J. W., Suravajhala, P., Ommeh, S. C., and Hide, W. (2010). Ten simple rules for organizing a virtual conference anywhere. PLoS Comput. Biol. 6, e1000650. doi:10.1371/journal.pcbi.1000650
21. Ginsparg, P. (1994). First steps towards electronic research communication. Comput. Phys. 8, 390-396.
22. Godlee, F. (2002). Making reviewers visible: openness, accountability, and credit. JAMA 287, 2762-2765
23. Godlee, F., Gale, C. R., and Martyn, C. N. (1998). Effect on the quality of peer review of blinding reviewers and asking them to sign their reports a randomized controlled trial. JAMA 280, 237-240.
24. Goodman, S. N., Berlin, J., Fletcher, S. W., and Fletcher, R. H. (1994). Manuscript quality before and after peer review and editing at annals of internal medicine. Ann. Intern. Med. 121, 11-21.
25. Greaves, S., Scott, J., Clarke, M., Miller, L., Hannay, T., Thomas, A., and Campbell, P. (2006). Overview: Nature’s peer review trial. Nature. doi: 10.1038/nature05535.
26. Harnad, S. (1990). Scholarly skywriting and the prepublication continuum of scientific enquiry. Psychol. Sci. 1, 342-344.
27. Hu, N., Pavlou, P. A., and Zhang, J. (2009). Overcoming the J-shaped distribution of product reviews. Commun. ACM 52, 144-147.
28. Ingelfinger, F. J. (1974). Peer review in biomedical publication. Am. J. Med. 56, 686-692.
29. Jefferson, T., Rudin, M., Folse, S. B., and Davidoff, F. (2007). Editorial peer review for improving the quality of reports of biomedical studies. Cochrane 41, MR000016.
30. Jefferson, T., Wager, E., and Davidoff, F. (2002a). Measuring the quality of editorial peer review. JAMA 287, 2786-2790.
31. Jefferson, T., Alderson, P., Wager, E., and Davidoff, F. (2002b). Effects of editorial peer review: a systematic review. JAMA 287, 2784-2786.
32. Justice, A. C., Cho, M. K., Winker, M. A., Berlin, J. A., Rennie, D., and PEER Investigators. (1998). Does masking author identity improve peer review quality? A randomized controlled trial. JAMA 280, 240-242.
33. Kassirer, J. P., and Campion, E. W. (1994). Peer review: crude and understudied, but indispensable. J. Am. Med. Assoc. 272, 96-97.
34. Katz, D. S., Proto, A. V., and Olmsted, W. W. (2002). Incidence and nature of unblinding by authors: our experience at two radiology journals with double-blinded peer review policies. Am. J. Roentgenol. 179, 1415-1417.
35. Kronick, D. A. (1990). Peer review in 18th-century scientific journalism. JAMA 263, 1321-1322.
36. Lawrence, P. A. (2003). The politics of publication. Nature 422, 259-261
37. Lee, K., Boyd, E., Holroyd-Leduc, J., Bacchetti, P., and Bero, L. (2006). Predictors of publication: characteristics of submitted manuscripts associated with acceptance at major biomedical journals. Med. J. Aust. 184, 621.
38. Link, A. M. (1998). Us and non-US submissions: an analysis of reviewer bias. JAMA 280, 246-247.
39. Lock, S. (1994). Does editorial peer review work? Ann. Intern. Med. 121, 60-61.
40. Lynch, J. R., Cunningham, M. R., Warme, W. J., Schaad, D. C., Wolf, F. M., and Leopold, S. S. (2007). Commercially funded and united states-based research is more likely to be published; good-quality studies with negative outcomes are not. J. Bone Joint Surg. Am. 89, 1010-1018.
41. Marsh, H. W., Bornmann, L., Mutz, R., Daniel, H.-D., and O’Mara, A. (2009). Gender effects in the peer reviews of grant proposals: a comprehensive meta-analysis comparing traditional and multilevel approaches. Rev. Educ. Res. 79, 1290-1326.
42. Masum, H., and Zhang, Y.-C. (2004). Manifesto for the reputation society. First Monday 9 [Online].
43. McCook, A. (2006). Is peer review broken? Scientist 20, 26.
44. McNutt, R. A., Evans, A. T., Fletcher, R. H., and Fletcher, S. W. (1990). The effects of blinding on the quality of peer review: a randomized trial. JAMA 263, 1371-1376.
45. Medo, M., Cimini, G. and Gualdi, S. (2011). Temporal effects in the growth of networks. Available at: http://arxiv.org/abs/1109.5560
46. Medo, M., and Wakeling, J. R. (2010). The effect of discrete vs. continuousvalued ratings on reputation and ranking systems. Europhys. Lett. 91, 48004.
47. Odlyzko, A. M. (1995). Tragic loss or good riddance? The impending demise of traditional scholarly journals. Int. J. Hum. Comput. Sci. 42, 71-122.
48. Olson, C. M., Rennie, D., Cook, D., Dickersin, K., Flanagin, A., Hogan, J. W., Zhu, Q., Reiling, J., and Pace, B. (2002). Publication bias in editorial decision making. JAMA 287, 2825-2828.
49. Opthof, T., Coronel, R., and Janse, M. J. (2002). The significance of the peer review process against the background of bias: priority ratings of reviewers and editors and the prediction of citation, the role of geographical bias. Cardiovasc. Res. 56, 339-346.
50. Osman, N., Sabater-Mir, J., and Sierra, C. (2011). "Simulating research behaviour," in 12th International Workshop on Multi-Agent-Based Simulation (MABS’11), Taipei.
51. Osman, N., Sabater-Mir, J., Sierra, C., de Pinninck Bas, A. P., Imran, M., Marchese, M., and Ragone, A. (2010a). Credit attribution for liquid publications. Deliverable D4.1, Liquid Publications Project. Available at: https://dev.liquidpub.org/svn/liquidpub/papers/deliverables/LP_D4.1.pdf
52. Osman, N., Sierra, C., and SabaterMir, J. (2010b). "Propagation of opinions in structural graphs," in ECAI 2010: Proceedings of the 19th European Conference on Artificial Intelligence, Vol. 215 of Frontiers in Artificial Intelligence and Applications, eds H. Coelho, R. Studer, and M. Wooldridge (Lisbon: IOS Press), 595-600.
53. Parra, C., Birukou, A., Casati, F., Saint-Paul, R., Wakeling, J. R., and Chlamtac, I. (2011). "UCount: a community-driven approach for measuring scientific reputation," in Proceedings of Altmetrics11: Tracking Scholarly Impact on the Social Web, Koblenz.
54. Purcell, G. P., Donovan, S. L., and Davidoff, F. (1998). Changes to manuscripts during the editorial process: characterizing the evolution of a clinical paper. J. Am. Med. Assoc. 280, 227-228.
55. Ragone, A., Mirylenka, K., Casati, F., and Marchese, M. (2011). "A quantitative analysis of peer review," in 13th International Society of Scientometrics and Informetrics Conference, Durban
56. Reinhart, M. (2009). Peer review of grant applications in biology and medicine. reliability, fairness, and validity. Scientometrics 81, 789-809
57. Ross, J. S., Gross, C. P., Desai, M. M., Hong, Y., Grant, A. O., Daniels, S. R., Hachinski, V. C., Gibbons, R. J., Gardner, T. J., and Krumholz, H. M. (2006). Effect of blinded peer review on abstract acceptance. J. Am. Med. Assoc. 295, 1675-1680.
58. Sense About Science. (2009). Peer Review Survey: Preliminary Results. Available at: http://www.senseabout science.org.uk/index.php/site/project/29/
59. Smith, R. (2006). Peer review: a flawed process at the heart of science and journals. J. R. Soc. Med. 99, 178-182
60. Spier, R. (2002). The history of the peerreview process. Trends Biotechnol. 20, 357-358.
61. Swan, A. (2007). Open access and the progress of science. Am. Sci. 95, 198-200.
62. Underwood, A. J. (2004). It would be better to create and maintain quality rather than worrying about its measurement. Mar. Ecol. Prog. Ser. 270, 283-286.
63. van Rooyen, S., Black, N., and Godlee, F. (1999). Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts. J. Clin. Epidemiol. 52, 625-629.
64. Walsh, E., Rooney, M., Appleby, L., and Wilkinson, G. (2000). Open peer review: a randomised controlled trial. Br. J. Psychiatry 176, 47-51.
65. Ware, M., and Monkman, M. (2008). Peer Review in Scholarly Journals: Perspective of the Scholarly Community An International Study. Survey Commissioned by the Publishing Research Consortium. Available at: http://www.publishingresearch.net/PeerReview.htm
66. Wenneras, C., and Wold, A. (1997). Nepotism and sexism in peer-review. Nature 387, 341-343.
67. Zhou, T., Kuscsik, Z., Liu, J.-G., Medo, M., Wakeling, J. R., and Zhang, Y.-C. (2010). Solving the apparent diversity-accuracy dilemma of recommender systems. Proc. Natl. Acad. Sci. U.S.A. 107, 4511-4515.