Designing next-generation platforms for evaluating scientific output: What scientists can learn from the social web

Frontiers in Computational Neuroscience

Volumn , Issue OCTOBER 2012, 2012, Pages

  Yarkoni, Tal ^a  

^a UNIVERSITY OF COLORADO   (United States)

Author keywords

Data sharing; Open access; Peer review; Publishing; Scientific evaluation

Indexed keywords

CORE ELEMENTS; CUSTOMIZABLE; DATA SHARING; EVALUATION MODELS; EVALUATION PLATFORMS; OPEN ACCESS; PEER REVIEW; PERFORMANCE METRICS; SCIENTIFIC COMMUNITY; SCIENTIFIC EVALUATION; SCIENTIFIC OUTPUT; SCIENTIFIC PUBLICATIONS; TRANSPARENT ACCESS; WEB APPLICATION;

INFORMATION TECHNOLOGY; PUBLISHING; REVIEWS;

QUALITY CONTROL;

COMPUTER INTERFACE; COMPUTER MODEL; COMPUTER NETWORK; COMPUTER PROGRAM; COMPUTER SIMULATION; INTERNET; PROGRAM DEVELOPMENT; QUALITY CONTROL; RELIABILITY; REVIEW; SOCIAL NETWORK;

EID: 84867607168     PISSN: 16625188     EISSN: None     Source Type: Journal    
DOI: 10.3389/fncom.2012.00072     Document Type: Review

References (81)

1. Adomavicius, G., and Tuzhilin, A. (2005). Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans. Knowl. Data Eng. 17, 734-749.
2. Baggs, J. G., Broome, M. E., Dougherty, M. C., Freda, M. C., and Kearney, M. H. (2008). Blinding in peer review: the preferences of reviewers for nursing journals. J. Adv. Nurs. 64, 131-138.
3. Barrington, L., Oda, R., and Lanckriet, G. (2009). "Smarter than genius? human evaluation of music recom-mender systems," in Proceedings of the International Society for Music Information Retrieval Conference, 357-362.
4. Basu, C., Cohen, W. W., Hirsh, H., and Nevill-Manning, C. (2011). Technical paper recommendation: a study in combining multiple information sources. J. Artif. Intell. Res. 14, 231-252.
5. Bath, F. J., Owen, V. E., and Bath, P. M. (1998). Quality of full and final publications reporting acute stroke trials: a systematic review. Stroke 29, 2203-2210.
6. Bennett, J., and Lanning, S. (2007). "The netflix prize," in Proceedings of KDD Cup and Workshop, Vol. 2007 (San Jose, CA), 8.
7. Bogers, T., and Van Den Bosch, A. (2008). "Recommending scientific articles using citeulike," in Proceedings of the 2008 ACM Conference on Recommender Systems RecSys 08 (Lausanne, Switzerland), 23, 287.
8. Bombieri, E. (2011). "The mathematical infinity," in Infinity: New Research Frontiers, eds M. Heller and W. H. Woodin (Cambridge, UK: Cambridge University Press), 55-75.
9. Bornmann, L., Mutz, R., and Daniel, H.-D. (2008). Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine. J. Am. Soc. Inf. Sci. Technol. 59, 830-837.
10. Bornmann, L., Mutz, R., and Daniel, H.-D. (2010). A reliability-generalization study of journal peer reviews: a multilevel meta-analysis of inter-rater reliability and its determinants. PLoS ONE 5:e14331. doi: 10.1371/journal.pone.0014331
11. Bornmann, L., Mutz, R., Hug, S. E., and Daniel, H.-D. (2011). A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants. J. Informetr. 5, 346-359.
12. Broad, W. (1981). The publishing game: getting more for less. Science 211, 1137-1139.
13. Casati, F., Giunchiglia, F., and Marchese, M. (2007). Publish and perish: why the current publication and review model is killing research and wasting your money. Ubiquity 2007, 3.
14. Casey, M. A., Veltkamp, R., Goto, M., Leman, M., Rhodes, C., and Slaney, M. (2008). "Content-based music information retrieval: current directions and future challenges," in Proceedings of the IEEE 96, 668-696.
15. Chew, F. (1991). Fate of manuscripts rejected for publication in the AJR. Am. J. Roentgenol. 156, 627-632.
16. Chondros, T. G. (2010a). "Archimedes influence in science and engineering," in The Genius of Archimedes-23 Centuries of Influence on Mathematics, Science and Engineering, Vol. 11, eds S. A. Paipetis and M. Ceccarelli (Dordrecht: Springer Netherlands), 411-425.
17. Chondros, T. G. (2010b). Archimedes life works and machines. Mech. Mach. Theory 45, 1766-1775.
18. Dawes, R., Faust, D., and Meehl, P. (1989). Clinical versus actuarial judgment. Science 243, 1668-1674.
19. Delamothe, T., and Smith, R. (1999). Moving beyond journals: the future arrives with a crash. BMJ 318, 1637-1639.
20. Dong, P., Loh, M., and Mondry, A. (2005). The "impact factor" revisited. Biomed. Digit. Libr. 2, 7.
21. Ducheneaut, N., Partridge, K., Huang, Q., Price, B., Roberts, M., Chi, E., Bellotti, V., and Begole, B. (2009). "Collaborative filtering is not enough? Experiments with a mixed-model recommender for leisure activities," in User Modeling Adaptation and Personalization, Vol. 5535, eds G.-J. Houben, G. McCalla, F. Pianesi, and M. Zancanaro (Trento, Italy), 295-306.
22. Dumais, S. T., and Nielsen, J. (1992). "Automating the assignment of submitted manuscripts to reviewers," in Proceedings of the 15th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval SIGIR 92, eds N. Belkin, P. Ingwersen, and A. M. Pejtersen (Copenhagen, Denmark: ACM Press), 233-244.
23. Egghe, L. (2006). Theory and practise of the g-index. Scientometrics 69, 131-152.
24. Ellison, G. (2002). The slowdown of the economics publishing process. J. Polit. Econ. 110, 947-993.
25. Fabiato, A. (1994). Anonymity of reviewers. Cardiovasc. Res. 28, 1134-1139.
26. Gipp, B., Beel, J., and Hentschel, C. (2009). "Scienstein: a research paper recommender system," in Proceedings of the International Conference on Emerging Trends in Computing ICETiC'09, Vol. 301, (Virudhunagar, Tamilnadu, India: Citeseer), 309-315.
27. Goodman, S. N., Berlin, J., Fletcher, S. W., and Fletcher, R. H. (1994). Manuscript quality before and after peer review and editing at Annals of Internal Medicine. Ann. Intern. Med. 121, 11-21.
28. Gotzsche, P. C., Delamothe, T., Godlee, F., and Lundh, A. (2010). Adequacy of authors' replies to criticism raised in electronic letters to the editor: cohort study. BMJ 341, c3926.
29. Hall, S. A., and Wilcox, A. J. (2007). The fate of epidemiologic manuscripts: a study of papers submitted to epidemiology. Epidemiology 18, 262-265.
30. Hanson, R. K., and Morton-Bourgon, K. E. (2009). The accuracy of recidivism risk assessments for sexual offenders: a meta-analysis of 118 prediction studies. Psychol. Assess. 21, 1-21.
31. Harnad, S. (1999, July 18). Free at last: the future of peer-reviewed journals. D-Lib Magazine.
32. Heath, T. L. (1897). The Works of Archimedes. Cambridge, UK: Cambridge University Press.
33. Hirsch, J. E. (2005). An index to quantify an individual's scientific research output. Proc. Natl. Acad. Sci. U.S.A. 102, 16569-16572.
34. Hojat, M., Gonnella, J., and Caelleigh, A. (2003). Impartial judgment by the "Gatekeepers" of science: fallibility and accountability in the peer review process. Adv. Health Sci. Educ. 8, 75-96.
35. Ioannidis, J. P. A., Tatsioni, A., and Karassa, F. B. (2010). Who is afraid of reviewers' comments? Or, why anything can be published and anything can be cited. Eur. J. Clin. Invest. 40, 285-287.
36. Jefferson, T. (1998). Redundant publication in biomedical sciences: scientific misconduct or necessity? Sci. Eng. Ethics 4, 135-140.
37. Jefferson, T., Rudin, M., Brodney Folse, S., and Davidoff, F. (2007). Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database Syst. Rev. 2, MR000016.
38. Justice, A. C., Cho, M. K., Winker, M. A., Berlin, J. A., and Rennie, D. (1998). Does masking author identity improve peer review quality? A randomized controlled trial. PEER Investigators. JAMA 280, 240-242.
39. Kingsley, D. (2007). The journal is dead, long live the journal. Horizon 15, 211-221.
40. Klarreich, E. (2011). The Global Math Commons. Simons Foundation. Retrieved July 17 2011, from https://simonsfoundation.org/mathematics-physical-sciences/news/-/asset_publisher/bo1E/content/the-global-math-commons
41. Kravitz, D. J., and Baker, C. I. (2011). Toward a new model of scientific publishing: discussion and a proposal. Front. Comput. Neurosci. 5:1-12. doi: 10.3389/fncom.2011. 00055
42. LaPorte, R. E., Marler, E., Akazawa, S., Sauer, F., Gamboa, C., Shenton, C., Glosser, C., Villasenor, A., and Maclure, M. (1995). The death of biomedical journals. BMJ 310, 1387-1390.
43. Lane, J. (2010). Let's make science metrics more scientific. Nature 464, 488-489.
44. Liu, J., Dolan, P., and Pedersen, E. R. (2010). "Personalized news recommendation based on click behavior," in Proceedings of the 15th International Conference on Intelligent User Interfaces. Search ACM (Hong Kong, China), 31-40.
45. Mahoney, M. J. (1977). Publication prejudices: an experimental study of confirmatory bias in the peer review system. Cognit. Ther. Res. 1, 161-175.
46. Maier, G. (2006). Impact factors and peer judgment: the case of regional science journals. Scientometrics 69, 651-667.
47. Massa, P., and Avesani, P. (2007). Trust metrics on controversial users: balancing between tyranny of the majority and echo chambers. Int. J. Semantic Web Infor. Syst. 3, 39-64.
48. Mayor, J. (2010). Frontiers: are scientists nearsighted gamblers? the misleading nature of impact factors. Front. Psychol. 1:215. doi: 10.3389/fpsyg.2010.00215
49. Miller, B. N., Albert, I., Lam, S. K., Konstan, J. A., and Riedl, J. (2003). "MovieLens unplugged: experiences with an occasionally connected rec-ommender system," in Proceedings of the 8th International Conference on Intelligent User Interfaces, Work ACM (Miami, FL), 263-266.
50. Naak, A., Hage, H., and Aïmeur, E. (2009). "A multi-criteria collaborative filtering approach for research paper recommendation in papyres," in 4th International Conference, MCETECH 2009. E-Technologies Innovation in an Open World (Ottawa, Canada), 25-39.
51. Neylon, C., and Wu, S. (2009). Article-level metrics and the evolution of scientific impact. PLoS Biol. 7:e1000242. doi: 10.1371/journal. pbio.1000242
52. Nielsen, M. (May 2009). Doing science in the open. Phys. World 30-35.
53. Nosek, B. A., and Bar-Anan, Y. (2012). Scientific utopia: I. Opening scientific communication. Psychol. Inq. 23, 217-243.
54. Odlyzko, A. M. (1995). Tragic loss or good riddance? The impending demise of traditional scholarly journals. Int. J. Hum. Comput. Stud. 42, 71-122.
55. Pathak, B., Garfinkel, R., Gopal, R. D., Venkatesan, R., and Yin, F. (2010). Empirical analysis of the impact of recommender systems on sales. J. Manag. Inf. Syst. 27, 159-188.
56. Pazzani, M. J., and Billsus, D. (2007). "Content-based recommendation systems," in The Adaptive Web, Vol. 4321, eds P. Brusilovsky, A. Kobsa, and W. Nejdl (Berlin, Germany: Springer), 325-341.
57. Peirce, C. S. (1932). Collected Papers of Charles Sanders Peirce. Boston, MA: Harvard University Press.
58. Phelan, O., McCarthy, K., and Smyth, B. (2009). "Using twitter to recommend real-time topical news," in Proceedings of the Third ACM Conference on Recommender Systems RecSys 09, (New York, NY: Systemspag), 385.
59. Platt, J. R. (1964). Strong inference. Science 146, 347-353.
60. Popper, K. (2002). The Logic of Scientific Discovery. Vol. 2, eds M. Archer, R. Bhaskar, A. Collier, T. Lawson, and A. Norrie (New York, NY: The Routledge), 513.
61. Poynder, R. (2011). PLoS ONE, open access, and the future of scholarly publishing. Available online at: http://richardpoynder.co.uk/PLoS_ONE.pdf
62. Priem, J., and Hemminger, B. H. (2010). Scientometrics 2.0, New metrics of scholarly impact on the social Web. First Monday, 15.
63. Pöschl, U. (2010). Interactive open access publishing and peer review: the effectiveness and perspectives of transparency and self-regulation in scientific communication and evaluation. Atmos. Chem. Phys. 19, 293-314.
64. Ray, J. (2000). The fate of manuscripts rejected by a general medical journal. Am. J. Med. 109, 131-135.
65. Sarwar, B., Karypis, G., Konstan, J., and Riedl, J. (2000). Analysis of recommendation algorithms for e-commerce. Organization 5, 158-167.
66. Schafer, J. B., Frankowski, D., Herlocker, J., and Sen, S. (2007). Collaborative filtering recom-mender systems. Int. J. Electronic Bus. 2, 77.
67. Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. BMJ 314, 497.
68. Shadbolt, N., Brody, T., Carr, L., and Harnad, S. (2006). "The open research web: a preview of the optimal and the inevitable," in Open Access: Key Strategic, Technical and Economic Aspects, ed N. Jacobs (Oxford, UK: Chandos Publishing).
69. Smith, R. (2006). Peer review: a flawed process at the heart of science and journals. J. R. Soc. Med. 99, 178-182.
70. Smith, R. (2010). Classical peer review: an empty gun. Breast Cancer Res. 12(Suppl. 4), S13.
71. Sutherland, W. J., Goulson, D., Potts, S. G., and Dicks, L. V. (2011). Quantifying the impact and relevance of scientific research. PLoS ONE 6:e27537. doi: 10.1371/jour-nal.pone.0027537
72. Torres, R., McNee, S. M., Abel, M., Konstan, J. A., and Riedl, J. (2004). "Enhancing digital libraries with Tech Lens," in Proceedings of the 2004 Joint ACMIEEE Conference on Digital Libraries JCDL 04 (Tucson, Arizona: ACM Press), 228-236.
73. Von Elm, E., Poglia, G., Walder, B., and Tramèr, M. R. (2004). Different patterns of duplicate publication: an analysis of articles used in systematic reviews. JAMA 291, 974-980.
74. Walsh, E., Rooney, M., Appleby, L., and Wilkinson, G. (2000). Open peer review: a randomised controlled trial. Br. J. Psychiatry 176, 47-51.
75. Ware, M. (2007). Peer Review in Scholarly Journals: Perspectives of the Scholarly Community - An International Study. Bristol, UK: Publishing Research Consortium, [Internet].
76. West, R., and McIlwaine, A. (2002). What do citation counts count for in the field of addiction? An empirical evaluation of citation counts and their link with peer ratings of quality. Addiction 97, 501-504.
77. White, M. J. (2006). The meta-analysis of clinical judgment project: fifty-six years of accumulated research on clinical versus statistical prediction stefania aegisdottir. Couns. Psychol. 34, 341-382.
78. Yoshii, K., Goto, M., Komatani, K., Ogata, T., and Okuno, H. G. (2008). An efficient hybrid music recom-mender system using an incrementally trainable probabilistic generative model. IEEE Trans. Audio Speech Lang. Process. 16, 435-447.
79. Young, N. S., Ioannidis, J. P. A., and Al-Ubaydli, O. (2008). Why current publication practices may distort science. PLoS Med. 5:e201. doi: 10.1371/journal.pmed.0050201
80. van Rooyen, S., Godlee, F., Evans, S., Smith, R., and Black, N. (1998). Effect of blinding and unmasking on the quality of peer review: a randomized trial. JAMA 280, 234-237.
81. van Rooyen, S., Delamothe, T., and Evans, S. J. (2010). Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial. BMJ 341, c5729.