Over-optimization of academic publishing metrics: Observing Goodhart's Law in action

GigaScience

Volumn 8, Issue 6, 2019, Pages

  Fire, Michael ^a   Guestrin, Carlos ^b  

^a BEN GURION UNIVERSITY OF THE NEGEV   (Israel)

^b University of Washington   (United States)

Author keywords

Academic publishing metrics; Big data; Data science; Goodhart's Law; Science of science; Scientometrics

Indexed keywords

ARTICLE; BIG DATA; BIOLOGY; DATA SCIENCE; HUMAN; PUBLISHING; SCIENTIST; VALIDITY; BENCHMARKING; BIBLIOMETRICS; JOURNAL IMPACT FACTOR;

BENCHMARKING; BIBLIOMETRICS; JOURNAL IMPACT FACTOR;

EID: 85067287872     PISSN: None     EISSN: 2047217X     Source Type: Journal    
DOI: 10.1093/gigascience/giz053     Document Type: Article

References (85)

1. Ware M, Mabe M. The STM report: An overview of scientific and scholarly journal publishing. 2015. https://www.stm-Assoc.org/2015 02 20 STM Report 2015.pdf. Accessed 12 May 2019.
2. Herrmannova D, Knoth P. An analysis of the Microsoft Academic Graph. D-Lib Mag 2016;22(9/10), doi:10.1045/september2016-herrmannova.
3. Bj ork BC. Have the "mega-journals" reached the limits to growth? PeerJ 2015;3:e981.
4. Kelly S. The continuing evolution of publishing in the biological sciences. Biol Open 2018;7(8), doi: 10.1242/bio.037325.
5. Roemer RC, Borchardt R. From bibliometrics to altmetrics: A changing scholarly landscape. Coll Res Lib News 2012;73(10):596-600.
6. Hirsch JE. An index to quantify an individuals scientific research output. Proc Natl Acad Sci U S A 2005;102(46):16569-72.
7. Garfield E. The agony and the ecstasy-The history and meaning of the journal impact factor. 2005. http://garfield.library.upenn.edu/papers/jifchicago2005.pdf.
8. Wilsdon J. The Metric Tide: Independent Review of the Role of Metrics in Research Assessment and Management. Sage; 2015.
9. Edwards MA, Roy S. Academic research in the 21st century: Maintaining scientific integrity in a climate of perverse incentives and hypercompetition. Environ Eng Sci 2017;34(1):51-61.
10. Biagioli M. Watch out for cheats in citation game. Nat News 2016;535(7611):201.
11. Campbell DT. Assessing the impact of planned social change. Eval program plann 1979;2(1):67-90.
12. Newton AC. Implications of Goodharts Law for monitoring global biodiversity loss. Conserv Lett 2011;4(4):264-68.
13. Mizen P. Central Banking, Monetary Theory and Practice: Essays in honour of Charles Goodhart, vol. 1. Edward Elgar; 2003.
14. Chrystal KA, Mizen PD, Mizen P. Goodharts Law: Its origins, meaning and implications for monetary policy. . In: Central Banking, Monetary Theory and Practice: Essays in honour of Charles Goodhart. Edward Elgar, 2003: 221-43.
15. Francescani C. NYPD report confirms manipulation of crime stats. Reuters 2012, . https://www.reuters.com/article/us-crime-newyork-statistics/nypd-report-confirms-manipulation-of-crime-stats-idUSBRE82818620120309. Accessed 12 May 2019.
16. Kliger AS. Quality measures for dialysis: Time for a balanced scorecard. Clini J Am Soc Nephrol 2016;11(2):363-68.
17. Šupak Smolčic V. Salami publication: Definitions and examples. Biochem Med (Zagreb) 2013;23(3):237-41.
18. Schofferman J, Wetzel FT, Bono C. Ghost and guest authors: You cant always trust who you read. Pain Med 2015;16(3):416-20.
19. Head ML, Holman L, Lanfear R et al. The extent and consequences of p-hacking in science. PLoS Biol 2015;13(3):e1002106.
20. Bartneck C, Kokkelmans S. Detecting h-index manipulation through self-citation analysis. Scientometrics 2010;87(1):85-98.
21. Kupferschmidt K. Tide of lies. Science 2018;361(6403):636-41.
22. Haug CJ. Peer-reviewfraud-hacking the scientific publication process. N Engl J Med 2015;373(25):2393-95.
23. Dansinger M. Dear plagiarist: A letter to a peer reviewer who stole and published our manuscript as his own. Ann Intern Med 2017;166(2):143-143.
24. Post A, Li AY, Dai JB, et al. c-index and subindices of the hindex: New variants of the h-index to account for variations in author contribution. Cureus 2018;10(5):e2629.
25. Romanovsky AA. Revised h index for biomedical research. Cell Cycle 2012;11(22):4118-21.
26. Wu Q. The w-index: A measure to assess scientific impact by focusing on widely cited papers. J Am Soc Inf Sci Technol 2010;61(3):609-14.
27. Waltman L, van Eck NJ, van Leeuwen TN, et al. Towards a new crown indicator: An empirical analysis. Scientometrics 2011;87(3):467-81.
28. Top publications. Google Scholar.https://scholar.google.com/citations?view op=top venues. Accessed 13 February 2019.
29. Journal Citation Reports (JCR). https://jcr.clarivate.com/. Accessed 12 May 2019.
30. Fortunato S, BergstromCT, Borner K, et al. Science of science. Science 2018;359(6379):eaao0185.
31. arXiv Monthly Submission Rates. https://arxiv.org/stats/monthly submissions. Accessed 20 January 2019.
32. Learn JR. What bioRxivs first 30, 000 preprints reveal about biologists. Nat News 2019, doi:10.1038/d41586-019-00199-6.
33. Davis P. Scientific Reports overtakes PLoS One as largest megajournal. The Scholarly Kitchen. https://scholarlykitchen.sspnet.org/2017/04/06/scientific-reports-overtakes-plos-one-As-largest-megajournal/. Accessed 9 July 2018.
34. Cronin B. Hyperauthorship: A postmodern perversion or evidence of a structural shift in scholarly communication practices? J Am Soc Inf Sci Technol 2001;52(7):58-69.
35. Von Bergen C, Bressler MS. Academes unspoken ethical dilemma: Author inflation in higher education. Res High Educ 2017;32.
36. Mallapaty S. Paper authorship goes hyper author. 2018. Nature Index. https://www.natureindex.com/news-blog/paper-Authorship-goes-hyper.
37. Abbott BP, Abbott R, Abbott T, et al. Observation of gravitational waves from a binary black hole merger. Phys Rev Lett 2016;116(6):061102.
38. Castelvecchi D. LIGOs unsung heroes. Nature News 2017. https://www.nature.com/news/ligo-s-unsung-heroes-1.22786.
39. Aboukhalil R. The rising trend in authorship. TheWinnower 2014;2:e141832, doi:10.15200/winn.141832.26907.
40. Wislar JS, Flanagin A, Fontanarosa PB, et al. Honorary and ghost authorship in high impact biomedical journals: A cross sectional survey. BMJ 2011;343:d6128.
41. Kennedy MS, Barnsteiner J, Daly J. Honorary and ghost authorship in nursing publications. J Nurs Scholarsh 2014;46(6):416-22.
42. Vera-Badillo FE, Napoleone M, Krzyzanowska MK, et al. Honorary and ghost authorship in reports of randomised clinical trials in oncology. Eur J Cancer 2016;66:1-8.
43. Economist T. Why research papers have so many authors. The Economist 2016. http://www.economist.com/news/science-And-Technology/21710792-scientific-publicationsare-getting-more-And-more-names-Attached-Them-why.Accessed 12 May 2019.
44. Lewison G, Hartley J. Whats in a title? Numbers ofwords and the presence of colons. Scientometrics 2005;63(2):341-56.
45. Lockwood G. Academic clickbait: Articles with positivelyframed titles, interesting phrasing, and no wordplay get more attention online. The Winnower 2016;3, doi:10.15200/winn.146723.36330.
46. Ucar I, L opez-Fernandino F, Rodriguez-Ulibarri P, et al. Growth in the number of references in engineering journal papers during the 1972-2013 period. Scientometrics 2014;98(3):1855-64.
47. G alvez A, Maqueda M, Martinez-Bueno M, et al. Scientific publication trends and the developing world: What can the volume and authorship of scientific articles tell us about scientific progress in various regions? American Sci 2000;88(6):526-33.
48. Jagsi R, Guancial EA, Worobey CC, et al. The "gender gap" in authorship of academic medical literature-A 35-year perspective. N Engl J Med 2006;355(3):281-87.
49. Johnson MR, Wagner NJ, Reusch J. Publication trends in toptier journals in higher education. J Appl Res Higher Educ 2016;8(4):439-54.
50. Aldhous P. Scientific publishing: The inside track. Nat News 2014;510(7505):330.
51. Porter A, Rafols I. Is science becoming more interdisciplinary? Measuring and mapping six research fields over time. Scientometrics 2009;81(3):719-45.
52. Fanelli D, Larivière V. Researchers individual publication rate has not increased in a century. PLoS One 2016;11(3):e0149504.
53. Dong Y, Ma H, Shen Z, et al. A century of science: Globalization of scientific collaborations, citations, and innovations. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining ACM; 2017:1437-46.
54. Yan E, Ding Y. Weighted citation: An indicator of an articles prestige. J Am Soc Inf Sci Technol 2010;61(8):1635-43.
55. Lehmann S, Jackson AD, Lautrup BE.Measures for measures. Nature 2006;444(7122):1003.
56. Garfield E. The meaning of the impact factor. Rev Int Psicol Clin Salud 2003;3(2):363-9.
57. Hirsch JE. Does the h index have predictive power? Proc Natl Acad Sci U S A 2007;104(49):19193-8.
58. Fong EA, Wilhite AW. Authorship and citation manipulation in academic research. PloS One 2017;12(12):e0187394.
59. The cost of salami slicing. Nat Mater 2005;4, doi:10.1038/nmat1305.
60. Delgado L opez-C ozar E, Robinson-Garcia N, Torres-Salinas D. The Google scholar experiment: How to index false papers and manipulate bibliometric indicators. J Assoc Inf Sci Technol 2014;65(3):446-54.
61. Van Bevern R, Komusiewicz C, Niedermeier R, et al. H-index manipulation by merging articles:Models, theory, and experiments. Artif Intell 2016;240:19-35.
62. Falagas ME, Kouranos VD, Arencibia-Jorge R, et al. Comparison of SCImago journal rank indicator with journal impact factor. FASEB J 2008;22(8):2623-28.
63. Lariviere V, Kiermer V, MacCallum CJ, et al. A simple proposal for the publication of journal citation distributions. BioRxiv 2016, doi:10.1101/062109.
64. Callaway E. Beat it, impact factor! Publishing elite turns against controversial metric. Nat News 2016;535(7611):210.
65. Altmetric. https://www.altmetric.com/. Accessed 14 February 2019.
66. Griffin SA, Oliver CW, Murray A. Altmetrics! Can you afford to ignore it? Br J Sports Med 2017;52(18):1160-1.
67. Semantic Scholar. https://www.semanticscholar.org. Accessed 14 February 2019.
68. Seglen PO. Why the impact factor of journals should not be used for evaluating research. BMJ 1997;314(7079):498.
69. Byrne A. Comment: Measure for measure. Nature 2017;546(7666):S22.
70. Hecht F, Hecht BK, Sandberg AA. The journal "impact factor": Amisnamed, misleading, misused measure. Cancer Genetics Cytogenet 1998;104(2):77-81.
71. Google Scholar. https://scholar.google.com. Accessed 13 February 2019.
72. Sinha A, Shen Z, Song Y, et al. An overview of Microsoft Academic Service (MAS) and applications. In: Proceedings of the 24th International Conference onWorldWideWeb ACM; 2015:243-46, doi:10.1145/2740908.2742839.
73. KDD Cup 2016: Whose papers are accepted themost: Towards measuring the impact of research institutions. 2016. http://www.kdd.org/kdd-cup/view/kdd-cup-2016/Data.
74. Semantic Scholar-An Overview of Microsoft Academic Service (MAS) and Applications. https://www.semanticscholar.org/paper/An-Overview-of-Microsoft-Academic-Service-(MAS)-And-Sinha-Shen/b6b6d2504fd57d27a0467654fa62169cc7dedbdd?navId=citing-papers. Accessed 14 February 2019.
75. Pitts M, Savvana S, Roy SB, et al. ALIAS: Author Disambiguation in Microsoft Academic Search Engine Dataset. In: Proceedings of the 17th International Conference on Extending Database Technology (EDBT 2014); 2014, doi:10.5441/002/edbt.2014.65.
76. How Microsoft Academic uses knowledge to address the problem of conflation/disambiguation. 2018. https://www.microsoft.com/en-us/research/project/academic/articles/microsoft-Academic-uses-knowledge-Address-problem-conflation-disambiguation/. Accessed 21 January 2019.
77. Tang J, Zhang J, Yao L, et al. ArnetMiner: extraction and mining of academic social networks. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Las Vegas, NV, 2008. 2008, doi:10.1145/1401890.1402008.
78. Butler D. Free journal-ranking tool enters citation market. Nature 2008;451:6.
79. Scimago Journal & Country Rank. https://www.scimagojr.com/journalrank.php. Accessed 14 February 2019.
80. Pycld2-Python Package. https://pypi.org/project/pycld2/. Accessed 14 February 2019.
81. Colavizza G, Franssen T, van Leeuwen T. An empirical investigation of the tribes and their territories: Are research specialisms rural and urban? J Informetr 2019;13(1):105-17.
82. Nature: Editorial Criteria and Processes. https://www.nature.com/nature/for-Authors/editorial-criteria-And-processes. Accessed 15 July 2018.
83. Low Y, Gonzalez JE, Kyrola A, et al. Graphlab: A new framework for parallel machine learning. arXiv 2014:1006.4990.
84. MongoDB. http://www.mongodb.com. Accessed 14 February 2019.
85. Fire M, Guestrin C. Supporting data for "Over-optimization of academic publishing metrics: observing Goodharts Law in action." GigaScience Database 2019. http://dx.doi.org/10.5524/100587.