Statistical modelling of citation exchange between statistics journals

Journal of the Royal Statistical Society. Series A: Statistics in Society

Volumn 179, Issue 1, 2016, Pages 1-63

  Varin, Cristiano ^a   Cattelan, Manuela ^b   Firth, David ^c  

^a CA' FOSCARI UNIVERSITY OF VENICE   (Italy)

^b UNIVERSITY OF PADOVA   (Italy)

^c UNIVERSITY OF WARWICK   (United Kingdom)

Author keywords

Bradley Terry model; Citation data; Export score; Impact factor; Journal ranking; Research evaluation; Stigler model

Indexed keywords

EID: 84955176281     PISSN: 09641998     EISSN: 1467985X     Source Type: Journal    
DOI: 10.1111/rssa.12124     Document Type: Article

References (141)

1. Adie, E. and Roe, W. (2013) Altmetric: enriching scholarly content with article-level discussion and metrics. Learnd Publish., 26, 11-17.
2. Adler, R., Ewing, J. and Taylor, P. (2009) Citation statistics (with discussion). Statist. Sci., 24, 1-14.
3. Agresti, A. (2013) Categorical Data Analysis, 3rd edn. New York: Wiley.
4. Alberts, B. (2013) Impact factor distortions. Science, 340, 787.
5. Amin, M. and Mabe, M. (2000) Impact factors: use and abuse. Perspect. Publish., 1, 1-6.
6. Archambault, E. and Larivière, V. (2009) History of the journal impact factor: contingencies and consequences. Scientometrics, 79, 635-649.
7. Arnold, D. N. and Fowler, K. K. (2011) Nefarious numbers. Not. Am. Math. Soc., 58, 434-437.
8. Bergstrom, C. (2007) Eigenfactor: measuring the value and the prestige of scholarly journals. Coll. Res. Lib. News, 68, 314-316.
9. Bishop, M. and Bird, C. (2007) BIB's first impact factor is 24.37. Brief. Bioinform., 8, 207.
10. Bollen, J., Rodriguez, M. A. and de Sompel, H. V. (2006) Journal status. Scientometrics, 69, 669-687.
11. Bornmann, L. (2014) Do altmetrics point to the broader impact of research?: an overview of benefits and disadvantages of altmetrics. J. Informetr., 8, 895-903.
12. Bornmann, L. and Marx, W. (2014) How to evaluate individual researchers working in the natural and life sciences meaningfully?: a proposal of methods based on percentiles of citations. Scientometrics, 98, 487-509.
13. Boyack, K. W., Klavans, R. and Börner, K. (2005) Mapping the backbone of science. Scientometrics, 64, 351-374.
14. Bradley, R. A. and Terry, M. E. (1952) The rank analysis of incomplete block designs: I, The method of paired comparisons. Biometrika, 39, 324-345.
15. Braun, T., Glänzel, W. and Schubert, A. (2006) A Hirsch-type index for journals. Scientometrics, 69, 169-173.
16. Brin, S. and Page, L. (1998) The anatomy of a large-scale hypertextual web search engine. Comput. Netwrks ISDN Syst., 30, 107-117.
17. Carpenter, M. P. and Narin, F. (1973) Clustering of scientific journals. J. Am. Soc. Inform. Sci., 24, 425-436.
18. Cattelan, M. (2012) Models for paired comparison data: a review with emphasis on dependent data. Statist. Sci., 27, 412-433.
19. Cattelan, M., Varin, C. and Firth, D. (2013) Dynamic Bradley-Terry modelling of sports tournaments. Appl. Statist., 62, 135-150.
20. Chen, K.-M, Jen, T.-H. and Wu, M. (2014) Estimating the accuracies of journal impact factor through bootstrap. J. Inform., 8, 181-196.
21. David, H. A. (1963) The Method of Paired Comparisons. New York: Hafner.
22. Fahrmeir, L. and Tutz, G. (1994) Dynamic stochastic models for time-dependent ordered paired comparison systems. J. Am. Statist. Ass., 89, 1438-1449.
23. Fan, J. and Li, R. (2001) Variable selection via nonconcave penalized likelihood and its oracle properties. J. Am. Statist. Ass., 96, 1348-1360.
24. Firth, D. (2012) qvcalc: quasi variances for factor effects in statistical models. R Package Version 0.8-8. (Available from CRAN.Rproject.org/package=qvcalc.)
25. Firth, D. and de Menezes, R. X. (2005) Quasi-variances. Biometrika, 91, 65-80.
26. Franceschet, M. (2010) Ten good reasons to use the Eigenfactor metrics. Inform. Process. Mangmnt, 46, 555-558.
27. Frandsen, T. F. (2007) Journal self-citations-analysing the JIF mechanism. J. Informetr., 1, 47-58.
28. Garfield, E. (1955) Citation indices for Science. Science, 122, 108-111.
29. Garfield, E. (1972) Citation analysis as a tool in journal evaluation. Science, 178, 471-479.
30. Glänzel, W. and Moed, H. F. (2002) Journal impact measures in bibliometric research. Scientometrics, 53, 171-193.
31. Glickman, M. E. (1999) Parameter estimation in large dynamic paired comparison experiments. Appl. Statist., 48, 377-394.
32. Goldstein, H. and Spiegelhalter, D. J. (1996) League tables and their limitations: statistical issues in comparisons of institutional performance (with discussion). J. R. Statist. Soc. A, 159, 385-443.
33. Gross, P. L. K. and Gross, E. M. (1927) College libraries and chemical education. Science, 66, 385-389.
34. Hall, P. G. (2009) Comment: Citation statistics. Statist. Sci., 24, 25-26.
35. Hall, P. G. (2011) 'Ranking our excellence, ' or 'assessing our quality, ' or whatever.... Inst. Math. Statist. Bull., 40, 12-14.
36. Hall, P. and Miller, H. (2009) Using the bootstrap to quantify the authority of an empirical ranking. Ann. Statist., 37, 3929-3959.
37. Hall, P. and Miller, H. (2010) Modeling the variability of rankings. Ann. Statist., 38, 2652-2677.
38. Institute of Electrical and Electronics Engineers Board of Directors (2013) IEEE position statement on 'Appropriate use of bibliometric indicators for the assessment of journals, research proposals, and individuals'. Institute of Electrical and Electronics Engineers.
39. Journal-Ranking.com (2007) Present Ranking Endeavors. Red Jasper. (Available from www.journal-ranking.com/ranking/web/content/intro.html.)
40. Jump, P. (2014) Light dose of metrics could ease REF pain. Times Higher Educ., no. 2178, Nov. 13th, 11. (Available from www.timeshighereducation.co.uk/news/regular-diet-of-metrics-lite-may-make-full-ref-more-palatable/2016912.article.)
41. Kaufman, L. and Rousseeuw, P. J. (1990) Finding Groups in Data: an Introduction to Cluster Analysis. Hoboken: Wiley.
42. Knorr-Held, L. (2000) Dynamic rating of sports teams. Statistician, 49, 261-276.
43. Lehmann, S., Lautrup, B. E. and Jackson, A. D. (2009) Comment: Citation statistics. Statist. Sci., 24, 17-20.
44. Leydesdorff, L. (2004) Clusters and maps of science based on bi-connected graphs in Journal Citation Reports. J. Documentn, 60, 371-427.
45. Leydesdorff, L. and Bornmann, L. (2011) How fractional counting of citations affects the impact factor: normalization in terms of differences in citation potentials among fields of science. J. Am. Soc. Inform. Sci. Technol., 62, 217-229.
46. Leydesdorff, L. and Opthof, T. (2010) Scopus' Source Normalized Impact per Paper (SNIP) versus the Journal Impact Factor based on fractional counting of citations. J. Am. Soc. Inform. Sci. Technol., 61, 2365-2369.
47. Leydesdorff, L., Radicchi, F., Bornmann, L., Castellano, C. and de Nooy, W.(2013) Field-normalized impact factors (IFs): a comparison of rescaling and fractionally counted IFs. J. Am. Soc. Inform. Sci. Technol., 64, 2299-2309.
48. Liner, G. H. and Amin, M. (2004) Methods of ranking economics journals. Atl. Econ. J., 32, 140-149.
49. Liu, X., Glänzel, W. and De Moor, B. (2012) Optimal and hierarchical clustering of large-scale hybrid networks for scientific mapping. Scientometrics, 91, 473-493.
50. Marx, W. and Bornmann, L. (2013) Journal impact factor: 'the poor man's citation analysis' and alternative approaches. Eur. Sci. Editng, 39, 62-63.
51. Masarotto, G. and Varin, C. (2012) The ranking lasso and its application to sport tournaments. Ann. Appl. Statist., 6, 1949-1970.
52. Moed, H. F. (2010) Measuring contextual citation impact of scientific journals. J. Informetr., 4, 265-277.
53. Morris, C. N. (1983) Parametric empirical Bayes inference: theory and applications. J. Am. Statist. Ass., 78, 47-65.
54. van Nierop, E. (2009) Why do statistics journals have low impact factors? Statist. Neerland., 63, 52-62.
55. Nocedal, J. and Wright, S. J. (2006) Numerical Optimization, 2nd edn. New York: Springer.
56. van Noorden, R. (2012) Researchers feel pressure to cite superfluous papers. Nat. News, Feb. 12th.
57. Palacios-Huerta, I. and Volij, O. (2004) The measurement of intellectual influence. Econometrica, 72, 963-977.
58. Pratelli, L., Baccini, A., Barbaresi, L. and Marcheselli, M. (2012) Statistical analysis of the Hirsch Index. Scand. J. Statist., 39, 681-694.
59. Putirka, K., Kunz, M., Swainson, I. and Thomson, J. (2013) Journal Impact Factors: their relevance and their influence on society-published scientific journals. Am. Mineral., 98, 1055-1065.
60. R Core Team (2015) R: a Language and Environment for Statistical Computing. Vienna: R Foundation for Statistical Computing.
61. Ritzberger, K. (2008) A ranking of journals in economics and related fields. Germ. Econ. Rev., 9, 402-430.
62. Seglen, P. O. (1997) Why the impact factor of journals should not be used for evaluating research. Br. Med. J., 314, 498-502.
63. Sevinc, A. (2004) Manipulating impact factor: an unethical issue or an editor's choice? Swiss Med. Wkly, 134, 410.
64. Silverman, B. W. (2009) Comment: Citation statistics. Statist. Sci., 24, 21-24.
65. Spiegelhalter, D. J., Best, N. G., Carlin, B. P. and van der Linde, A. (2002) Bayesian measures of model complexity and fit (with discussion). J. R. Statist. Soc. B, 64, 583-639.
66. Stigler, S. M. (1994) Citation patterns in the journals of statistics and probability. Statist. Sci., 9, 94-108.
67. Stigler, G. J., Stigler, S. M. and Friedland, C. (1995) The journals of economics. J. Polit. Econ., 103, 331-359.
68. Takeuchi, K. (1976) Distribution of informational statistics and a criterion of model fitting (in Japanese). Suri-Kagaku, 153, 12-18.
69. Theoharakis, V. and Skordia, M. (2003) How do statisticians perceive statistics journals? Am. Statistn, 57, 115-123.
70. Tibshirani, R. (1996) Regression shrinkage and selection via the lasso. J. R. Statist. Soc. B, 58, 267-288.
71. Tibshirani, R., Saunders, M., Rosset, S., Zhu, J. and Kneight, K. (2005) Sparsity and smoothness via the fused lasso. J. R. Statist. Soc. B, 67, 91-108.
72. Turner, H. and Firth, D. (2012) Bradley-Terry models in R: the BradleyTerry2 package. J. Statist. Softwr., 48, 1-21.
73. Waltman, L. and Van Eck, N. J. (2013) Source normalized indicators of citation impact: an overview of different approaches and an empirical comparison. Scientometrics, 96, 699-716.
74. Waltman, L., van Eck, J. N., van Leeuwen, T. N. and Visser, M. S. (2013) Some modifications to the SNIP journal impact indicator. J. Informetr., 7, 272-285.
75. Wedderburn, R. W. M. (1974) Quasi-likelihood, generalized linear models, and the Gauss-Newton method. Biometrika, 61, 439-447.
76. West, J. D. (2010) Eigenfactor: ranking and mapping scientific knowledge. PhD Dissertation. University of Washington, Seattle.
77. Wilhite, A. W. and Fong, E. A. (2012) Coercive citation in academic publishing. Science, 335, 542-543.
78. Zitt, M. and Small, H. (2008) Modifying the journal impact factor by fractional citation weighting: the audience factor. J. Am. Soc. Inform. Sci. Technol., 59, 1856-1860.
79. Zou, H. (2006) The adaptive lasso and its oracle properties. J. Am. Statist. Ass., 101, 1418-1429.
80. Agresti, A. (2013) Categorical Data Analysis, 3rd edn. New York: Wiley.
81. Alberts, B. (2013) Impact factor distortions. Science, 340, 787.
82. American Society for Cell Biology (2015) San Francisco Declaration on Research Assessment (DURA). American Society for Cell Biology. (Available from http://am.ascb.org/dora/.)
83. Bartolucci, F., Dardanoni, V. and Peracchi, F. (2015) Ranking scientific journals via latent class models for polytomous item response data. J. R. Statist. Soc. A, 178, 1025-1049.
84. Batista, P. D., Campiteli, M. G. and Kinouchi, O. (2006) Is it possible to compare researchers with different scientific interests? Scientometrics, 68, 179-189.
85. Benjamini, Y. and Hochberg, Y. (1995) Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Statist. Soc. B, 57, 289-300.
86. Bien, J. and Tibshirani, R. (2011) Hierarchical clustering with prototypes via minimax linkage. J. Am. Statist. Ass., 106, 1075-1084.
87. Bornmann, L. and Daniel, H.-D. (2008) What do citation counts measure?: a review of studies on citing behavior. J. Documentn, 64, 45-80.
88. Boyack, K. W., Klavans, R. and Börner, K. (2005) Mapping the backbone of science. Scientometrics, 64, 351-374.
89. Carter, D. and Spirling, A. (2008) Under the influence?: intellectual exchange in political science. Polit. Sci. Polit., 41, 375-378.
90. Case, D. O. and Higgins, G. M. (2000) How can we investigate citation behavior?: a study of reasons for citing literature in communication. J. Am. Soc. Inform. Sci., 51, 635-645.
91. Cattelan, M. and Varin, C. (2013) Hybrid pairwise likelihood analysis of animal behavior experiments. Biometrics, 69, 1002-1011.
92. Chatterjee, S., Diaconis, P. and Sly, A. (2011) Random graphs with a given degree sequence. Ann. Appl. Probab., 21, 1400-1435.
93. Cho, H. and Fryzlewicz, P. (2011) Multiscale interpretation of taut string estimation and its connection to Unbalanced Haar wavelets. Statist. Comput., 21, 671-681.
94. Cocchi, D., Cavaliere, G., Freo, M., Giannerini, S., Mazzocchi, M., Trivisano, C. and Viroli, C. (2014) A support for classifying scientific papers in a University Department. Proced. Econ. Finan., 17, 47-54.
95. Colquhoun, D. (2003) Challenging the tyranny of impact factors. Nature, 423, 479.
96. Colquhoun, D. (2007) How should universities be run to get the best out of people? University College London, London. (Available from http://www.dcscience.net/2007/08/03/how-should-universities-be-run-to-get-the-best-out-of-people/.)
97. Colquhoun, D. (2014a) Should metrics be used to assess research performance?: a submission to HEFCE. University College London, London. (Available from http://www.dcscience.net/2014/06/18/should-metrics-be-used-to-assess-research-performance-a-submission-to-hefce/.)
98. Colquhoun, D. (2014b) Publish and perish at Imperial College London: the death of Stefan Grimm. University College London, London. (Available from http://www.dcscience.net/2014/12/01/publish-and-perish-at-imperial-college-london-the-death-of-stefan-grimm/.)
99. Colquhoun, D. and Hawkes, A. G. (1995) The principles of the stochastic interpretation of ion channel mechanisms. In Single Channel Recording eds B. Sakmann and E. Nehe ), pp. 397-482. New York: Plenum.
100. Colquhoun, D., Hawkes, A. G. and Srodzinski, K. (1996) Joint distributions of apparent open times and shut times of single ion channels and the maximum likelihood fitting of mechanisms. Philos. Trans. R. Soc. Lond. A, 354, 2555-2590.
101. Davidian, M. (2013) Transforming our culture of peer review. Amstat News, 435, 3-4.(Available from http://magazine.amstat.org/blog/2013/09/01/peerreview/.)
102. Firth, D. (2009) Bradley-Terry models in R. J. Statist. Softwr., 12, 1-12.
103. Franceschet, M. (2012) The large-scale structure of journal citation networks. J. Am. Soc. Inform. Sci. Technol., 63, 837-842.
104. Gelman, A., Pasarica, C. and Dodhai, R. (2002) Let's practice what we preach: turning tables into graphs. Am. Statistn, 56, 121-130.
105. Hawkes, A. G. (1971) Spectra of some self-exciting and mutually exciting point processes. Biometrika, 58, 83-90.
106. Hicks, D., Wouters, P., Waltman, L., de Rijcke, S. and Rafols, I. (2015) Bibliometrics: the Leiden manifesto for research metrics. Nature, 520, 429-431.
107. Hoff, P. D., Raftery, A. E. and Handcock, M. S. (2002) Latent space approaches to social network analysis. J. Am. Statist. Ass., 97, 1090-1098.
108. Hull, D. L. (1988) Science as a Process: an Evolutionary Account of the Social and Conceptual Development of Science. Chicago: University of Chicago Press.
109. Ji, P. and Jin, J. (2014) Coauthorship and citation networks for statisticians. Preprint arXiv:1410.2840.
110. Jin, J. (2015) Fast community detection by SCORE. Ann. Statist., 43, 57-89.
111. Kass, R. E. and Raftery, A. E. (1995) Bayes factors. J. Am. Statist. Ass., 90, 773-795.
112. Ke, T., Fan, J. and Wu, Y. (2015) Homogeneity pursuit. J. Am. Statist. Ass., 110, 175-194.
113. Ke, Q., Ferrara, E., Radicchi, F. and Flammini, A. (2015) Defining and identifying sleeping beauties in science. Proc. Natn. Acad. Sci. USA, 112, 7426-7431.
114. Krivitsky, P. N. (2012) Exponential-family random graph models for valued networks. Electron. J. Statist., 6, 1100-1128.
115. Krivitsky, P. N. and Handcock, M. S. (2008) Fitting latent cluster models for social networks with latentnet. J. Statist. Softwr., 24, no.5.
116. Krivitsky, P. N. and Handcock, M. S. (2015) . R Package Version 2.7.0. Statnet Project, University of Wollongong, Wollongong.
117. Krivitsky, P. N., Handcock, M. S., Raftery, A. E. and Hoff, P. D. (2009) Representing degree distributions, clustering, and homophily in social networks with latent cluster random effect models. Socl Netwrks, 31, 204-213.
118. Lancho-Barrantes, B. S., Guerrero-Bote, V. and de Moya-Anégon, F. (2010) What lies behind the averages and significance of citation indicators in different disciplines? J. Inform. Sci., 36, 371-382.
119. Marchetti, Y. and Zhou, Q. (2014) Solution path clustering with adaptive concave penalty. Electron. J. Statist., 8, 1569-1603.
120. Masarotto, G. and Varin, C. (2012) The ranking lasso and its application to sports tournaments. Ann. Appl. Statist., 6, 1949-1970.
121. Murtagh, F. and Kurtz, M. J. (2016) The Classification Society's bibliography over four decades: history and current analysis. J. Classificn, 33, in the press.
122. Mustafee, N., Bessis, N., Taylor, S. J. E. and Sotiriadis, S. (2013) Exploring the e-science knowledge base through co-citation analysis. Proced. Comput. Sci., 19, 586-593.
123. Ning, Y. and Liu, H. (2014) A general theory of hypothesis tests and confidence regions for sparse high dimensional models. Preprint arXiv:1412.8765.
124. van Noorden, R. (2013) Brazilian citation scheme outed. Nature, 500, 510-511.
125. van Noorden, R., Maher, B. and Nuzzo, R. (2014) The top 100 papers. Nature, 514, 550-553.
126. Nowicki, K. and Snijders, T. A. B. (2001) Estimation and prediction for stochastic blockstructures. J. Am. Statist. Ass., 96, 1077-1087.
127. Pan, W., Shen, X. and Liu, B. (2013) Cluster analysis: unsupervised learning via supervised learning with a non-convex penalty. J. Mach. Learn. Res., 14, 1865-1889.
128. Rasch, G. (1961) On general laws and the meaning of measurement in psychology. In Proc. 4th Berkeley Symp. Mathematical Statistics and Probability (ed. J. Neyman), vol., pp. 321-333.Berkeley: University of California Press.
129. Rojas, C. and Wahlberg, B. (2014) On change point detection using the fused lasso method. Preprint.
130. Seglen, P. O. (1997) Why the impact factor of journals should not be used for evaluating research. Br. Med. J., 314, 498-502.
131. Simons, G. and Yao, Y.-C. (1999) Asymptotics when the number of parameters tends to infinity in the Bradley-Terry model for paired comparisons. Ann. Statist., 27, 1041-1060.
132. Socher, R. (2014) Recursive deep learning for natural language processing and computer vision. PhD Thesis. Computer Science Department, Stanford University, Stanford.
133. Spiegelhalter, D. J., Best, N. G., Carlin, B. P. and van der Linde, A. (2002) Bayesian measures of model complexity and fit (with discussion). J. R. Statist. Soc. B, 64, 583-639.
134. Stigler, S. M. (1978) Laplace's early work: chronology and citations. Isis, 69, 234-254.
135. Stigler, S. M. (1994) Citation patterns in the journals of statistics and probability. Statist. Sci., 9, 94-108.
136. Tibshirani, R. J., Rinaldo, A., Tibshirani, R. and Wasserman, L. (2015) Uniform asymptotic inference and the bootstrap after model selection. Preprint arXiv:1506.06266.
137. Times Higher Education (2015) World university rankings 2013-2014 methodology. Times Higher Educ.
138. Wang, F. (2012) Development of joint estimating equation approaches to merging clustered or longitudinal datasets from multiple biomedical studies. PhD Dissertation. Department of Biostatistics, University of Michigan, Ann Arbor.
139. Yan, T., Leng, C. and Zhu, J. (2015) Asymptotics in directed exponential random graph models with an increasing bi-degree sequenceAnn. Statist., to be published.
140. Yan, T. and Xu, J. (2013) A central limit theorem in the β-model for undirected random graphs with a diverging number of vertices. Biometrika, 100, 519-524.
141. Zhoa, Y., Levina, E. and Zhu, J. (2011) Community extraction for social networks. Proc. Natn. Acad. Sci. USA, 108, 7321-7326.