Alternative metrics in scientometrics: A meta-analysis of research into three altmetrics

Scientometrics

Volumn 103, Issue 3, 2015, Pages 1123-1144

  Bornmann, Lutz ^a  

^a MAX PLANCK INSTITUTE   (Germany)

Author keywords

Altmetrics; Blogging; Mendeley; Meta analysis; Microblogging; Online reference managers; Twitter

Indexed keywords

EID: 85027929276     PISSN: 01389130     EISSN: 15882861     Source Type: Journal    
DOI: 10.1007/s11192-015-1565-y     Document Type: Article

References (89)

1. Bar-Ilan, J. (2012a). JASIST 2001–2010. Bulletin of the American Society for Information Science and Technology, 38(6), 24–28. doi:10.1002/bult.2012.1720380607.
2. Bar-Ilan, J. (2012b). JASIST@mendeley. Paper presented at the altmetrics12: An ACM Web Science Conference 2012 Workshop, Evanston, IL, USA.
3. Bar-Ilan, J., Haustein, S., Peters, I., Priem, J., Shema, H., & Terliesner, J. (2012). Beyond citations: Scholars’ visibility on the social Web. In E. Archambault, Y. Gingras, & V. Lariviere (Eds.), Proceedings of the 17th international conference on science and technology indicators (pp. 98–109). Montreal, QC: Science-Metrix and OST.
4. Bar-Ilan, J., Shema, H., & Thelwall, M. (2014). Bibliographic references in Web 2.0. In B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics: Harnessing multi-dimensional indicators of performance (pp. 307–325). Cambridge, MA: MIT Press.
5. Bik, H. M., & Goldstein, M. C. (2013). An introduction to social media for scientists. PLoS Biology, 11(4), e1001535. doi:10.1371/journal.pbio.1001535.
6. Bonetta, L. (2007). Scientists enter the blogosphere. Cell, 129(3), 443–445.
7. Börner, K., Sanyal, S., & Vespignani, A. (2007). Network science. Annual Review of Information Science and Technology, 41(1), 537–607. doi:10.1002/aris.2007.1440410119.
8. Bornmann, L. (2012). Measuring the societal impact of research. EMBO Reports, 13(8), 673–676.
9. Bornmann, L. (2013). What is societal impact of research and how can it be assessed? A literature survey. Journal of the American Society of Information Science and Technology, 64(2), 217–233.
10. Bornmann, L. (2014). Validity of altmetrics data for measuring societal impact: A study using data from Altmetric and F1000Prime. Journal of Informetrics, 8(4), 935–950.
11. Bornmann, L., & Daniel, H.-D. (2008). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64(1), 45–80. doi:10.1108/00220410810844150.
12. Bornmann, L., Leydesdorff, L., & Mutz, R. (2013). The use of percentiles and percentile rank classes in the analysis of bibliometric data: Opportunities and limits. Journal of Informetrics, 7(1), 158–165.
13. Bornmann, L., Stefaner, M., de Moya Anegón, F., & Mutz, R. (2014). What is the effect of country-specific characteristics on the research performance of scientific institutions? Using multi-level statistical models to rank and map universities and research-focused institutions worldwide. Journal of Informetrics, 8(3), 581–593. doi:10.1016/j.joi.2014.04.008.
14. Bradburn, M. J., Deeks, J. J., & Altman, D. G. (1998). Metan—An alternative meta-analysis command. Stata Technical Bulletin, 44, 4–15.
15. Cheung, M. W. L. (2014). Modeling dependent effect sizes with three-level meta-analyses: A structural equation modeling approach. Psychological Methods, 19(2), 211–229. doi:10.1037/A0032968.
16. Colledge, L. (2014). Snowball metrics recipe book. Amsterdam: Snowball Metrics program partners.
17. Colson, V. (2011). Science blogs as competing channels for the dissemination of science news. Journalism, 12(7), 889–902. doi:10.1177/1464884911412834.
18. Costas, R., Zahedi, Z., & Wouters, P. (2014). Do altmetrics correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. Retrieved February 11, from http://arxiv.org/abs/1401.4321
19. Darling, E. S., Shiffman, D., Cote, I. M., & Drew, J. A. (2013). The role of Twitter in the life cycle of a scientific publication. PeerJ PrePrints, 1, e16v11. doi:10.7287/peerj.preprints.16v1.
20. de Bellis, N. (2014). History and evolution of (biblio)metrics. In B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics: Harnessing multi-dimensional indicators of performance (pp. 23–44). Cambridge, MA: MIT Press.
21. Dinsmore, A., Allen, L., & Dolby, K. (2014). Alternative perspectives on impact: The potential of ALMs and altmetrics to inform funders about research impact. PLoS Biology, 12(11), e1002003. doi:10.1371/journal.pbio.1002003.
22. Duggan, M., & Smith, A. (2014). Social Media Update 2013. Retrieved March 28, from http://pewinternet.org/Reports/2013/Social-Media-Update.aspx
23. Eagly, A. H. (2005). Refereeing literature review submissions to journals. In R. J. Sternberg (Ed.), Reviewing scientific works in psychology. Washington, DC: American Psychological Association (APA).
24. Eysenbach, G. (2011). Can tweets predict citations? Metrics of social impact based on Twitter and correlation with traditional metrics of scientific impact. J Med Internet Res, 13(4), e123.
25. Fausto, S., Machado, F. A., Bento, L. F. J., Iamarino, A., Nahas, T. R., & Munger, D. S. (2012). Research blogging: Indexing and registering the change in Science 2.0. PLoS ONE, 7(12), e50109. doi:10.1371/journal.pone.0050109.
26. Galloway, L. M., Pease, J. L., & Rauh, A. E. (2013). Introduction to altmetrics for science, technology, engineering, and mathematics (STEM) librarians. Science & Technology Libraries, 32(4), 335–345. doi:10.1080/0194262X.2013.829762.
27. Glass, G. V. (1976). Primary, secondary, and meta-analysis. Educational Researcher, 5, 3–8.
28. Groth, P., & Gurney, T. (2010). Studying scientific discourse on the Web using bibliometrics: A chemistry blogging case study. Paper presented at the WebSci10: Extending the Frontiers of Society On-Line, Raleigh, NC, USA. http://wiki.few.vu.nl/sms/images/9/9c/Websci10-FINAL-29-4-2010f.pdf
29. Gunn, W. (2013). Social signals reflect academic impact: What it means when a scholar adds a paper to mendeley. Information Standards Quarterly, 25(2), 33–39.
30. Hammarfelt, B. (2014). Using altmetrics for assessing research impact in the humanities. Scientometrics. doi:10.1007/s11192-014-1261-3.
31. Haustein, S. (2014). Readership metrics. In B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics: Harnessing multi-dimensional indicators of performance (pp. 327–344). Cambridge, MA: MIT Press.
32. Haustein, S., & Peters, I. (2012). Using social bookmarks and tags as alternative indicators of journal content description. firstmonday, 17(11).
33. Haustein, S., Peters, I., Bar-Ilan, J., Priem, J., Shema, H., & Terliesner, J. (2014a). Coverage and adoption of altmetrics sources in the bibliometric community. Scientometrics. doi:10.1007/s11192-013-1221-3.
34. Haustein, S., Peters, I., Sugimoto, C. R., Thelwall, M., & Larivière, V. (2014b). Tweeting biomedicine: An analysis of tweets and citations in the biomedical literature. Journal of the Association for Information Science and Technology, 65(4), 656–669. doi:10.1002/asi.23101.
35. Haustein, S., & Siebenlist, T. (2011). Applying social bookmarking data to evaluate journal usage. Journal of Informetrics, 5(3), 446–457. doi:10.1016/j.joi.2011.04.002.
36. Henning, V. (2010). The top 10 journal articles published in 2009 by readership on Mendeley | Mendeley Blog. Retrieved June 27, 2014, from http://www.mendeley.com/blog/academic-features/the-top-10-journal-articles-published-in-2009-by-readership-on-mendeley/
37. Holmberg, K., & Thelwall, M. (2014). Disciplinary differences in Twitter scholarly communication. Scientometrics. doi:10.1007/s11192-014-1229-3.
38. Kjellberg, S. (2010). I am a blogging researcher: Motivations for blogging in a scholarly context. First Monday, 15(8).
39. Kovic, I., Lulic, I., & Brumini, G. (2008). Examining the medical blogosphere: An online survey of medical bloggers. J Med Internet Res, 10(3), e28.
40. Li, X., & Thelwall, M. (2012). F1000, Mendeley and traditional bibliometric indicators. In E. Archambault, Y. Gingras, & V. Lariviere (Eds.), The 17th international conference on science and technology indicators (pp. 541–551). Montreal: Repro-UQAM.
41. Li, X., Thelwall, M., & Giustini, D. (2012). Validating online reference managers for scholarly impact measurement. Scientometrics, 91(2), 461–471. doi:10.1007/s11192-011-0580-x.
42. Lin, J., & Fenner, M. (2013). The many faces of article-level metrics. Bulletin of the American Society for Information Science and Technology, 39(4), 27–30. doi:10.1002/bult.2013.1720390409.
43. Liu, C. L., Xu, Y. Q., Wu, H., Chen, S. S., & Guo, J. J. (2013). Correlation and interaction visualization of altmetric indicators extracted from scholarly social network activities: Dimensions and structure. Journal of Medical Internet Research, 15(11), 17. doi:10.2196/jmir.2707.
44. Luzón, M. J. (2013). Public communication of science in blogs: Recontextualizing scientific discourse for a diversified audience. Written Communication. doi:10.1177/0741088313493610.
45. Mahrt, M., Weller, K., & Peters, I. (2012). Twitter in scholarly communication. In K. Weller, A. Bruns, J. Burgess, M. Mahrt, & C. Puschmann (Eds.), Twitter and society (pp. 399–410). New York, NY: Peter Lang.
46. Marsh, H. W., Bornmann, L., Mutz, R., Daniel, H. D., & O’Mara, A. (2009). Gender effects in the peer reviews of grant proposals: A comprehensive meta-analysis comparing traditional and multilevel approaches. Review of Educational Research, 79(3), 1290–1326. doi:10.3102/0034654309334143.
47. Matt, G. E., & Navarro, A. M. (1997). What meta-analyses have and have not taught us about psychotherapy effects: A review and future directions. Clinical Psychology Review, 17(1), 1–32.
48. Mewburn, I., & Thomson, P. (2013). Why do academics blog? An analysis of audiences, purposes and challenges. Studies in Higher Education, 38(8), 1105–1119. doi:10.1080/03075079.2013.835624.
49. Mohammadi, E., & Thelwall, M. (2013). Assessing the Mendeley readership of social science and humanities research. In J. Gorraiz, E. Schiebel, C. Gumpenberger, & M. Ho (Eds.), Proceedings of ISSI 2013 Vienna: 14th International society of scientometrics and informetrics conference (pp. 200–214). Vienna, Austria: Austrian Institute of Technology GmbH.
50. Mohammadi, E., & Thelwall, M. (2014). Mendeley readership altmetrics for the social sciences and humanities: Research evaluation and knowledge flows. Journal of the Association for Information Science and Technology, n/a–n/a. doi:10.1002/asi.23071.
51. Neylon, C., Willmers, M., & King, T. (2014). Rethinking impact: Applying altmetrics to southern African research. Ottawa: International Development Research Centre.
52. NISO Alternative Assessment Metrics Project. (2014). NISO Altmetrics Standards Project White Paper. Retrieved July 8, 2014, from http://www.niso.org/apps/group_public/document.php?document_id=13295&wg_abbrev=altmetrics
53. Priem, J. (2014). Altmetrics. In B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics: Harnessing multi-dimensional indicators of performance. Cambridge, MA: MIT Press.
54. Priem, J., & Costello, K. L. (2010). How and why scholars cite on Twitter. Proceedings of the American Society for Information Science and Technology, 47(1), 1–4. doi:10.1002/meet.14504701201.
55. Priem, J., Costello, K., & Dzuba, T. (2012, 2012/12/16). Prevalence and use of Twitter among scholars. Retrieved June 23, 2014, from http://figshare.com/articles/Prevalence:and_use_of_Twitter_among_scholars/104629
56. Priem, J., & Hemminger, B. M. (2010). Scientometrics 2.0: Toward new metrics of scholarly impact on the social Web. First Monday, 15(7).
57. Priem, J., Piwowar, H., & Hemminger, B. (2012). Altmetrics in the wild: Using social media to explore scholarly impact. Retrieved March 27, from http://arxiv.org/html/1203.4745
58. Puschmann, C. (2014). (Micro)Blogging Science? Notes on potentials and constraints of new forms of scholarly communication. In S. Bartling & S. Friesike (Eds.), Opening science (pp. 89–106). Cham, Switzerland: Springer International Publishing.
59. Puschmann, C., & Mahrt, M. (2012). Scholarly blogging: A new form of publishing or science journalism 2.0? In A. Tokar, M. Beurskens, S. Keuneke, M. Mahrt, I. Peters, C. Puschmann, K. Weller, & T. van Treeck (Eds.), Science and the Internet (pp. 171–182). Dusseldorf University Press: Düsseldorf.
60. Rodgers, E. P., & Barbrow, S. (2013). A look at altmetrics and its growing significance to research libraries. Ann Arbor, MI: The University of Michigan University Library.
61. Schlögl, C., Gorraiz, J., Gumpenberger, C., Jack, K., & Kraker, P. (2013). Download vs. vitiation vs. readership data: The case of an information systems journal. In J. Gorraiz, E. Schiebel, C. Gumpenberger, M. Hörlesberger, & H. Moed (Eds.), Proceedings of the 14th international society of scientometrics and informetrics conference. Austria: Austrian Institute of Technology, Vienna.
62. Schlögl, C., Gorraiz, J., Gumpenberger, C., Jack, K., & Kraker, P. (2014). A comparison of citations, downloads and readership data for an information systems journal. Research Trends (37), 14–18.
63. Schubert, A., & Braun, T. (1986). Relative indicators and relational charts for comparative assessment of publication output and citation impact. Scientometrics, 9(5–6), 281–291.
64. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin Company.
65. Shema, H. (2014). Scholarly blogs are a promising altmetric source. Research Trends (37), 11-13.
66. Shema, H., Bar-Ilan, J., & Thelwall, M. (2012a). Research blogs and the discussion of scholarly information. PLoS ONE, 7(5), e35869. doi:10.1371/journal.pone.0035869.
67. Shema, H., Bar-Ilan, J., & Thelwall, M. (2012b). Self-citation of bloggers in the science blogosphere. In A. Tokar, M. Beurskens, S. Keuneke, M. Mahrt, I. Peters, C. Puschmann, K. Weller, & T. van Treeck (Eds.), Science and the Internet (pp. 183–192). Du¨sseldorf: Du¨sseldorf University Press.
68. Shema, H., Bar-Ilan, J., & Thelwall, M. (2014). Do blog citations correlate with a higher number of future citations? Research blogs as a potential source for alternative metrics. Journal of the Association for Information Science and Technology, 65(5), 1018–1027. doi:10.1002/asi.23037.
69. Shema, H., Bar-Ilan, J., & Thelwall, M. (in press). How is research blogged? A content analysis approach. Journal of the Association for Information Science and Technology.
70. Shuai, X., Pepe, A., & Bollen, J. (2012). How the scientific community reacts to newly submitted preprints: Article downloads, Twitter mentions, and citations. Plos One, 7(11). doi: ARTN e47523 doi:10.1371/journal.pone.0047523.
71. StataCorp. (2013). Stata statistical software: Release 13. College Station, TX: Stata Corporation.
72. Sud, P., & Thelwall, M. (in press). Not all international collaboration is beneficial: The Mendeley readership and citation impact of biochemical research collaboration. Journal of the Association for Information Science and Technology.
73. Taylor, M. (2013). Towards a common model of citation: Some thoughts on merging altmetrics and bibliometrics. Research Trends (35), 19–22.
74. Thelwall, M. (2014, January 2). Five recommendations for using alternative metrics in the future UK Research Excellence Framework. Retrieved from http://blogs.lse.ac.uk/impactofsocialsciences/2014/10/23/alternative-metrics-future-uk-research-excellence-framework-thelwall/
75. Thelwall, M., Haustein, S., Lariviere, V., & Sugimoto, C. R. (2013). Do altmetrics work? Twitter and ten other social web services. PLoS ONE,. doi:10.1371/journal.pone.0064841.
76. Thelwall, M., & Maflahi, N. (in press). Are scholarly articles disproportionately read in their own country? An analysis of Mendeley readers. Journal of the Association for Information Science and Technology.
77. Thorsen, E. (2013). Blogging on the ice: Connecting audiences with climate-change sciences. International Journal of Media & Cultural Politics, 9(1), 87–101. doi:10.1386/macp.9.1.87_1.
78. Torres-Salinas, D., Cabezas-Clavijo, A., & Jimenez-Contreras, E. (2013). Altmetrics: New indicators for scientific communication in Web 2.0. Comunicar, 41, 53–60.
79. Tramer, M. R., Reynolds, D. J. M., Moore, R. A., & McQuay, H. J. (1997). Impact of covert duplicate publication on meta-analysis: A case study. British Medical Journal, 315(7109), 635–640.
80. Wainer, J., & Vieira, P. (2013). Correlations between bibliometrics and peer evaluation for all disciplines: The evaluation of Brazilian scientists. Scientometrics, 96(2), 395–410. doi:10.1007/s11192-013-0969-9.
81. Weller, K., Dröge, E., & Puschmann, C. (2011). Citation analysis in Twitter: Approaches for defining and measuring information flows within Tweets during scientific conferences. In M. Rowe, M. Stankovic, A.-S. Dadzie, & M. Hardey (Eds.), Making Sense of Microposts (MSM2011) (pp. 1–12). Heraklion: CEUR Workshop Proceedings.
82. Weller, K., & Peters, I. (2012). Citations in Web 2.0. In A. Tokar, M. Beurskens, S. Keuneke, M. Mahrt, I. Peters, C. Puschmann, T. van Treeck, & K. Weller (Eds.), Science and the Internet (pp. 209–222). Düsseldorf: Du¨sseldorf University Press.
83. Weller, K., & Puschmann, C. (2011, June 14–17 2011). Twitter for Scientific Communication: How Can Citations/References be Identified and Measured? Paper presented at the ACM WebSci’11, Koblenz, Germany.
84. White, H. D. (2005). On extending informetrics: An opinion paper. In P. Ingwersen & B. Larsen (Eds.), Proceedings of the 10th International conference of the international society for scientometrics and informetrics (Vol. 2, pp. 442–449). Stockholm, Sweden: Karolinska University Press.
85. Wolinsky, H. (2011). More than a blog. EMBO Reports, 12(11), 1102–1105.
86. Wouters, P. (2014). The citation: From culture to infrastructure. In B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics: Harnessing multi-dimensional indicators of performance (pp. 47–66). Cambridge, MA: MIT Press.
87. Yan, K. K., & Gerstein, M. (2011). The spread of scientific information: Insights from the web usage statistics in PLoS article-level metrics. PloS One, 6(5). doi: ARTNe19917 doi:10.1371/journal.pone.0019917
88. Zahedi, Z., Costas, R., & Wouters, P. (2014). How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications. Scientometrics. doi:10.1007/s11192-014-1264-0.
89. Zubiaga, A., Spina, D., Martınez, R., & Fresno, V. (2014). Real-time classification of twitter trends. Journal of the Association for Information Science and Technology, n/a–n/a. doi:10.1002/asi.23186.