The Use of Altmetrics as a Tool for Measuring Research Impact

Australian Academic and Research Libraries

Volumn 46, Issue 2, 2015, Pages 121-134

  Barnes, Cameron ^a  

^a UNIVERSITY OF NEW ENGLAND   (Australia)

Author keywords

altmetrics; bibliometrics; F1000Prime; Internet social networks; Mendeley; research impact; scientific communication; social web; Twitter

Indexed keywords

EID: 84930042360     PISSN: 00048623     EISSN: None     Source Type: Journal    
DOI: 10.1080/00048623.2014.1003174     Document Type: Article

References (89)

1. E.Adie, Euan. 2013. “Horizons: Impact Doesn't Need to be All About Academic Citations.” Altmetric, August 9. Accessed May 16, 2014. http://www.altmetric.com/blog/broaden-your-horizons-impact-doesnt-need-to-be-all-about-citations/.
2. R.Adler, Robert, J.EwingJohn, and P.TaylorPeter. 2008. Citation Statistics: A Report from the International Mathematical Union (IMU) in Cooperation with the International Council of Industrial and Applied Mathematics (ICIAM) and the Institute of Mathematical Statistics (IMS). Berlin: International Mathematical Union.
3. L.Allen, Liz, C.JonesCeri, K.DolbyKevin, D.LynnDavid, M.WalportMark, and R.W.SchererRoberta W. 2009. “Looking for Landmarks: The Role of Expert Review and Bibliometric Analysis in Evaluating Scientific Publication Outputs.” PLoS ONE 4 (6): e5910. Accessed March 26, 2014. doi:10.1371/journal.pone.0005910.
4. Australian Research Council. 2013. Annual Report 2012–2013. Canberra: Australian Research Council. Accessed April 25, 2014. http://www.arc.gov.au/about_arc/annual_report.
5. J.Bar-Ilan, Judit. 2012a. “JASIST@mendeley.” Altmetrics, June 16. Accessed March 21, 2014. http://altmetrics.org/altmetrics12/bar-ilan/.
6. J.Bar-Ilan, Judit. 2012b. “JASIST 2001–2010.” Bulletin of the American Society for Information Science and Technology 38 (6): 24–28. Accessed March 21, 2014. http://www.asis.org/Bulletin/Aug-12/AugSep12_Bar-Ilan.html.
7. J.Bar-Ilan, Judit, S.HausteinStephanie, I.PetersIsabella, J.PriemJason, H.ShemaHadas, and J.TerliesnerJens. 2012. “Beyond Citations: Scholars' Visibility on the Social Web.” In Proceedings of the 17th International Conference on Science and Technology Indicators, edited by E.ArchambaultEric, Y.GingrasYves, and V.LarivièreVincent, 98–109. Montréal: Science-Metrix.
8. C.Barnes, Cameron. 2014. “The Emperor's New Clothes: The h-Index as a Guide to Resource Allocation in Higher Education.” Journal of Higher Education Policy and Management 36 (5): 456–470. Accessed December 13, 2014. doi:10.1080/1360080X.2014.936087.
9. L.Bornmann, Lutz. 2014. “Do Altmetrics Point to the Broader Impact of Research? An Overview of Benefits and Disadvantages of Altmetrics.” Journal of Informetrics 8 (4): 895–903. Accessed December 12, 2014. doi:10.1016/j.joi.2014.09.005.
10. L.Bornmann, Lutz, and H.-D.DanielHans-Dieter. 2008. “What Do Citation Counts Measure? A Review of Studies on Citing Behavior.” Journal of Documentation 64 (1): 45–80. Accessed August 23, 2014. doi:10.1108/00220410810844150.
11. L.Bornmann, Lutz, and L.LeydesdorffLoet. 2013. “The Validation of (Advanced) Bibliometric Indicators Through Peer Assessments: A Comparative Study Using Data from InCites and F1000.” Journal of Informetrics 7 (2): 286–291. Accessed May 15, 2014. doi:10.1016/j.joi.2012.12.003.
12. S.Boslaugh, Sarah, and P.WattersPaul. 2008. Statistics in a Nutshell. Sebastopol, CA: O'Reilly.
13. D.Brabham, Daren. 2013. Crowdsourcing. Cambridge, MA: MIT Press.
14. Á.Cabezas-Clavijo, Álvaro, and D.Torres-SalinasDaniel. 2010. “Indicadores de Uso y Participación en las Revistas Científicas 2.0: El Caso de PloS One.” El Profesional de la Información 19 (4): 431–434. Accessed March 23, 2014. doi:10.3145/epi.2012.jul.15.
15. J.Careless, James. 2013. “Altmetrics 101: A Primer.” Information Today 30 (2): 1–36. Accessed March 25, 2014. http://www.infotoday.com/it/feb13/Careless–Altmetrics-101-A-Primer.shtml.
16. T.Carpenter, Todd. 2012. “Altmetrics – Replacing the Impact Factor is Not the Only Point.” The Scholarly Kitchen, November 12. Accessed March 22, 2014. http://scholarlykitchen.sspnet.org/2012/11/14/altmetrics-replacing-the-impact-factor-is-not-the-only-point.
17. D.Colquhoun, David, and A.PlestedAndrew. 2014. “Why You Should Ignore Altmetrics and Other Bibliometric Nightmares.” DC's Improbable Science, January 16. Accessed May 8, 2014. http://www.dcscience.net/?p=6369.
18. R.Costas, Roderigo, Z.ZahediZohreh, and P.WoutersPaul. 2014. “Do Altmetrics Correlate with Citations? Extensive Comparison of Altmetric Indicators with Citations from a Multidisciplinary Perspective.” Accessed April 21, 2014. ArchivX 1401.4321.
19. D.Crotty, David. 2012. “Post-Publication Peer Review: What Value Do Usage-Based Metrics Offer.” The Scholarly Kitchen, April 19. Accessed April 26, 2014. http://scholarlykitchen.sspnet.org/2012/04/19/post-publication-peer-review-what-value-do-usage-based-metrics-offer/.
20. L.G.Daniel, Larry G. 1998. “Statistical Significance Testing: A Historical Overview of the Misuse and Misinterpretation with Implications for the Editorial Policies of Educational Journals.” Research in the Schools 5 (2): 23–32.
21. P.Davis, Phil. 2012. “Tweets and Our Obsession With Alt Metrics.” The Scholarly Kitchen, January 4. Accessed May 8, 2014. http://scholarlykitchen.sspnet.org/2012/01/04/tweets-and-our-obsession-with-alt-metrics/.
22. P.Evans, Perry, and M.KrauthammerMichael. 2011. “Exploring the Use of Social Media to Measure Journal Article Impact.” AMIA Annual Symposium Proceedings Archive, Vol. 2011, 374–381. Accessed March 21, 2014. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3243242/.
23. Evidence Ltd. 2007. The Use of Bibliometrics to Measure Research Quality in UK Higher Education Institutions. London: Universities UK.
24. G.Eysenbach, Gunther. 2011. “Can Tweets Predict Citations? Metrics of Social Impact Based on Twitter and Correlation with Traditional Metrics of Scientific Impact.” Journal of Medical Internet Research 13: e123. Accessed May 8, 2014. doi:10.2196/jmir.2012.
25. S.Fausto, Sibele, F.A.MachadoFabio A., L.F.J.BentoLuiz Fernando J., A.IamarinoAtila, T.R.NahasTatiana R., D.S.MungerDavid S., and M.PercMatjaz. 2012 “Research Blogging: Indexing and Registering the Change in Science 2.0.” PLoS One 7 (12): e50109. Accessed May 15, 2014. doi:10.1371/journal.pone.0050109.
26. M.Fenner, Martin. 2010. “ReaderMeter: Researcher-Level Metrics Based on Readership.” Gobbbledygook, October 3. Accessed April 26, 2014. http://blogs.plos.org/mfenner/2010/10/03/readermeter-researcher-level-metrics-based-on-readership/.
27. M.Fenner, Martin. 2011. “CrowdoMeter – Or Trying to Understand Tweets About Journal Papers.” Goobledygook, December 20. Accessed June 8, 2014. http://blogs.plos.org/mfenner/2011/12/20/crowdometer-or-trying-to-understand-tweets-about-journal-papers/.
28. M.Fenner, Martin. 2014. “Altmetrics and Other Novel Measures for Scientific Impact.” In Opening Science: The Evolving Guide on How the Internet Is Changing Research, Collaboration and Scholarly Publishing, edited by S.BartlingSönke and S.FriesikeSascha, 170–189. Berlin: Springer.
29. blog, August 29. Accessed May 15, 2014. http://www.swets.com/blog/altmetrics-for-librarians-and-institutions-part-i#.U3P1XhC8yds.
30. blog, August 31. Accessed May 15, 2014. http://www.swets.com/blog/altmetrics-for-librarians-and-institutions-part-ii#.Ut5O4NG4ZxK.
31. F.Galligan, Finbar, and S.Dyas-Correia, Sharon. (2013). “Altmetrics: Rethinking the way we measure.” Serials Review 39 (1): 56–61. Accessed January 21, 2015. 10.1016/j.serrev.2013.01.003.
32. L.M.Galloway,, J.L.Pease, and A.E.Rauh. 2013. “Introduction to Altmetrics for Science, Technology, Engineering, and Mathematics (STEM) Librarians.” Science & Technology Libraries 32 (4): 335–345. Accessed May 15, 2014. doi:10.1080/0194262X.2013.829762.
33. F.Galton, Frances. 1907. “Vox Populi.” Nature 75 (1949): 450–451. doi:10.1038/075450a0.
34. E.Garfield, Eugene. 1979. “Is Citation Analysis a Legitimate Evaluation Tool?.” Scientometrics 1 (4): 359–375. doi:10.1007/BF02019306.
35. E.Garfield, Eugene. 1996. “Fortnightly Review: How Can Impact Factors be Improved?.” BMJ 313 (7054): 411–413. 10.1136/bmj.313.7054.411.
36. E.Garfield, Eugene. 1999. “Journal Impact Factor: A Brief Review.” Canadian Medical Association Journal 161 (8): 979–998.
37. E.Garfield, Eugene. 2006. “The History and Meaning of the Journal Impact Factor.” JAMA 295 (1): 90–93. Accessed April 1, 2014. doi:10.1001/jama.295.1.90.
38. J.Gill, Jeff. 1999. “The Insignificance of Null Hypothesis Significance Testing.” Political Research Quarterly 52 (3): 647–674. Accessed May 2, 2014. doi:10.1177/106591299905200309.
39. W.Glänzel, Wolfgang, B.SchlemmerBalázs, and B.ThijsBart. 2003. “Better Late than Never? On the Chance to Become Highly Cited Only Beyond the Standard Bibliometric Time Horizon.” Scientometrics 58 (3): 571–586. Accessed May 25, 2014. doi:10.1023/B:SCIE.0000006881.30700.ea.
40. F.J.Gravetter, Frederick J., and L.B.WallnauLarry B. 2013. Statistics for the Behavioral Sciences. Belmont, CA: Wadsworth.
41. B.Hammarfelt, Björn. 2014. “Using Altmetrics for Assessing Research Impact in the Humanities.” Scientometrics 101: 1419–1430. Accessed April 26, 2014. doi:10.1007/s11192-014-1261-3.
42. S.Haustein, Stefanie, I.PetersIsabella, C.R.Sugimoto, M.ThelwallMike, and V.LarivièreVincent. 2014. “Tweeting Biomedicine: An Analysis of Tweets and Citations in the Biomedical Literature.” Journal of the Association For Information Science and Technology 65 (4): 656–669. Accessed May 15, 2014. doi:10.1002/asi.23101.
43. S.Haustein, Stefanie, and T.SiebenlistTobias. 2011. “Applying Social Bookmarking Data to Evaluate Journal Usage.” Journal of Informetrics 5 (3): 446–457. Accessed June 9, 2014. doi:10.1016/j.joi.2011.04.002.
44. V.Henning, Victor. 2010. “The Top 10 Journal Articles Published in 2009 by Readership on Mendeley.” Mendeley Blog, January 28, p. 1. Accessed April 14, 2014. http://www.mendeley.com/blog/academic-features/the-top-10-journal-articles-published-in-2009-by-readership-on-mendeley/.
45. Higher Education Funding Council for England. 2012. “REF 2014.” In Assessment Framework and Guidance on Submissions. Bristol: Higher Education Funding Council for England Accessed May 24, 2014. http://www.ref.ac.uk/media/ref/content/pub/assessmentframeworkandguidanceonsubmissions/GOS%20including%20addendum.pdf.
46. K.Holmberg, Kim, and M.ThelwallMike. 2014. “Disciplinary Differences in Twitter Scholarly Communication.” Scientometrics 101 (2): 1027–1042. Accessed December 14, 2014. doi:10.1007/s11192-014-1229-3.
47. J.Howe, Jeff. 2006. “The Rise of Crowdsourcing.” Wired, pp. 1–4, June 14. Accessed May 15, 2014. http://www.wired.com/wired/archive/14.06/crowds.html.
48. J.Howe, Jeff. 2009. Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business. London: Random House Business.
49. S.Huggett, 2012. “F1000 Journal Rankings: An Alternative Way to Evaluate the Scientific Impact of Scholarly Communications.” Research Trends 26: 7–11. Accessed May 15, 2014. http://www.researchtrends.com/wp-content/uploads/2012/01/Research_Trends_Issue26.pdf.
50. S.Konkiel, Stacy. 2013. “Altmetrics: A 21st-Century Solution to Determining Research Quality.” Online Searcher 37 (4): 10–15.
51. C.J.Kowalski, Charles J. 1972. “On the Effects of Non-Normality on the Distribution of the Sample Product-Moment Correlation Coefficient.” Journal of the Royal Statistical Society. Series C (Applied Statistics) 21 (1): 1–12.
52. S.Lapinski, Scott, H.PiwowarHeather, and J.PriemJason. 2013. “Riding the Crest of the Altmetrics Wave: How Librarians Can Help Prepare Faculty for the Next Generation of Research Impact Metrics.” College & Research Libraries News 74 (6): 292–300. Accessed May 2, 2014. http://crln.acrl.org/content/74/6/292.long.
53. X.Li, Xuemei, and M.ThelwallMike. 2012. “F1000, Mendeley and Traditional Bibliometric Indicators.” In Proceedings of the 17th International Conference on Science and Technology Indicators, edited by E.ArchambaultEric, Y.GingrasYves, and V.LarivièreVincent, 541–551. Montréal: Science-Metrix.
54. X.Li, Xuemei, M.ThelwallMike, and D.GiustiniDean. 2012. “Validating Online Reference Managers for Scholarly Impact Measurement.” Scientometrics 91 (2): 461–471. Accessed May 14, 2014. doi:10.1007/s11192-011-0580-x.
55. C.L.Liu,, Y.Q.Xu, H.WuHui, S.S.Chen, and J.J.Guo. 2013. “Correlation and Interaction Visualization of Altmetric Indicators Extracted from Scholarly Social Network Activities: Dimensions and Structure.” Journal of Medical Internet Research 15 (11): e259. Accessed May 10, 2014. doi:10.2196/jmir.2707.
56. P.Martin, Paul, and P.BatesonPaul. 1993. Measuring Behaviour: An Introductory Guide. Cambridge: Cambridge University Press.
57. E.Mohammadi, Ehsan, and M.ThelwallMike. 2013. “Assessing Non-Standard Article Impact Using F1000 Labels.” Scientometrics 97 (2): 383–395. Accessed May 15, 2014. doi:10.1007/s11192-013-0993-9.
58. E.Mohammadi, Ehsan, and M.ThelwallMike. 2014. “Mendeley Readership Altmetrics for the Social Sciences and Humanities: Research Evaluation and Knowledge Flows.” Journal of the Association for Information Science and Technology 65 (8): 1627–1638. Accessed December 12, 2014. doi:10.1002/asi.23071.
59. E.Mohammadi, Ehsan, M.ThelwallMike, S.HuasteinStefanie, and V.LarivièreVincent. 2014. “Mendeley Readership Altmetrics for the Social Sciences and Humanities: Research Evaluation and Knowledge Flows.” Journal of the Association for Information Science and Technology 65 (8): 1627–1638. Accessed December 14, 2014. 10.1002/asi.23071.
60. R.Monastersky, Richard. 2005. “The Number that's Devouring Science.” The Chronicle of Higher Education 58 (2): A12.
61. D.Nations, Daniel. 2014. “What Is a Retweet?” About.com Web Trends, March 5. Accessed April 26. http://webtrends.about.com/od/twitter/a/twitter_help_what_is_a_retweet.htm.
62. T.O'Reilly, Tim. 2005. “What is Web 2.0? Design Patterns and Business Models for the Next Generation of Software.” O'Reilly, September 30. Accessed May 15, 2014. http://oreilly.com/web2/archive/what-is-web-20.html.
63. D.A.Pendlebury,. 2009. “The Use and Misuse of Journal Metrics and Other Citation Indicators.” Archivum Immunologiae et Therapiae Experimentalis 57 (1): 1–11. 10.1007/s00005-009-0008-y.
64. H.Piwowar, Heather. 2013. “Introduction Altmetrics: What, Why and Where?.” Bulletin of the American Society for Information Science and Technology 39 (4): 8–9. Accessed May 15, 2014. doi:10.1002/bult.2013.1720390404.
65. H.Piwowar, Heather, and J.PriemJason. 2013. “The Power of Altmetrics on a CV.” Bulletin of the American Society for Information Science and Technology 39 (4): 10–13. doi:10.1002/bult.2013.1720390405.
66. J.Priem, Jason, P.GrothPaul, D.TaraborelliDario, and C.A.OuzounisChristos A. 2012a. “The Altmetrics Collection.” PLoS ONE 7 (11): e48753. doi:10.1371/journal.pone.0048753.
67. J.Priem, Jason, H.PiwowarHeather, and B.M.HemmingerBradley M. 2012b. “Altmetrics in the Wild: Using Social Media to Explore Scholarly Impact.” Accessed May 16, 2014. http://arxiv.org/abs/1203.4745v1.
68. J.Priem, Jason, D.TaraborelliDario, P.GrothPaul, and C.NeylonCameron. 2010. “Altmetrics: A Manifesto.” Accessed March 21, 2014. http://altmetrics.org/manifesto/.
69. Research Councils UK. 2011. RCUK Impact Requirements: Frequently Asked Questions. Accessed May 24, 2014. http://www.rcuk.ac.uk/RCUK-prod/assets/documents/impacts/RCUKImpactFAQ.pdf.
70. 2005. “Revolutionizing Peer Review?” Nature Neuroscience 8 (4): 397–397. doi:10.1038/nn0405-397.
71. J.Richardson, Joanna, T.Nolan-BrownTherese, P.LoriaPat, and S.J.BradburyStephanie J. 2012. “Library Research Support in Queensland: A Survey.” Australian Academic & Research Libraries 43 (4): 258–277. doi:10.1080/00048623.2012.10722287.
72. R.C.Roemer, Robin Chin, and R.BorchardtRachel. 2014, January. Accessed December 10. http://www.ala.org/acrl/publications/keeping_up_with/altmetrics “Keeping Up With … Altmetrics.” Keeping Up With ….
73. A.Schubert, András, and W.GlänzelWolfgang. 1986. “Mean Response Time – A New Indicator of Journal Citation Speed with Application to Physics Journals.” Czechoslovak Journal of Physics 36 (1): 121–125. 10.1007/BF01599743.
74. P.O.Seglen,. 1997. “Why the Impact Factor of Journals Should Not be Used for Evaluating Research.” BMJ 314 (7079): 497–497. doi:10.1136/bmj.314.7079.497.
75. H.Shema, Hadas, J.Bar-IlanJudit, and M.ThelwallMike. 2014. “Do Blog Citations Correlate with Higher Number of Future Citations? Research Blogs as a Potential Source for Alternative Metrics.” Journal of the Association for Information Science and Technology 65 (1): 1019–1027. Accessed May 16, 2014. doi:10.1002/asi.23037.
76. X.Shuai, Xin, A.PepeAlberto, and J.BollenJohan. 2012. “How the Scientific Community Reacts to Newly Submitted Preprints: Article Downloads, Twitter Mentions, and Citations.” PLoS ONE 7: e47523. Accessed May 16, 2014. doi:10.1371/journal.pone.0047523.
77. K.Simons, Kai. 2008. “The Misused Impact Factor.” Science 322 (5899): 165–165. Accessed May 15, 2014. doi:10.1126/science.1165316.
78. P.Sud, Pardeep, and M.ThelwallMike. 2014. “Evaluating Altmetrics.” Scientometrics 98 (2): 1131–1143. Accessed May 16, 2014. doi:10.1007/s11192-013-1117-2.
79. D.Tapscott, Don, and A.D.WilliamsAnthony D. 2006. Wikinomics: How Mass Collaboration Changes Everything. New York: Portfolio.
80. D.Taraborelli, Dario. 2008. “Soft Peer Review: Social Software and Distributed Scientific Evaluation.” In Proceedings of the 8th International Conference on the Design of Cooperative Systems, edited by P.HassanalyParina, A.RamrajsinghAthissingh, D.RandallDave, P.SalembierPaascal, and M.TixierMattheu, 99–110. Carry-le-Rouet: Institut d'Etudes Politiques d'Aix-en-Provence.
81. M.Taylor, Mike. 2013. “Value of Bibliometrics: The Challenges of Measuring Social Impact Using Altmetrics.” Research Trends 33: 11–13. Accessed August 25, 2014. http://www.researchtrends.com/wp-content/uploads/2013/05/Research_Trends_Issue_33.pdf.
82. M.Thelwall, Mike, S.HausteinStefanie, V.LarivièreVincent, C.R.SugimotoCassidy R., and L.BornmannLutz. 2013. “Do Altmetrics Work? Twitter and Ten Other Social Web Services.” PLoS ONE 8 (5): e64841. Accessed May 16, 2014. doi:10.1371/journal.pone.0064841.
83. Thomson-Reuters. 2008. “Using Bibliometrics: A Guide to Evaluating Research Performance with Citation Data.”. Accessed May 24, 2014. http://thomsonreuters.com/products/ip-science/04_030/using-bibliometrics-a-guide-to-evaluating-research-performance-with-citation-data.pdf.
84. D.Torres-Salinas, Daniel, Á.Cabezas-ClavijoÁlvaro, and E.Jiménez-ContrerasEvaristo. 2013. “Altmetrics: New Indicators for Scientific Communication in Web 2.0.” Comunicar 21 (41): 53–60. Accessed April 2, 2014. doi:10.3916/C41-2013-05.
85. L.Vaughan, Liwen, and D.ShawDebora. 2005. “Web Citation Data for Impact Assessment: A Comparison of Four Science Disciplines.” Journal of the American Society for Information Science and Technology 56 (10): 1075–1087. doi:10.1002/asi.20199.
86. L.Waltman, Ludo, and R.CostasRodrigo. 2014. “F1000 Recommendations as a Potential New Data Source for Research Evaluation: A Comparison with Citations.” Journal of the Association for Information Science and Technology 65 (3): 433–445. Accessed April 2, 2014. doi:10.1002/asi.23040.
87. J.Wang, Jian. 2013. “Citation Time Window Choice for Research Impact Evaluation.” Scientometrics 94 (3): 851–872. Accessed May 6, 2014. 10.1007/s11192-012-0775-9.
88. D.A.Wardle, David A. 2010. “Do ‘Faculty of 1000’ (F1000) Ratings of Ecological Publications Serve as Reasonable Predictors of Their Future Impact?” Ideas in Ecology and Evolution 3: 11–15. Accessed April 21, 2014. doi:10.4033/iee.2010.3.3.c.
89. Z.Zahedi, Zohreh, R.CostasRodrigo, and P.WoutersPaul. 2014. “How Well Developed Are Altmetrics? A Cross-Disciplinary Analysis of the Presence of ‘Alternative Metrics’ in Scientific Publications.” Scientometrics 101 (2): 1491–1513. Accessed May 16, 2014. doi:10.1007/s11192-014-1264-0.