Measuring the impact of research: Lessons from the UK's research excellence framework 2014

PLoS ONE

Volumn 11, Issue 6, 2016, Pages

  Chowdhury, Gobinda ^a   Koya, Kushwanth ^a   Philipson, Pete ^a  

^a NORTHUMBRIA UNIVERSITY   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

EXTRACT; FREQUENCY ANALYSIS; HUMAN; HUMAN EXPERIMENT; PATENT; PUBLICATION; SCIENTIST; UNIVERSITY; JOURNAL IMPACT FACTOR; MEASUREMENT ACCURACY; MEDICAL RESEARCH; UNITED KINGDOM;

BIOMEDICAL RESEARCH; DATA ACCURACY; HUMANS; JOURNAL IMPACT FACTOR; UNITED KINGDOM;

EID: 84975303419     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0156978     Document Type: Article

References (41)

1. Olson S & Merrill S. (2011). Measuring the impact of federal investments in research. National Academies (US) committee on measuring economic and other returns on federal research investments. National Academies Press (US): Washington (DC).
2. European Commission. (2015). Horizon 2020 indicators. Assessing the results and impact of Horizon 2020. Available at https://www.ffg.at/sites/default/files/downloads/page/horizon2020indicators.pdf (accessed December 1, 2015).
3. Economic and Social Research Council. (2009). Measuring the impact of ESRC funding. A report prepared for the Economic and Social Research Council. Available at http://www.esrc.ac.uk/files/research/evaluation-and-impact/measuring-the-impact-of-esrc-funding/ (accessed December 1, 2015).
4. Engineering and Physical Sciences Research Council. (2012). Research performance and economic impact report 2011/2012. Available at https://www.epsrc.ac.uk/newsevents/pubs/researchperformance-and-economic-impact-report-2011-12/ (accessed December 1, 2015)
5. Bornmann L. (2012). Measuring the societal impact of research. EMBO Reports. 13(8), 373-376.
6. Taylor, M. (2013). The challenges of measuring social impact using Altmetrics. Research Trends. Issue 33. Available at http://www.researchtrends.com/issue-33-june-2013/the-challenges-of-measuringsocial-impact-using-altmetrics/. (accessed November 14, 2015).
7. Reeves, M. (2002). Measuring the economic and social impact of the arts: a review. Art Council England. Available at http://www.artscouncil.org.uk/media/uploads/documents/publications/340.pdf. (accessed November 16, 2015).
8. Hirsch J. E. (2005). An index to quantify an individual's scientific research output. Proc Natl Acad Sci U.S.A. 102(46), 16569-16572. PMID: 16275915
9. Kousha K & Thelwall M. (2014) Web impact metrics for research assessment. in Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact. Edited by Cronin B and Sugimoto CR. MIT Press: Cambridge
10. Priem J. (2014). Altmetrics. in Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact. Edited by Cronin B and Sugimoto CR. MIT Press: Cambridge
11. Bornmann L. (2013). What is societal impact of research and how can it be assessed? a literature survey. Journal of the American Society for Information Science and Technology, 64(2), 217-233. doi: 10.1002/asi.22803
12. Saisana M., d'Hombres B. & Saltelli A. (2011). Rickety numbers: volatility of university rankings and policy implications. Research Policy. 40, 165-177
13. Rafols I., Leydesdorff L., O'Hare A., Nightingale P. & Stirling A. (2012). How journal rankings can suppress interdisciplinary research: a comparison between innovation studies and business & management. Research Policy. 41, 1262-1282.
14. Butler L. (2003). Explaining Australia' increased share of ISI publications-the effects of a funding formula based on publication counts. Research Policy. 32, 143-155.
15. Van Raan A. F. J. (2005). Fatal attraction: conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics. 62(1), 133-143
16. Hicks D., Wouters P., Waltman L., de Rijcke S. & Rafols I. (2015). Bibliometrics: The Leiden Manifesto for Research metrics. Nature. 520 (7548), 429-431 doi: 10.1038/520429a PMID: 25903611
17. Wilsdon J.,Allen L., Belfiore E., Campbell P., Curry S., Hill S., et al. (2015). The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management. doi: 10.13140/RG.2.1.4929.1363
18. Pepe A. & Kurtz M. J. (2012). A measure of total research impact independent of time and discipline. PLOS ONE. 7(11)
19. Sutherland W. J., Goulson D., Potts S. G. & Dicks L. V. (2011). Quantifying the impact and relevance of scientific research. PLOS ONE. 6(11)
20. REF 2014. (2014)About the REF. REF 2014. Available at http://www.ref.ac.uk/about/. (accessed May 15, 2015)
21. The Complete University Guide. (2016). Methodology in building the league tables for UK Universities. Available at http://www.thecompleteuniversityguide.co.uk/league-tables/methodology. (accessed January 24, 2016).
22. Hefce. (2015). Annual funding allocations. Hefce. Available at http://www.hefce.ac.uk/funding/annallocns/. (accessed June 12, 2015)
23. Jump, P. (2015). Winners and losers in HEFCE funding allocations: REF-based distribution of £1.6bn research pot results in some big changes. Times Higher Education. Available at https://www.timeshighereducation.co.uk/news/winners-and-losers-in-hefce-funding-allocations/2019306.article. (accessed June 12, 2015)
24. Jump, P. (2013). Australia prepares for (research) impact. Times Higher Education. Available at https://www.timeshighereducation.com/news/australia-prepares-for-research-impact/2005011.article. (accessed January 26, 2016)
25. REF 2014. (2014) Assessment framework and guidance on submissions. REF 2014. Available at http://www.ref.ac.uk/media/ref/content/pub/assessmentframeworkandguidanceonsubmissions/GOS%20including%20addendum.pdf. (accessed May 10, 2015)
26. Jump, P. (2015). REF 2014: impact element cost £55 million. Times Higher Education. 2015. Available at https://www.timeshighereducation.co.uk/news/ref-2014-impact-element-cost-55-million/2019439.article. (accessed June 12, 2015)
27. Smith S., Ward V. & House A. (2011). 'Impact' in the proposals for the UK's Research Excellence Framework: Shifting the boundaries of academic autonomy. Research Policy. 40(10), 1369-1379
28. Fernandez-Armesto, F. (2009). Poisonous impact. Times Higher Education. Available at https://www.timeshighereducation.com/comment/columnists/poisonous-impact/409403.article. (accessed November 10, 2015).
29. Oswald, A. (2011). REF should stay out of the game. Independent. Available at http://www.independent.co.uk/news/education/higher/andrew-oswald-ref-should-stay-out-of-the-game-1827306.html. (accessed November 10, 2015)
30. Shepherd, J. (2009). Humanities research threatened by demands of 'economic impact'. The Guardian. Available at http://www.theguardian.com/education/2009/oct/13/research-funding-economic-impact-humanities. (accessed November 10, 2015).
31. Wolff, J. (2013). Nobody wants their research impact to be graded 'considerable' in the REF. The Guardian. Available at http://www.theguardian.com/education/2013/oct/28/jonathan-wolff-research-excellence-framework. (accessed January 26, 2016).
32. Swain, H. (2013). Research funding emphasis on 'impact' spawns a new industry. The Guardian. Available at http://www.theguardian.com/education/2013/nov/12/research-excellence-framework-impact-consultants. (accessed November 09, 2015).
33. REF 2014. (2014) Results & submissions of Clinical Medicine UoA. REF 2014. Available at http://results.ref.ac.uk/Results/ByUoA/1. (accessed June 06, 2015)
34. Manville, C., Guthrie, S., Henham, ML., Garrod, B., Sousa, S., Kirtley, A., et al. (2015). Assessing impact submissions for REF 2014: An evaluation. Rand Europe. Available at https://www.rand.org/content/dam/rand/pubs/research-reports/RR1000/RR1032/RAND-RR1032.pdf (accessed December 15, 2015)
35. Times Higher Education. (2015). Higher Education Green Paper at a glance. Times Higher Education. Available at https://www.timeshighereducation.com/news/higher-education-green-paper-glance. (accessed November 14, 2015).
36. Journal Citation Reports. (2015). Journal Citation Reports Database. Thomson Reuters.
37. Cribari-Neto F & Zeileis A. (2010). Beta Regression in R. Journal of Statistical Software. 34(2), 1-24.
38. Ferrari S. L. & Cribari-Neto F. (2004). Beta Regression for Modeling Rates and Proportions. Journal of Applied Statistics, 31(7), 799-815.
39. Simas A. B., Barreto-Souza W. & Rocha A. V. (2010). Improved Estimators for a General Class of Beta Regression Models. Computational Statistics & Data Analysis, 54(2), 348-366.
40. R Core Team. (2013). R: A language and environment for statistical computing. R foundation for Statistical Computing, Vienna, Austria. URL http://R-project.org
41. REF 2014. (2014). REF 2014 panel overview reports. REF 2014. Available at http://www.ref.ac.uk/panels/paneloverviewreports/. (accessed 10 November, 2015).