Arts and humanities research evaluation: No metrics please, just data

Journal of Documentation

Volumn 71, Issue 4, 2015, Pages 817-833

  Thelwall, Mike ^a   Delgado, Maria M ^b  

^a UNIVERSITY OF WOLVERHAMPTON   (United Kingdom)

^b QUEEN MARY UNIVERSITY OF LONDON   (United Kingdom)

Author keywords

Arts and humanities research; Contextual information; Information science and documentation; Research

Indexed keywords

EID: 84933510902     PISSN: 00220418     EISSN: None     Source Type: Journal    
DOI: 10.1108/JD-02-2015-0028     Document Type: Article

References (78)

1. AHRC (2014), “Arts and Humanities Research Council Delivery Plan 2015-2016”, available at: www.ahrc.ac.uk/News-and-Events/Publications/Documents/AHRC%20Delivery%20Plan%202015-16%20%28A%29.pdf (accessed 13 February 2015).
2. AHRC (2015), “Understanding your project: a guide to self-evaluation”, available at: www.ahrc.ac.uk/What-We-Do/Build-the-evidence-base/Pages/Self-evaluation.aspx (accessed 13 February 2015).
3. Aksnes, D.W. and Taxt, R.E. (2004), “Peer reviews and bibliometric indicators: a comparative study at a Norwegian university”, Research Evaluation, Vol. 13 No. 1, pp. 33-41.
4. Anderson, R.C., Narin, F. and McAllister, P. (1978), “Publication ratings versus peer ratings of universities”, Journal of the American Society for Information Science, Vol. 29 No. 2, pp. 91-103.
5. Arts Council England (2013), “The Arts Council Plan 2011-15”, available at: www.artscouncil.org.uk/media/uploads/pdf/Arts_Council_Plan_2011-15.pdf (accessed 13 February 2015).
6. Åström, F. (2007), “Changes in the LIS research front: time‐sliced cocitation analyses of LIS journal articles, 1990-2004”, Journal of the American Society for Information Science and Technology, Vol. 58 No. 7, pp. 947-957.
7. Bate, J. (Ed.) (2011), The Public Value of the Humanities, Bloomsbury Academic, London.
8. BBC (2014), “Annual report 2013/14”, available at: www.bbc.co.uk/annualreport/2014/glossary (accessed 13 February 2015).
9. Belfiore E. and Bennett, O. (2008), The Social Impact of the Arts: An Intellectual History, Palgrave Macmillan, Houndmills.
10. Belfiore, E. and Bennett, O. (2010), “Beyond the ‘toolkit approach’: arts impact evaluation research and the realities of cultural policy‐making”, Journal for Cultural Research, Vol. 14 No. 2, pp. 121-142.
11. Belfiore, E. and Upchurch, A. (Eds) (2013), Humanities in the Twenty-First Century: Beyond Utility and Markets, Palgrave Macmillan, Basingstoke.
12. Belling, C. (2010), “Commentary: sharper instruments: on defending the humanities in undergraduate medical education”, Academic Medicine, Vol. 85 No. 6, pp. 938-940.
13. Boyack, K.W. and Klavans, R. (2010), “Co-citation analysis, bibliographic coupling, and direct citation: which citation approach represents the research front most accurately?”, Journal of the American Society for Information Science and Technology, Vol. 61 No. 12, pp. 2389-2404.
14. Brooks, T.A. (1986), “Evidence of complex citer motivations”, Journal of the American Society for Information Science, Vol. 37 No. 1, pp. 34-36.
15. Brown, C.D. (2002), “Straddling the humanities and social sciences: the research process of music scholars”, Library & Information Science Research, Vol. 24 No. 1, pp. 73-94.
16. Cronin, B. (2014), “Scholars and scripts, spoors and scores” in Cronin, B. and Sugimoto, C.R. (Eds), Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact, MIT Press, Cambridge, MA, pp. 3-21.
17. Dancy, C.P. and Reidy, J. (2004), Statistics Without Maths for Psychology, Pearson Education Limited, Harlow.
18. Day, R.E. (2014), “The data – it is me!”, in Cronin, B. and Sugimoto, C.R. (Eds), Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact, MIT Press, Cambridge, MA, pp. 47-66.
19. Delgado, M.M. and Bottoms, S. (2010), “Investigación sobre teatro y artes escénicas en el reino unido”, Cairon, Vol. 13 No. 1, pp. 153-187.
20. Didegah, F. and Thelwall, M. (2013), “Which factors help authors produce the highest impact research? Collaboration, journal and document properties”, Journal of Informetrics, Vol. 7 No. 4, pp. 861-873.
21. ERA (2015), “ERA discipline matrix”, available at: www.arc.gov.au/era/era_2015/2015_keydocs.htm (accessed 13 February 2015).
22. Franceschet, M. and Costantini, A. (2011), “The first Italian research assessment exercise: a bibliometric perspective”, Journal of Informetrics, Vol. 5 No. 2, pp. 275-291.
23. Garfield, E. (1979), Citation Indexing: Its Theory and Application in Science, Technology, and Humanities, Wiley, New York, NY.
24. Glänzel, W. (2002), “Coauthorship patterns and trends in the sciences (1980-1998): a bibliometric study with implications for database indexing and search strategies”, Library Trends, Vol. 50 No. 3, pp. 461-473.
25. Gussak, D. (2006), “Effects of art therapy with prison inmates: a follow-up study”, The Arts in Psychotherapy, Vol. 33 No. 3, pp. 188-198.
26. Heaney, C.J. (1992), “Evaluation of music therapy and other treatment modalities by adult psychiatric inpatients”, Journal of Music Therapy, Vol. 29 No. 2, pp. 70-86.
27. HEFCE (2009), “Report on the pilot exercise to develop bibliometric indicators for the research excellence framework”, available at: www.hefce.ac.uk/pubs/year/2009/200939/name,63842,en.html; www.hefce.ac.uk/pubs/year/2011/201103/name,63893,en.html (accessed 13 February 2015).
28. HEFCE (2011), “Analysis of data from the pilot exercise to develop bibliometric indicators for the REF: the effect of using normalised citation scores for particular staff characteristics”, available at: www.hefce.ac.uk/pubs/year/2011/201103/name,63893,en.html (accessed 13 February 2015).
29. HEFCE (2014), “Citation data”, available at: www.ref.ac.uk/about/guidance/citationdata/(accessed 13 February 2015).
30. Hellqvist, B. (2010), “Referencing in the humanities and its implications for citation analysis”, Journal of the American Society for Information Science and Technology, Vol. 61 No. 2, pp. 310-318.
31. Heritage, P. (2005), “Parallel power: shakespeare, gunfire and silence”, Contemporary Theatre Review, Vol. 15 No. 4, pp. 392-405.
32. Holden, J. (2004), “Creating cultural value: how culture has become a tool of government policy”, Demos, available at: www.demos.co.uk/files/CapturingCulturalValue.pdf (accessed 13 February 2015).
33. Hudson, R. (2007), “How linguistics has influenced schools in England”, Language and Linguistics Compass, Vol. 1 No. 4, pp. 227-242.
34. Hyland, K. (2004), Disciplinary Discourses: Social Interactions in Academic Writing, University of Michigan Press, Ann Arbor, MI.
35. Ioannidis, J.P. (2005), “Contradicted and initially stronger effects in highly cited clinical research”, JAMA, Vol. 294 No. 2, pp. 218-228.
36. Johnson, L.M. (2008), “A place for art in prison: art as a tool for rehabilitation and management”, Southwest Journal of Criminal Justice, Vol. 5 No. 2, pp. 100-120.
37. Kontos, P.C. and Naglie, G. (2007), “Expressions of personhood in alzheimer’s disease: an evaluation of research-based theatre as a pedagogical tool”, Qualitative Health Research, Vol. 17 No. 6, pp. 799-811.
38. Korevaar, J.C. and Moed, H. (1996), “Validation of bibliometric indicators in the field of mathematics”, Scientometrics, Vol. 37 No. 1, pp. 117-130.
39. Kousha, K. and Thelwall, M. (in press), “Can amazon.com reviews help to assess the wider impacts of books?”, Journal of the Association for Information Science and Technology.
40. Lariviere, V., Gingras, Y. and Archambault, É. (2006), “Canadian collaboration networks: a comparative analysis of the natural sciences, social sciences and the humanities”, Scientometrics, Vol. 68 No. 3, pp. 519-533.
41. MacRoberts, M.H. and MacRoberts, B.R. (1996), “Problems of citation analysis”, Scientometrics, Vol. 36 No. 3, pp. 435-444.
42. Merton, R.K. (1973), The Sociology of Science: Theoretical and Empirical Investigations, University of Chicago Press, Chicago, IL.
43. Moed, H.F. (2006), Citation Analysis in Research Evaluation, Springer, Berlin.
44. Moed, H.F. (2008), “UK research assessment exercises: informed judgments on research quality or quantity?”, Scientometrics, Vol. 74 No. 1, pp. 153-161.
45. Mryglod, O., Kenna, R., Holovatch, Y. and Berche, B. (2013). “Comparison of a citation-based indicator and peer review for absolute and specific measures of research-group excellence”, Scientometrics, Vol. 97 No. 3, pp. 767-777.
46. Nederhof, A.J. and Van Raan, A.F. (1993), “A bibliometric analysis of six economics research groups: a comparison with peer review”, Research Policy, Vol. 22 No. 4, pp. 353-368.
47. Nelson, R. (2013), Practice as Research in the Arts: Principles, Protocols, Pedagogies, Resistances, Palgrave Macmillan, Houndmills.
48. Norris, M. and Oppenheim, C. (2003), “Citation counts and the research assessment exercise v- archaeology and the 2001 RAE”, Journal of Documentation, Vol. 59 No. 6, pp. 709-730.
49. Nussbaum, M.C. (2012), Not for Profit: Why Democracy Needs the Humanities, Princeton University Press, Princeton, NJ.
50. Ochsner, M., Hug, S.E. and Daniel, H.D. (2014), “Setting the stage for the assessment of research quality in the humanities. Consolidating the results of four empirical studies”, Zeitschrift für Erziehungswissenschaft, Vol. 17 No. 6, pp. 111-132.
51. Oldfield, A., (2006), Interactive Music Therapy in Child and Family Psychiatry Clinical Practice, Research and Teaching Jessica Kingsley, London.
52. Oppenheim, C. (1995), “The correlation between citation counts and the 1992 research assessment exercise ratings for British library and information science university departments”, Journal of Documentation, Vol. 51 No. 1, pp. 18-27.
53. Oppenheim, C. (1997), “The correlation between citation counts and the 1992 research assessment exercise ratings for British research in genetics, anatomy and archaeology”, Journal of Documentation, Vol. 53 No. 5, pp. 477-487.
54. Oppenheim, C. and Summers, M.A. (2008), “Citation counts and the research assessment exercise, part VI: unit of assessment 67 (music)”, Information Research: An International Electronic Journal, Vol. 13 No. 2, available at: www.informationr.net/ir/13-2/paper342.html
55. PBRF (2013), “Quality evaluation guidelines 2012”, available at: www.tec.govt.nz/Documents/Publications/PBRF-Quality-Evaluation-Guidelines-2012.pdf (accessed 13 February 2015).
56. Priem, J., Taraborelli, D., Groth, P. and Neylon, C. (2010), “Altmetrics: a manifesto”, available at: http://altmetrics.org (accessed 13 February 2015).
57. REF2014 (2012a), “Assessment framework and guidance on submissions”, available at: www.ref.ac.uk/media/ref/content/pub/assessmentframeworkandguidanceonsubmissions/GOS%20including%20addendum.pdf (accessed 13 February 2015).
58. REF2014 (2012b), “Panel criteria and working methods, main panel D criteria”, available at: www.ref.ac.uk/media/ref/content/pub/panelcriteriaandworkingmethods/01_12_2D.pdf (accessed 13 February 2015).
59. REF2014 (2015a), “Research excellence framework 2014: overview report by main panel A and sub-panels 1-6”, available at: www.ref.ac.uk/panels/paneloverviewreports/ (accessed 13 February 2015).
60. REF2014 (2015b), “Research excellence framework 2014: overview report by main panel B and sub-panels 7-15”, available at: www.ref.ac.uk/panels/paneloverviewreports/ (accessed 13 February 2015).
61. REF2014 (2015c), “Research excellence framework 2014: overview report by main panel C and sub-panels 16-26”, available at: www.ref.ac.uk/panels/paneloverviewreports/ (accessed 13 February 2015).
62. REF2014 (2015d), “Research excellence framework 2014: overview report by main panel D and sub-panels 27-36”, available at: www.ref.ac.uk/panels/paneloverviewreports/ (accessed 13 February 2015).
63. Renfrew, K. and Green, H. (2014), “Support for arts and humanities researchers post PhD”, British Academy and The Arts and Humanities Research Council, available at: www..ac.uk/News-and-Events/News/Documents/Support%20for%20Arts%20and%20Humanities%20Researchers%20Post-PhD.pdf (accessed 13 February 2015).
64. Rinia, E.J., Van Leeuwen, T.N., Van Vuren, H.G. and Van Raan, A.F. (1998), “Comparative analysis of a set of bibliometric indicators and central peer review criteria: evaluation of condensed matter physics in the Netherlands”, Research Policy, Vol. 27 No. 1, pp. 95-107.
65. Rothman, J. (2014), “Fixing the PhD”, The New Yorker, 4 June, available at: www.newyorker.com/books/joshua-rothman/fixing-the-ph-d (accessed 13 February 2015).
66. Seglen, P.O. (1998), “Citation rates and journal impact factors are not suitable for evaluation of research”, Acta Orthopaedica, Vol. 69 No. 3, pp. 224-229.
67. Small, H. (2013), The Value of the Humanities, Oxford University Press, Oxford.
68. Smith, A.T. and Eysenck, M. (2002), “The correlation between RAE ratings and citation counts in psychology”, available at: http://cogprints.org/2749/
69. Stige, B. (2005), “Research as practice”, Nordic Journal of Music Therapy, Vol. 14 No. 2, pp. 90-90.
70. Thelwall, M. and Sud, P. (2014), “No citation advantage for monograph-based collaborations?”, Journal of Informetrics, Vol. 8 No. 1, pp. 276-283.
71. Thomas, P.R. and Watkins, D.S. (1998), “Institutional research rankings via bibliometric analysis and direct peer review: a comparative case study with policy implications”, Scientometrics, Vol. 41 No. 3, pp. 335-355.
72. Thompson, J. (2009), Performance Affects: Applied Theatre and the End of Effect, Palgrave Macmillan, Basingstoke.
73. Torres-Salinas, D., Robinson-Garcia, N., Campanario, J.M. and López-Cózar, E.D. (2014), “Coverage, field specialisation and the impact of scientific publishers indexed in the book citation index”, Online Information Review, Vol. 38 No. 1, pp. 24-42.
74. Vacca, J.S. (2004), “Educated prisoners are less likely to return to prison”, Journal of Correctional Education, Vol. 55 No. 4, pp. 297-305.
75. van Raan, A.F.J. (1998), “In matters of quantitative studies of science: the fault of theorists is offering too little and asking too much”, Scientometrics, Vol. 43 No. 1, pp. 129-139.
76. Wainer, J. and Vieira, P. (2013), “Correlations between bibliometrics and peer evaluation for all disciplines: the evaluation of Brazilian scientists”, Scientometrics, Vol. 96 No. 2, pp. 395-410.
77. Warner, J. (2000), “A critical review of the application of citation studies to the research assessment exercises”, Journal of Information Science, Vol. 26 No. 6, pp. 453-459.
78. Wolffe, J. (2011), “Why religious history matters: perspectives from 1851”, in Bate, J. (Ed.), The Public Value of the Humanities, Bloomsbury Academic, London, pp. 44-55.