Assessing legal research: Sense and Nonsense of Peer Review versus Bibliometrics and the Need for a European Approach

German Law Journal

Volumn 12, Issue 3, 2011, Pages 901-929

  van Gestel, Rob ^a   Vranken, Jan ^a  

^a TILBURG UNIVERSITY   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84881701994     PISSN: None     EISSN: 20718322     Source Type: Journal    
DOI: 10.1017/S2071832200017144     Document Type: Article

References (52)

1. See Dan Jerker B. Svantesson, International Ranking of LawJournals:CanitBeDoneandat What Costs? 29 LEGAL STUDIES 681 (2009). The author even argues that truly ground-breaking work may be more likely to appear from marginal, dissident or unexpected sources, rather than from well-established and entrenched mainstream publication media.
2. See STEVEN WOODING & JOHN GRANT, ASSESSING RESEARCH: THE RESEARCHERS VIEW (2003).
3. Kevin Campbell et 01, 26 JOURNAL OF LAw AND SOCIETY 470,349-350 (1999).
4. KEY PERSPECTIVESLTD., A ComparativeReview of Research Assessment RegimesinFiveCountriesandtheRole of LibrariesintheResearch Assessment Process, 8(2009), available at: http://www.oclc.org/research/publications/library/2009/2009-09 .pdf (last accessed: 3 January 2011).
5. Margit Osterloh &BrunoS.Frey, ResearchGovernanceinAcademia:AreThere Alternatives to Academic Rankings? 3 CESIFO Working Paper SeriesNo. 27974 (2009), available at: http://ssrn.com /abstract= 1460691 (last accessed: 3 January 2011).
6. For example, evenasearlyas2001 the UKResearch Assessment Exercise assessed over 2,400 submissions and examined over 150,000 publications.
7. Vlaamse Interuniversitaire Raad (Flemish Inter-University Council, VLlR), ModelvoorIntegrale Kwaliteitsevaluatie van het Onderzoek inde Rechtswetenschappen (2004). (The Verbeke Committee Report2004). Available at: http://www.vlir.be/media/docs/Onderzoeksbeleid/notitie KZR 22sept2004.pdf (last accessed: 3 January 2011).
8. Stolker Committee Report, 30 (2005); Smits Committee Report, 12 (2007). No longer available online. Onfile with the authors.
9. An example of the latter is revisiting jurisprudence 5 or 10 years later, to see whether the conclusions of an earlier investigation are still valid. Should this not be the case (wholly), the most interesting theoretical question becomes, 'Why not?'
10. See MARK TUSHNET, WEAK COURTS, STRONG RIGHTS: JUDICIAL REVIEW AND SOCIAL WELFARE RIGHTS IN COMPARATIVE CONSTITUTIONAL LAw (2008)
11. See, for example, the following special issue: 2 INTERNATIONAL JOURNAL OF LAw IN CONTEXT 217 (2006), and further on law and neuroscience.
12. ExpertAdvisoryGroup for an RQF, TheResearchQualityFramework: Assessing theQualityand Impact ofResearch in Australia 11 (2005), available at: http://www.dest.gov.au/NR/rdonlyres/E32ECC65-05CO-4041-A2B8 - 75ADEC69E159/4467/rgf issuespaper.pdf (last accessed: 3 January 2011).
13. Mathias M. Siems, LegalOriginality, 28 OXFORD JOURNAL OF LEGAL STUDIES 174 (2008).
14. See for example Deborah L. Rhode, LegalScholarship, 115 HARVARD LAw REVIEW 1327, 1340 (2002),
15. See Douglas W. Vick, InterdisciplinarityandtheDiscipline of Law, 31 OXFORD JOURNAL OF LEGAL STUDIES 163, 164 (2004),
16. An extensive description of doctrinal legal research can be found in MICHAEL SALTER AND JULIE MASON, WRITING LAw DISSERTATIONS (2007). See also, FIONA COWNIE, LEGAL ACADEMICS: CULTURE AND IDENTITIES (2004).
17. Smits Committee Report 20, 27 (2007), supra, note 18. The book that the chairman of the committee, Jan Smits, later wrote, JAN M. SMITS, OMSTREDEN RECHTSWETENSCHAP (Contentious Legal Scolarship) (2009), shows how little he expects from methodological requirements.
18. Verbeke Committee Report (2004), supra, note 15, cited in the Stalker Committee Report 30 (2005), supra, note 18
19. Stalker Committee Report 31 (2005), supra, note 18, mentioned 4,000 words, but only as a general and flexible guideline.
20. The quality of research of this kind is the subject of much concern in the US as well. Compare among others, Richard Posner, LegalScholarshipToday, 115 HARVARD LAw REVIEW 1314, 1314-1326, 1323-1326 (2002), and the harsh criticisms leveled throughout their paper by Lee Epstein &Gary King, TheRules ofInference, 69 CHICAGO LAw REVIEW 1 (2002).
21. Bruno de Witte, EuropeanUnionLaw: A Unified 'Academic Discipline?, 34 EUI WORKING PAPERS RSCAS (2008), available at: http://cadmus.eui.eu/d space/handle/1 814/10028 (last accessed: 3 January 2011).
22. For more extensive information, see Ron Johnston, On Structuring Subjective Judgements: Originality, Significance andRigour, 62 HIGHER EDUCATION QUARTERLY 120,120-147,132-133 (2008).
23. Society of LegalScholars, AHRB JOURNAL REFERENCE LIST (2005), available at: http://www.l egalschol ars.ac.uk/pubdocs/05/joint letter ahrb.pdf (last accessed: 3 January 2011).
24. A complete reprint can be found online: Lon L. Fuller, TheCase of the Speluncean Explorers, 62 HARVARD LAw REVIEW 4 (1949), available at: http://www.nullapoena.de/stud/explorers.html (last accessed: 8 March 2011).
25. See E-Petitions, THE OFFICIAL SITE OF THE PRIME MINISTER'S OFFICE, available at: http://petitions.numberlO.gov.uk/REFandimpact (last accessed: 3 January 2011). The first signs are that the UK government might actually change the societal impact agenda proposed for the Research Excellence Framework. See David Willetts, DavidWillettsdeliversfirst keynote speech asMinisterforUniversitiesandScience, DEPARTMENT FOR BUSINESS INNOVATION & SKILLS, available at: http://www.bis.gov.uk/news/speeches/david-wi lletts-keynotespeech (last accessed: 3 January 2011).
26. For a good overview see, ANN C. WELLER, EDITORIAL PEER REVIEW: ITSSTRENGTHS ANDWEAKNESSES (2001).
27. These disadvantages or p e e r r e v i e w are t h o r o u g h l y d o c u m e n t e d. See, f o r e x a m p l e, MICHELE LAMONT, H o w PROFESSORS THINK: INSIDE THE CURIOUS WORLD OF ACADEMIC JUDGMENT (2009). We also r e f e r t o t h e results o f t h e lSI Peer Review Survey, w h i c h can be f o u n d o n l i n e: PeerReviewSurvey2009:PreliminaryFindings, SENSE ABOUT SCIENCE, a v a i l a b l e at: h t t p: / / w w w .sen s e a b o u t science . o r g.uk / i n d e x . p h p / s i t e / p r ojec t /39 5 (last v i s i t e d 3 January 2011);
28. PARLIAMENTARY OFFICE OF SCIENCE AND TECHNOLOGY, PEER REVIEW, Postnote no. 182, 1-4 (2002), available at: http://w w w .parliament.uk /p ost /pn1 8 2.pdf (last accessed: 3 January 2011); and Johnston, (note 41), 120-147.
29. J. Scott Amstrong, PeerReview ofJournals:EvidenceonQualityControl,Fairnessand Innovation, 3 SCIENCE AND ENGINEERING ETHICS 63, 15 (1997). See also James Bradley, PerniciousPublicationPractices, 18 BULLETIN OF THE PSYCHONOMIC SOCIETY 31, 34 (1981).
30. More background can be found in PEN (note 2), 1-8 among others.
31. Report on the pilot exercise to develop bibliometric indicators for the REF (Executive Summary, 2009), available at: http://hefce.ac.uk! pubs! hefce! 2009! 09 39 (last accessed: 3 January 2011).
32. SeeThe Excellence in Research for Australia (ERA) Initiative. Available at: http://www.arc.gov.au!era!d efault.htm (last accessed: 3 January 2011). The ERA scheme also requires as one of its components international journal ranking. Svantesson (note 3), 680 is very critical about the linkage between assessing research quality and ranking.
33. Forfas/HEA, Research strengths inIreland:A bibliometric study of thepublicresearchbase (2009). Available at: http://www.forfas.ie!media!forfas091209 bib liometric st udy.pdf (last accessed: 3 January 2011).
34. Hanne Foss Hansen, ResearchEvaluation: Methods, PracticeandExperience, 52-60 DANISH AGENCY FOR SCIENCE, TECHNOLOGY AND INNOVATION (2009).
35. Serge Gutwirth, Evaluatie rechtswetenschappelijkonderzoek, 168 NieuwJuridisch Weekblad 674,677(2007).
36. In the Netherlands, for example, the faculties of Tilburg University, Rotterdam University, and Amsterdam University.
37. The Social Science Citation Index (lSI) normally uses a period of one or two years. See the KNAW report, Judging researchonItsMerits 18 (2005). Available at http://www.knaw.nl/publicaties/pdf/20051029.pdf. (last accessed: 3 January 2011). This document points out that where the humanities and social sciences are concerned, this period is usually too short to properly judge the impact of a publication.
38. Harry T. Edwards, TheGrowingDisjunction between LegalEducationandtheLegalProfession, 91 MICHIGAN LAw REVIEW 34 (1992) caused quite a stir in the US.
39. Tom Sastry & Bahram Bekhradnia, Using metrics toallocateresearchfunds: A short evaluation of alternatives to theResearch, 18 (2006), available at: http://www.hepi.ac.uk/fi les/23RAEandmetricsfu lireport.pdf (last accessed: 3 January 2011).
40. According to the KNAW Report, Kwaliteitszorginde wetenschap 6 (2008), existing quality assurance systems are seen as too much ofa burden.
41. RENE Boomkens, Topkitschenslowscience.Kritiekvande academische rede (Top Kitsch and Slow Science: A Critique on Academic Reason) 87 (2009).
42. Arguing that this is a real issue is G. Frankenberg, Stranger thanParadise: Identity & Politicsin Comparative Law, 259 UTAH LAw REVIEW 259 (1997).
43. See SMITS (note 31), 116. For the opinions of Antonin Scalia, compare Melissa A. Waters, JusticeScaliaonthe Use of ForeignLawin Constitutional Interpretation: Unidirectional Monologue or Co-constitutive Dialogue,12 TULSA JOURNAL CaMP&INT'LLAw 149 (2004).
44. Among others Jan Vranken, Alseenarrestnietbevalt,latenwij het gewoon weg(We leave out judicial decisions we don't like), in Ex lIBRISHANS NIEUWENHUIS 77 (Alex Geert Castermans et al, eds., 2009).
45. Martijn Hesselink, AEuropeanLegal Method? OnEuropeanPrivateLawandScientific Method, 15 EUROPEAN LAw JOURNAL 20, 22 (2009).
46. Compare Carel J.J. Stolker, LegalJournals:InPursuit of aMoreScientific Approach, 2 EUROPEAN JOURNAL OF LEGAL EDUCATION 87, 89 (2005).
47. Rob van Gestel & Jan Vranken, LegalScholarlyPapers.Naarcriteriavoor methodologische verantwoording (Three Methodological Requirements forLegalResearchPapers), 82 NEDERLANDS JURISTENBLAD 1448 (2007). Available in English on file with the authors.
48. MIKEMCCONVILLE & WING HONG CHUI (eds.), RESEARCH METHODS FOR LAw (2007).
49. BARBARA E. LOVITTS, MAKING THE IMPLICIT EXPLICIT: CREATING PERFORMANCE EXPECTATIONS FOR THE DISSERTATION (2007).
50. Dame Hazel Genn, Martin Partington & Sally Wheeler, LawintheRealWorld:ImprovingOur Understanding of HowLawWorks,Final Report and Recommendations (2006).
51. Tracey E.George, AnEmpirical Study of EmpiricalLegalScholarship:TheTopLawSchools, 81 INDIANA LAw JOURNAL 141 (2006).
52. JAN VRANKEN, EXPLORING THE JURIST'S FRAME OF MIND (2006).