The evaluators' eye: Impact assessment and academic peer review

The Evaluators' Eye: Impact Assessment and Academic Peer Review

Volumn , Issue , 2018, Pages 1-230

  Derrick, Gemma ^a  

^a LANCASTER UNIVERSITY   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85044653760     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1007/978-3-319-63627-6     Document Type: Book

References (257)

1. Aubé, C., V. Rousseau, and S. Tremblay. 2011. Team size and quality of group experience: The more the merrier? Group Dynamics: Theory, Research, and Practice 15 (4): 357.
2. Baron, R.S. 2005. So right it’s wrong: Groupthink and the ubiquitous nature of polarised group decision making. Advances in Experimental Social Psychology 37: 219-253.
3. Bernardin, H.J., H. Hennessey, and J. Peyrefitte. 1995. Age, racial, and gender bias as a function criterion specificity: A test of expert testimony. Human Resource Management Review 5 (1): 63-77.
4. Bornmann, L., G. Wallon, and A. Ledin. 2008. Does the committee peer review select the best applicants for funding? An investigation of the selection process for two european molecular biology organization programmes. PLoSOne 3 (10): e3480.
5. Bourdieu, P. 1975. The specificity of the scientific field and the social conditions of the progress of reason. Information (International Social Science Council) 14 (6): 19-47.
6. Chubin, D.E. 1994. Grants peer review in theory and practice. Evaluation Review 18 (1): 20-30.
7. Chubin, D.E., and E.J. Hackett. 1990. Peerless science: Peer review and US science policy. Albany: State University of New York Press.
8. Comer, D.R. 1995. A model of social loafing in real work groups. Human Relations 48 (6): 647-667.
9. Cooper, J., K.A. Kelly, and K. Weaver. 2001. Attitudes, norms, and social groups. In Blackwell Handbook of social psychology: Group processes, ed. M.A. Hogg and R.S. Tindale, 259-282. Oxford: Blackwell.
10. Dahler-Larsen, P. 2007. Evaluation and public management. In The Oxford Handbook of public management, ed. E. Ferlie, L.E. Lynn Jr., and C. Pollitt. Oxford: Oxford University Press.
11. Dahler-Larsen, P. 2011. The evaluation society. Palo Alto, CA: Stanford University Press.
12. Dahler-Larsen, P. 2012. Constitutive effects as a social accomplishment: A qualitative study of the political in testing. Education Inquiry 3 (2): 171-186.
13. Dahler-Larsen, P. 2014. Constitutive effects of performance indicators: Getting beyond unintended consequences. Public Management Review 16 (7): 969-986.
14. Derrick, G.E., and G.N. Samuel. 2014. The impact evaluation scale: Group panel processes and outcomes in societal impact evaluation. Social Science and Medicine, in press.
15. Epley, N., and T. Gilovich. 2006. The anchoring-and-adjustment heuristic: Why the adjustments are insufficient. Psychological Science 17 (4): 311-318.
16. Epley, N., B. Keysar, L. Van Boven, and T. Gilovich. 2004. Perspective taking as egocentric anchoring and adjustment. Journal of Personality and Social Psychology 87 (3): 327.
17. Esser, J. 1998. Alive and well after 25 years: A review of groupthink research. Organizational Behavior and Human Decision Processes 73 (2/3): 116-141.
18. Faigman, D.L., J. Monahan, and C. Slobogin. 2014. Group to individual (G2i) inference in scientific expert testimony. The University of Chicago Law Review 81 (2): 417-480.
19. Gallo, S.A., J.H. Sullivan, and S.R. Glisson. 2016. The influence of peer reviewer expertise on the evaluation of research funding applications. PLoS One 11 (10): e0165147.
20. Hall, D., and S. Buzwell. 2013. The problem of free-riding in group projects: Looking beyond social loafing as reason for non-contribution. Active Learning in Higher Education 14 (1): 37-49.
21. Hemlin, S., and S.B. Rasmussen. 2006. The shift in academic quality control. Science, Technology, & Human Values 31 (2): 173-198.
22. Holbrook, J.B., and R. Frodeman. 2011. Peer review and the exante assessment of societal impacts. Research Evaluation 20 (3): 239-246.
23. Huutoniemi, K. 2012. Communicating and compromising on disciplinary expertise in the peer review of research proposals. Social Studies of Science 42 (6): 897-921.
24. Janis, I.L. 1982. Groupthink: Psychological studies of policy decisions and fiascoes. Boston, MA: Houghton Mifflin Company.
25. Kerr, N.L., R.J. MacCoun, and G.P. Kramer. 1996. Bias in judgement: Comparing individuals and groups. Psychological Review 103: 687-719.
26. Lamont, M. 2009. How professors think: Inside the curious world of academic judgement. Cambridge, MA: Harvard University Press.
27. Langfeldt, L. 2001. The decision-making constraints and processes of grant peer review, and their effects on the review outcome. Social Studies of Science 31 (6): 820-841.
28. Langfeldt, L. 2006. The policy challenges of peer review: Managing bias, conflict of interests and multidisciplinary assessments. Research Evaluation 15 (1): 31-41.
29. Latane, B., K. Williams, and S. Harkins. 1979. Many hands make light the work: The causes and consequences of social loafing. Journal of Personality and Social Psychology 37 (6): 822-832.
30. Lee, C.J. 2012. A Kuhnian critique of psychometric research on peer review. Philosophy of Science 79 (5): 859-870.
31. Lee, C.J., C.R. Sugimoto, G. Zhang, and B. Cronin. 2013. Bias in peer review. Journal of the American Society for Information Science and Technology 64 (1): 2-17.
32. Levi, D. 2015. Group dynamics for teams. London: Sage Publications.
33. Luukkonen, T. 2012. Conservatism and risk-taking in peer review: Emerging ERC practices. Research Evaluation 21: 48-60.
34. Manville, C., S. Guthrie, M.-L. Henham, B. Garrod, S. Sousa, A. Kirtkey, S. Castle-Clarke, and T. Ling. 2015. Assessing impact submissions for REF2014: An evaluation. Cambridge: RAND Europe.
35. Merton, R.K. 1973. The sociology of science: Theoretical and empirical investigations. Chicago: University of Chicago press.
36. Porter, A.L., and F.A. Rossini. 1985. Peer review of interdisciplinary research proposals. Science, Technology, & Human Values 10 (3): 33-38.
37. Roumbanis, L. 2016. Academic judgments under uncertainty: A study of collective anchoring effects in Swedish Research Council panel groups. Social Studies of Science 47: 1-22.
38. Samuel, G.N., and G.E. Derrick. 2015. Societal impact evaluation: Exploring evaluator perceptions of the characterization of impact under the REF2014. Research Evaluation 24 (3): 229-241.
39. Simms, A., and T. Nichols. 2014. Social loafing: A review of the literature. Journal of Management Policy and Practice 15 (1): 58.
40. Taylor, J. 2011. The assessment of research quality in UK universities: Peer review or metrics? British Journal of Management 22 (2): 202-217.
41. Travis, G.D.L., and H.M. Collins. 1991. New light on old boys: Cognitive and institutional particularism in the peer review system. Science, Technology, & Human Values 16 (3): 322-341.
42. van Arensbergen, P., I. van der Weijden, and P. van den Besselaar. 2014. The selection of talent as a group process. A literature review on the social dynamics of decision making in grant panels. Research Evaluation 23 (4): 298-311.
43. Viner, N., P. Powell, and R. Green. 2004. Institutionalized biases in the award of research grants: A preliminary analysis revisiting the principle of accumulative advantage. Research Policy 33 (3): 443-454.
44. Abramo, G., C.A. D’Angelo, and F. Rosati. 2015. Selection committees for academic recruitment: Does gender matter? Research Evaluation 24 (4): 392-404.
45. Aldag, R.J., and S.R. Fuller. 1993. Beyond fiasco: A reappraisal of the group- think phenomenon and a new model of group decision processes. Psychological Bulletin 113 (3): 533.
46. Andersen, L.B., and T. Pallesen. 2008. “Not just for the money?” How financial incentives affect the number of publications at Danish research institutions. International Public Management Journal 11 (1): 28-47.
47. Arribas-Ayllon, M., and V. Walkerdine. 2008. Foucauldian discourse analysis. In The Sage Handbook of qualitative research in psychology, ed. Carla Willig and Wendy Stainton-Rogers, 91-108. London: Sage.
48. Auranen, O., and M. Nieminen. 2010. University research funding and publication performance-An international comparison. Research Policy 39 (6): 822-834.
49. Bornmann, L. 2012. Measuring the societal impact of research. EMBO Reports 13 (8): 673-676.
50. Bornmann, L. 2013. What is the societal impact of research and how can it be assessed? A literature survey. Journal of the American Society of Information Science and Technology 64 (2): 217-233.
51. Bornmann, L., G. Wallon, and A. Ledin. 2008. Does the committee peer review select the best applicants for funding? An investigation of the selection process for two european molecular biology organization programmes. PLoSOne 3 (10): e3480.
52. Bourdieu, P. 1975. The specificity of the scientific field and the social conditions of the progress of reason. Information (International Social Science Council) 14 (6): 19-47.
53. Bourdieu, P. 1984. Distinction: A social critique of the judgement of taste. Cambridge, MA: Harvard University Press.
54. Bourdieu, Pierre. 1997. Capital cultural, escuela y espacio social. Siglo xxi.
55. Buxton, M., and S. Hanney. 1996. How can payback from health services research be assessed? Journal of Health Services Research 1 (1): 35-43.
56. Cetina, K.K., J. Clark, C. Modgil, S. Modgil, I.B. Cohen, K. Duffin, S. Strickland, R. Feldhay, Y. Elkana, and R.K. Merton. 1991. Merton’s sociology of science: The first and the last sociology of science? JSTOR 20 (4): 522-526.
57. Chubb, J., and R. Watermeyer. 2016. Artifice or integrity in the marketization of research impact? Investigating the moral economy of (pathways to) impact statements within research funding proposals in the UK and Australia. Studies in Higher Education 1-13. https://doi.org/10.1080/03075079.20l6.1l44l82.
58. Chubin, D.E., and E.J. Hackett. 1990. Peerless science: Peer review and US science policy. Albany: State University of New York Press.
59. Cole, S., J.R. Cole, and G.A. Simon. 1981. Chance and consensus in peer review. Science 214 (4523): 881-886.
60. Collins, H.M., and R. Evans. 2002. The third wave of science studies: Studies of expertise and experience. Social Studies of Science 32 (2): 235-296.
61. Comer, D.R. 1995. A model of social loafing in real work groups. Human Relations 48 (6): 647-667.
62. Dahler-Larsen, P. 2007. Evaluation and public management. In The Oxford Handbook of public management, ed. E. Ferlie, L.E. Lynn Jr., and C. Pollitt. Oxford: Oxford University Press.
63. Dahler-Larsen, P. 2014. Constitutive effects of performance indicators: Getting beyond unintended consequences. Public Management Review 16 (7): 969-986.
64. de Jong, S.P., J. Smit, and L. van Drooge. 2015. Scientists’ response to societal impact policies: A policy paradox. Science and Public Policy 43 (1): 102-114.
65. De Rijcke, S., P.F. Wouters, A.D. Rushforth, T.P. Franssen, and B. Hammarfelt. 2016. Evaluation practices and effects of indicator use-A literature review. Research Evaluation 25 (2): 16-169.
66. Deem, R., S. Hillyard, and M. Reed. 2007. Knowledge, higher education, and the new managerialism: The changing management of UK universities. Oxford: Oxford University Press.
67. Derrick, G., I. Meijer, and E. van Wijk. 2014. Unwrapping “impact” for evaluation: A co-word analysis of the UK REF2014 policy documents using VOSviewer. Proceedings of the Science and Technology Indicators Conference.
68. Derrick, G.E., A.S. Haynes, S. Chapman, and WD. Hall. 2011. The association between four citation metrics and peer rankings of research influence of Australia researchers in six fields of public health. PLoSOne 6: e18521.
69. Derrick, G.E., and V. Pavone. 2013. Democratising research evaluation: Achieving greater public engagement with bibliometrics-informed peer review. Science and Public Policy 40 (5): 563-575.
70. Derrick, G.E., and G.N. Samuel. 2014. The impact evaluation scale: Group panel processes and outcomes in societal impact evaluation. Social Science and Medicine, in press.
71. Derrick, G.E., and G.N. Samuel. 2016. The evaluation scale: Exploring decisions about societal impact in peer review panels. Minerva 54 (1): 75-97.
72. Donovan, C., L. Butler, A.J. Butt, T.H. Jones, and S.R. Hanney. 2014. Evaluation of the impact of National Breast Cancer Foundation-funded research. The Medical Journal of Australia 200 (4): 214-218.
73. Fern ández-Zubieta, A., A. Geuna, and C. Lawson. 2015. Mobility and productivity of research scientists 1. Global Mobility of Research Scientists: The Economics of Who Goes Where and Why 105.
74. Finkel, A. 2015. Research Engagement for Australia (REA): Measuring research engagement between universities and end users. Presentation to Universities Australia.
75. Fogelholm, M., S. Leppinen, A. Auvinen, J. Raitanen, A. Nuutinen, and K. Vaananen. 2012. Panel discussion does not improve reliability of peer review for medical research grant proposals. Journal of Clinical Epidemiology 65 (1): 47-52.
76. Frodeman, R., and J. Parker. 2009. Intellectual merit and broader impact: The National Science Foundation’s broader impacts criterion and the question of peer review. SocialEpistemology 23 (3-4): 337-345.
77. Geuna, A., and M. Piolatto. 2016. Research assessment in the UK and Italy: Costly and difficult, but probably worth it (at least for a while). Research Policy 45 (1): 260-271.
78. Gibbons, M., C. Limoges, H. Nowotny, S. Schwartzman, and P. Scott. 1994. The new production of knowledge: The dynamics of science and research in contemporary societies. London: SAGE.
79. Giraudeau, B., C. Leyrat, A. Le Gouge, J. Leger, and A. Caille. 2011. Peer review of grant applications: A simple method to identify proposals with discordant reviews. PLoSOne 6 (11): e27557.
80. Grant, J., P-B. Brutscher, S. Kirk, L. Butler, and S. Wooding. 2010. Capturing research impacts: A review of International practice. Documented briefing. RAND Corporation.
81. Greene, J.C. 1997. Evaluation as advocacy. Evaluation Practice 18: 25-36.
82. Greene, J.C. 2000. Challenges in practicing deliberative democratic evaluation. New Directions for Evaluation 2000 (85): 13-26.
83. Hackett, E.J., and D. E. Chubin. 2003. Peer review for the 21st century: Applications to education research. Ed. National Research Council. Washington, DC.
84. Hall, D., and S. Buzwell. 2013. The problem of free-riding in group projects: Looking beyond social loafing as reason for non-contribution. Active Learning in Higher Education 14 (1): 37-49.
85. Harnad, S. 1985. Rational disagreement in peer review. Science, Technology, & Human Values 10 (3): 55-62.
86. Haynes, A.S., G.E. Derrick, S. Chapman, S. Redman, W.D. Hall, J. Gillespie, and H. Sturk. 2011. From “our world” to the “real world”: Exploring the views and behaviour of policy-influential Australian public health researchers. Social Science & Medicine 72: 1047-1055.
87. HEFCE. 2010. REF2014: Panel criteria and working methods.http://www.ref.ac.uk/media/ref/content/pub/panelcriteriaandworkingmethods/01_12.pdf. Accessed 1 Mar 2016.
88. Hemlin, S., and S.B. Rasmussen. 2006. The shift in academic quality control. Science, Technology, & Human Values 31 (2): 173-198.
89. Herbst, M. 2007. Financing public universities. New York: Springer.
90. Hicks, D. 2012. Performance-based university research funding systems. Research policy 41 (2): 251-261.
91. Hicks, D., P.F. Wouters, L. Waltman, S. De Rijcke, and I. Rafols. 2015. The Leiden Manifesto for research metrics. Nature 520 (7548): 429.
92. Huutoniemi, K. 2010. Evaluating interdisciplinary research. Oxford: Oxford University Press.
93. Huutoniemi, K. 2012. Communicating and compromising on disciplinary expertise in the peer review of research proposals. Social Studies of Science 42 (6): 897-921.
94. Ingwersen, P., and B. Larsen. 2014. Influence of a performance indicator on Danish research production and citation impact 2000-12. Scientometrics 101 (2): 1325-1344.
95. Janis, I.L. 1982. Groupthink: Psychological studies of policy decisions and fiascoes. Boston, MA: Houghton Mifflin Company.
96. Johnston, S.C., J.D. Rootenberg, S. Katrak, W.S. Smith, and J.S. Elkins. 2006. Effect of a US National Institutes of Health programme of clinical trials on public health and costs. Lancet 367 (9519): 1319-1327.
97. Knorr-Cetina, K.D. 1983. The ethnographic study of scientific work: Towards a constructivist interpretation of science. London: Sage.
98. Knorr-Cetina, K.D. 1991. Epistemic cultures: Forms of reason in science. History of Political Economy 23 (1): 105-122.
99. Knott, M. 2015. Academic publications to become less important when funding university research. Sydney: Sydney Morning Herald.
100. Kuruvilla, S., N. Mays, and G. Walt. 2007. Describing the impact of health services and policy research. Journal of Health Services Research & Policy 12 (suppl 1): 23-31.
101. Lamont, M. 2009. How professors think: Inside the curious world of academic judgement. Cambridge, MA: Harvard University Press.
102. Lamont, M., and K. Huutoniemi. 2011. Opening the black box of evaluation: How quality is recognized by peer review panels. Bulletin SAGW2: 47-49.
103. Langfeldt, L. 2001. The decision-making constraints and processes of grant peer review, and their effects on the review outcome. Social Studies of Science 31 (6): 820-841.
104. Langfeldt, L. 2004. Expert panels evaluating research: Decision-making and sources of bias. Research Evaluation 13 (1): 51-62.
105. Langfeldt, L. 2006. The policy challenges of peer review: Managing bias, conflict of interests and multidisciplinary assessments. Research Evaluation 15 (1): 31-41.
106. Latane, B., K. Williams, and S. Harkins. 1979. Many hands make light the work: The causes and consequences of social loafing. Journal of Personality and Social Psychology 37 (6): 822-832.
107. Laudel, G., and G. Origgi. 2006. Introduction to a special issue on the assessment of interdisciplinary research. Oxford: Oxford University Press.
108. Leisyte, L., and J.R. Dee. 2012. Understanding academic work in a changing institutional environment. In Higher education: Handbook of theory and research, ed. J.D. Smart, 123-206. New York: Springer.
109. Levi, D. 2015. Group dynamics for teams. London: Sage Publications.
110. Luukkonen, T. 2012. Conservatism and risk-taking in peer review: Emerging ERC practices. Research Evaluation 21: 48-60.
111. Mark, M.M., and R.L. Shotland. 1985. Stakeholder-based evaluation and value judgments. Evaluation Review 9 (5): 605-626.
112. Martin, B.R. 2011. The Research Excellence Framework and the ‘impact agenda’: Are we creating a Frankenstein monster? Research Evaluation 20 (3): 247-254.
113. Mayo, N.E., J. Brophy, M.S. Goldberg, M.B. Klein, S. Miller, R.W. Platt, and J. Ritchie. 2006. Peering at peer review revealed high degree of chance associated with funding of grant applications. Journal of Clinical Epidemiology 59 (8): 842-848.
114. Merton, R.K. 1973. The sociology of science: Theoretical and empirical investigations. Chicago: University of Chicago press.
115. Molas-Gallart, J., and P. Tang. 2011. Tracing ‘productive interactions’ to identify social impacts: An example from the social sciences. Research Evaluation 20 (3): 219-226.
116. Mollick, E., and R. Nanda. 2015. Wisdom or madness? Comparing crowds with expert evaluation in funding the arts. Management Science 62 (6): 1533-1553.
117. Nowotny, H., P. Scott, and M. Gibbons. 2001. Re-thinking science: Knowledge and the public in an age of uncertainty. Argentina: SciELO.
118. Olbrecht, M., and L. Bornmann. 2010. Panel peer review of grant applications: What do we know from research in social psychology on judgement and decision making in groups? Research Evaluation 19 (4): 293-304.
119. Oortwijn, W.J., S.R. Hanney, A. Ligtvoet, S. Hoorens, S. Wooding, J. Grant, M.J. Buxton, and L.M. Bouter. 2008. Assessing the impact of health technology assessment in the Netherlands. International Journal of Technology Assessment in Health Care 24 (03): 259-269.
120. Pielke, R.A., and R. Byerly. 1998. Beyond basic and applied. Physics Today 51 (2): 42-46.
121. Pier, E.L., J. Raclaw, A. Kaatz, M. Brauer, M. Carnes, M.J. Nathan, and C.E. Ford. 2017. ‘Your comments are meaner than your score’: Score calibration talk influence intra- and inter-panel variability during scientific grant peer review. Research Evaluation 26 (1): 1-14.
122. Porter, A.L., and F.A. Rossini. 1985. Peer review of interdisciplinary research proposals. Science, Technology, & Human Values 10 (3): 33-38.
123. Porter, R. 2005. What do grant reviewers really want, anyway? Journal of Research Administration 36 (2): 5-13,13.
124. Roumbanis, L. 2016. Academic judgments under uncertainty: A study of collective anchoring effects in Swedish Research Council panel groups. Social Studies of Science 47: 1-22.
125. Roy, R. 1985. Funding science: The real defects of peer review and an alternative to it. Science, Technology, & Human Values 10 (3): 73-81.
126. Salter, A.J., and B.R. Martin. 2001. The economic benefits of publicly funded basic research: A critical review. Research policy 30 (3): 509-532.
127. Sandstrom, U., U. Heyman, and P. Van den Besselaar. 2014. The complex rela- tionahip between competitive funding and performance. In Context counts: Pathways to master big and little data -STI, ed. E. Noyons, 523-533. Leiden: CWTS. https://doi.org/10.13140/2.1.5036.6728.
128. Simms, A., and T. Nichols. 2014. Social loafing: A review of the literature. Journal of Management Policy and Practice 15 (1): 58.
129. Sivertsen, G., and J. Schneider. 2012. Evaluering av den bibliometriske forskningsindikator.
130. Smith, R. 2001. Measuring the social impact of research-Difficult but necessary. British Medical Journal 323: 528. https://doi.org/10.1136/bmj.323.7312.528
131. Smith, S., V. Ward, and A. House. 2011. ‘Impact’ in the proposals for the UK’s Research Excellence Framework: Shifting the boundaries of academic autonomy. Research Policy 40 (10): 1369-1379.
132. Solomon, M. 2006. Groupthink versus the wisdom of crowds: The social epis- temology of deliberation and dissent. The Southern Journal of Philosophy 44 (S1): 28-42.
133. Spaapen, J., and L. Van Drooge. 2011. Introducing ‘productive interactions’ in social impact assessment. Research Evaluation 20 (3): 211-218.
134. Thaler Richard, H., and R. Sunstein Cass. 2008. Nudge: Improving decisions about health, wealth, and happiness. New Haven, CT: Yale University Press.
135. Travis, G.D.L., and H.M. Collins. 1991. New light on old boys: Cognitive and institutional particularism in the peer review system. Science, Technology, & Human Values 16 (3): 322-341.
136. van Arensbergen, P. 2014. Talent proof. Selection processes in research funding and careers. Den Haag: Rathenau Instituut.
137. van Arensbergen, P., I. van der Weijden, and P. van den Besselaar. 2014. The selection of talent as a group process. A literature review on the social dynamics of decision making in grant panels. Research Evaluation 23 (4): 298-311.
138. Van der Meulen, Barend B, and Arie Rip. 2000. Evaluation of societal quality of public sector research in the Netherlands. Research Evaluation 9 (1): 11-25.
139. Watermeyer, R. 2015. Lost in the ‘third space’: The impact of public engagement in higher education on academic identity, research practice and career progression. European Journal of Higher Education 5 (3): 331-347.
140. Watermeyer, R. 2016. Impact in the REF: Issues and obstacles. Studies in Higher Education 41 (2): 199-214.
141. Watermeyer, R., and A. Hedgecoe. 2016. Selling ‘impact’: Peer reviewer projections of what is needed and what counts in REF impact case studies. A retrospective analysis. Journal of Education Policy 31 (5): 651-665.
142. Willmott, H. 2011. Journal list fetishism and the perversion of scholarship: Reactivity and the ABS list. Organization 18 (4): 429-442.
143. Wooding, S., S. Hanney, M. Buxton, and J. Grant. 2005. Payback arising from research funding: Evaluation of the Arthritis Research Campaign. Rheumatology 44 (9): 1145-1156.
144. Wooding, S., S. Hanney, A. Pollitt, M. Buxton, and J. Grant. 2011. Project retrosight: Understanding the returns from cardiovascular and stroke research: the policy report. Rand Health Quarterly 1 (1): 16.
145. Aksnes, D.W., and R.E. Taxt. 2004. Peer reviews and bibliometric indicators: A comparative study at a Norwegian University. Research Evaluation 13 (1): 33-41.
146. Bence, V., and C. Oppenhein. 2005. The evolution of the UK’s research assessment exercise: Publications, performance and perceptions. Journal of Educational Administration and History 37 (2): 137-155.
147. Bornmann, L. 2012. Measuring the societal impact of research. EMBO Reports 13 (8): 673-676.
148. Bornmann, L., and H.-D. Daniel. 2005. Committee peer review at an international research foundation: Predictive validity and fairness of selection decisions on post-graduate fellowship applications. Research Evaluation 14 (1): 15-20.
149. Bornmann, L., and W. Marx. 2014. How should the societal impact of research be generated and measured? A proposal for a simple and practicable approach to allow interdisciplinary comparisons. Scientometrics 98 (1): 211-219.
150. Bornmann, L., G. Wallon, and A. Ledin. 2008. Does the committee peer review select the best applicants for funding? An investigation of the selection process for two european molecular biology organization programmes. PLoSOne 3 (10): e3480.
151. Dahler-Larsen, P. 2011. The evaluation society. Palo Alto, CA: Stanford University Press.
152. Dahler-Larsen, P. 2012. Constitutive effects as a social accomplishment: A qualitative study of the political in testing. Education Inquiry 3 (2): 171-186.
153. Dahler-Larsen, P. 2014. Constitutive effects of performance indicators: Getting beyond unintended consequences. Public Management Review 16 (7): 969-986.
154. Derrick, G.E., and G.N. Samuel. 2017. The future of societal impact assessment using peer review: Pre-evaluation training, consensus building and inter-reviewer reliability. Palgrave Communications. https://doi.org/10.1057/palcomms.2017.40
155. Donovan, C. 2011. State of the art in assessing research impact: Introduction to a special issue. Research Evaluation 20 (3): 175-179.
156. Epley, N., B. Keysar, L. Van Boven, and T. Gilovich. 2004. Perspective taking as egocentric anchoring and adjustment. Journal of Personality and Social Psychology 87 (3): 327.
157. Franceschet, M., and A. Costantini. 2011. The first Italian research assessment exercise: A bibliometric perspective. Journal of Informetrics 5 (2): 275-291.
158. Frodeman, R., and J. Parker. 2009. Intellectual merit and broader impact: The National Science Foundation’s broader impacts criterion and the question of peer review. SocialEpistemology 23 (3-4): 337-345.
159. Grant, J., P-B. Brutscher, S. Kirk, L. Butler, and S. Wooding. 2010. Capturing research impacts: A review of International practice. Documented briefing. RAND Corporation.
160. Greene, J.C. 2000. Challenges in practicing deliberative democratic evaluation. New Directions for Evaluation 2000 (85): 13-26.
161. HEFCE. 2010. REF2014: Panel criteria and working methods.http://www.ref.ac.uk/media/ref/content/pub/panelcriteriaandworkingmethods/01_12.pdf. Accessed 1 Mar 2016.
162. HEFCE. 2011. Assessment framework and guidance on submissions. REF 2014.
163. Holbrook, J.B. 2010. The use of societal impacts considerations in grant proposal peer review: A comparison of five models. Technology & Innovation 12 (3): 213-224.
164. Holbrook, J.B., and S. Hrotic. 2013. Blue skies, impacts, and peer review. A Journal on Research Policy & Evaluation. https://doi.org/10.13130/2282-5398/2914.
165. King’s College London and Digital Science. 2015. The nature, scale and beneficiaries of research impact: An initial analysis of Research Excellence Framework (REF) 2014 impact case studies. King’s College London and Digital Science.
166. Lamont, M. 2009. How professors think: Inside the curious world of academic judgement. Cambridge, MA: Harvard University Press.
167. Manville, C., S. Guthrie, M.-L. Henham, B. Garrod, S. Sousa, A. Kirtkey, S. Castle-Clarke, and T. Ling. 2015. Assessing impact submissions for REF2014: An evaluation. Cambridge: RAND Europe.
168. Mark, M.M., G.T. Henry, and G. Julnes. 2000. Evaluation: An integrated framework for understanding, guiding and improving policies and programs. San Francisco, CA: Jossey-Bass.
169. Norman, R. 2002. Managing through measurement or meaning? Lessons from experience with New Zealand’s public sector performance management systems. International Review of Administrative Sciences 68 (4): 619-628.
170. Rinia, E.J., T.N. van Leeuwen, H.G. van Vuren, and A.FJ. van Raan. 1998. Comparative analysis of a set of bibliometric indicators and central peer review criteria: Evaluation of condensed matter physics in the Netherlands. Research Policy 27 (1): 95-107.
171. Technopolis. 2010. REF research impact pilot exercise lessons-learned project: Feedback on pilot submissions.
172. Thaler Richard, H., and R. Sunstein Cass. 2008. Nudge: Improving decisions about health, wealth, and happiness. New Haven, CT: Yale University Press.
173. van den Besselaar, P., and L. Leydesdorff. 2009. Past performance, peer review and project selection: A case study in the social and behavioral sciences. Research Evaluation 18 (4): 273-288.
174. Vedung, E. 1997. Public policy and program evaluation. New Brunswick and London: Transaction Publishers.
175. www.ref.ac.uk.2014. Consistency across UOAs: REF2014. http://www.ref.ac.uk/2014/results/analysis/consistencyacrossuoas/. Accessed 20 Sep 2016.
176. Blau, P. 1964. Exchange and power in social life. New York: Wiley.
177. Collins, H., and R. Evans. 2007. Rethinking expertise. Chicago and London: University of Chicago Press.
178. Derrick, G.E., and G.N. Samuel. 2016. The evaluation scale: Exploring decisions about societal impact in peer review panels. Minerva 54 (1): 75-97.
179. Derrick, G.E., and G.N. Samuel. 2017. The future of societal impact assessment using peer review: Preevaluation training, consensus building and inter-reviewer reliability. Palgrave Communications. https://doi.org/10.1057/palcomms.2017.40
180. Donovan, C. 2011. State of the art in assessing research impact: Introduction to a special issue. Research Evaluation 20 (3): 175-179.
181. Evans, Robert, and Harry Collins. 2010. Interactional expertise and the imitation game. In Trading zones and interactional expertise, ed. M. Gorman, 53-70. Cambridge: MIT Press.
182. Greenhalgh, T., J. Raftery, S. Hanney, and M. Glover. 2016. Research impact: A narrative review. BMC Medicine 14 (1): 78.
183. Huutoniemi, K. 2010. Evaluating interdisciplinary research. Oxford: Oxford University Press.
184. Lamont, M. 2009. How professors think: Inside the curious world of academic judgement. Cambridge, MA: Harvard University Press.
185. Luukkonen, T. 2012. Conservatism and risk-taking in peer review: Emerging ERC practices. Research Evaluation 21: 48-60.
186. Pier, E.L., J. Raclaw, A. Kaatz, M. Brauer, M. Carnes, M.J. Nathan, and C.E. Ford. 2017. ‘Your comments are meaner than your score’: Score calibration talk influence intra- and inter-panel variability during scientific grant peer review. Research Evaluation 26 (1): 1-14.
187. Samuel, G.N., and G.E. Derrick. 2015. Societal impact evaluation: Exploring evaluator perceptions of the characterization of impact under the REF2014. Research Evaluation 24 (3): 229-241.
188. Sinclair, C., and P. Foley. 2009. Skin cancer prevention in Australia. British Journal of Dermatology 161 (s3): 116-123.
189. Spaapen, J., and L. Van Drooge. 2011. Introducing ‘productive interactions’ in social impact assessment. Research Evaluation 20 (3): 211-218.
190. Sunstein, C.R., and R. Hastie. 2015. Wiser: Getting beyondgroupthink to make groups smarter. Boston, MA: Harvard Business Review Press.
191. Bonner, B.L., and M.R. Baumann. 2012. Leveraging members expertise to improve knowledge transfer and demonstrability in groups. Journal of Personality and Social Psychology 102 (2): 337-350.
192. Chubin, D.E., and E.J. Hackett. 1990. Peerless science: Peer review and US science policy. Albany: State University of New York Press.
193. Collins, H. 2004. Interactional expertise as a third kind of knowledge. Phenomenology and the Cognitive Sciences 3: 125-143.
194. Collins, H. 2010. Interdisciplinary peer review and interactional expertise. Sociologica 3. https://doi.org/10.2383/33638.
195. Collins, H., and R. Evans. 2007. Rethinking expertise. Chicago and London: University of Chicago Press.
196. Cooper, J., K.A. Kelly, and K. Weaver. 2001. Attitudes, norms, and social groups. In Blackwell Handbook of social psychology: Group processes, ed. M.A. Hogg and R.S. Tindale, 259-282. Oxford: Blackwell.
197. Greene, J.C. 2000. Challenges in practicing deliberative democratic evaluation. New Directions for Evaluation 2000 (85): 13-26.
198. HEFCE. 2010. REF2014: Panel criteria and working methods.http://www.ref.ac.uk/media/ref/content/pub/panelcriteriaandworkingmethods/01_12.pdf. Accessed 1 Mar 2016.
199. Jasanoff, S. 1990. The fifith branch: Science advisors as policymakers. Cambridge, MA: Harvard University Press.
200. Jasanoff, S. 2003. (No?) Accounting for expertise. Science and Public Policy 30 (3): 157-162.
201. Jasanoff, S. 2012. Science and public reason. London and New York: Routledge (Taylor & Francis Group).
202. Lamont, M. 2009. How professors think: Inside the curious world of academic judgement. Cambridge, MA: Harvard University Press.
203. Lamont, M., and K. Huutoniemi. 2011. Comparing customary rules of fairness: Evaluative practices in various types of peer review panels. In Social knowledge in the making, ed. C. Camie, N. Gross, and M. Lamont, 209-232. Chicago, IL: University of Chicago.
204. Langfeldt, L. 2001. The decision-making constraints and processes of grant peer review, and their effects on the review outcome. Social Studies of Science 31 (6): 820-841.
205. Levi, D. 2015. Group dynamics for teams. London: Sage Publications.
206. Mark, M.M., and R.L. Shotland. 1985. Stakeholder-based evaluation and value judgments. Evaluation Review 9 (5): 605-626.
207. Mow, K.E. 2010. Inside the black box: Research grant funding and peer review in Australian research councils. Saarbrücken: Lambert Academic Publishing.
208. Samuel, G.N., and G.E. Derrick. 2015. Societal impact evaluation: Exploring evaluator perceptions of the characterization of impact under the REF2014. Research Evaluation 24 (3): 229-241.
209. van Arensbergen, P., I. van der Weijden, and P. van den Besselaar. 2014. The selection of talent as a group process. A literature review on the social dynamics of decision making in grant panels. Research Evaluation 23 (4): 298-311.
210. von Schomberg, R. 2013. A vision of responsible innovation. In Responsible innovation, ed. R. Owen, M. Heintz, and J. Bessant. London: John Wiley.
211. Abercrombie, N., S. Hill, and B. Turner. 1994. The Penguin dictionary of sociology. 3rd ed. New York: Penguin Group.
212. Aldag, R.J., and S.R. Fuller. 1993. Beyond fiasco: A reappraisal of the group- think phenomenon and a new model of group decision processes. Psychological Bulletin 113 (3): 533.
213. Bailar, J. 2011. Reliability, fairness, objectivity and other inappropriate goals in peer review. Behavioral and Brain Sciences 14 (01): 137-138.
214. Barge, S., and H. Gehlbach. 2012. Using the theory of satisficing to evaluate the quality of survey data. Research in Higher Education 53 (2): 182-200.
215. Baron, R.S. 2005. So right it’s wrong: Groupthink and the ubiquitous nature of polarised group decision making. Advances in Experimental Social Psychology 37: 219-253.
216. Byron, M. 2005. Simon’s revenge: Or, incommensurability and satisficing. Analysis 65 (4): 311-315.
217. Cole, S., J.R. Cole, and G.A. Simon. 1981. Chance and consensus in peer review. Science 214 (4523): 881-886.
218. Derrick, G.E., and G.N. Samuel. 2014. The impact evaluation scale: Group panel processes and outcomes in societal impact evaluation. Social Science and Medicine, in press.
219. Eckberg, D.L. 1991. When nonreliability of reviews indicates solid science. Behavioural and Brain Sciences 14 (1): 145-146.
220. Harkins, S., B. Latane, and K. Williams. 1980. Social loafing: Allocating effort or taking it easy? Journal of Experimental Social Psychology 16: 457-465.
221. Harkins, S., and R.E. Petty. 1982. Effects of task difficulty and task uniqueness on social loafing. Journal of Personality and Social Psychology 43: 1214-1229.
222. Huutoniemi, K. 2010. Evaluating interdisciplinary research. Oxford: Oxford University Press.
223. Ingham, A., G. Levinger, J. Graves, and v. Peckham. 1974. The Ringelmann effect: Studies of group size and group performance. Journal of Experimental Social Psychology 10: 371-384.
224. Janis, I.L. 1982. Groupthink: Psychological studies of policy decisions and fiascoes. Boston, MA: Houghton Mifflin Company.
225. Kadar, N. 2010. Systemic bias in peer review: Suggested causes, potential remedies. Journal of Laparoendoscopic & Advanced Surgical Techniques 20 (2): 123-128.
226. Karau, S.J., and J.W. Hart. 1998. Group cohesiveness and social loafing: Effects of social interaction manipulation on individual motivation within groups. Group Dynamics: Theory, Research, and Practice 2 (3): 185-191.
227. Kerr, N.L. 1983. Motivation losses in small groups: A social dilenma analysis. Journal of Personality and Social Psychology 45 (4): 819-828.
228. Kravitz, D.A., and B. Martin. 1986. Ringelmann rediscovered: The original article. Journal of Personality and Social Psychology 50 (5): 936-941.
229. Lamont, M. 2009. How professors think: Inside the curious world of academic judgement. Cambridge, MA: Harvard University Press.
230. Lamont, M., and K. Huutoniemi. 2011. Comparing customary rules of fairness: Evaluative practices in various types of peer review panels. In Social knowledge in the making, ed. Charles Camic, Neil Gross, and Michèle Lamont, 209-232. Chicago: University of Chicago Press.
231. Langfeldt, L. 2001. The decision-making constraints and processes of grant peer review, and their effects on the review outcome. Social Studies of Science 31 (6): 820-841.
232. Langfeldt, L. 2004. Expert panels evaluating research: Decision-making and sources of bias. Research Evaluation 13 (1): 51-62.
233. Langfeldt, L., and S. Kyvik. 2011. Researchers as evaluators: Tasks, tensions and politics. Higher Education 62: 199-212.
234. Latane, B., K. Williams, and S. Harkins. 1979. Many hands make light the work: The causes and consequences of social loafing. Journal of Personality and Social Psychology 37 (6): 822-832.
235. Laudel, G. 2006. Conclave in the Tower of Babel: How peers review interdisciplinary research proposals. Research Evaluation 15 (1): 57-68.
236. Lee, C.J., C.R. Sugimoto, G. Zhang, and B. Cronin. 2013. Bias in peer review. Journal of the American Society for Information Science and Technology 64 (1): 2-17.
237. Marsh, H.W, U.W Jayasinghe, and N.W Bond. 2008. Improving the peerreview process for grant applications: Reliability, validity, bias and generaliz- ability. American Psychologist 63 (3): 160-168.
238. Petty, R.E., S.G. Harkins, K. Williams, and B. Latane. 1977. The effects of group size on cognitive effort and evaluation. Personality and Social Psychology Bulletin 3: 579-582.
239. Ringlemann, M. 1913. Recherches sur les moteurs animes: Travail de l’homme. Annales de l’Institut NationalAgronomique 12 (1): 1-40.
240. Samuel, G.N., and G.E. Derrick. 2015. Societal impact evaluation: Exploring evaluator perceptions of the characterization of impact under the REF2014. Research Evaluation 24 (3): 229-241.
241. Schmid, A.A. 2004. Conflict and cooperation: Institutional and behavioural economics. Malden, MA: Blackwell.
242. Schwartz, B., A. Ward, J. Monterosso, S. Lyubomirsky, K. White, and D.R. Lehman. 2002. Maximizing versus satisficing: Happiness is a matter of choice. Journal of Personality and Social Psychology 83 (5): 1178.
243. Simms, A., and T. Nichols. 2014. Social loafing: A review of the literature. Journal of Management Policy and Practice 15 (1): 58.
244. Simon, H.A. 1957. Models of man. New York: Wiley.
245. Simon, H.A. 1972. Theories of bounded rationality. Decision and Organization 1 (1): 161-176.
246. Simon, H.A., and A.C. Stedry. 1968. Psychology and economics. In Handbook of social psychology, ed. G. Lindzey and E. Aronson. Reading, MA: Addision-Wesley.
247. Solomon, M. 2006. Groupthink versus the wisdom of crowds: The social epis- temology of deliberation and dissent. The Southern Journal of Philosophy 44 (S1): 28-42.
248. Sorkin, R.D., S. Luan, and J. Itzkowitz. 2001. Rational models of social conformity and social loafing. Subjective probability, utility and decision making conference. Amsterdam: Netherlands.
249. Sunstein, C.R., and R. Hastie. 2015. Wiser: Getting beyondgroupthink to make groups smarter. Boston, MA: Harvard Business Review Press.
250. Williams, K., S. Harkins, and B. Latane. 1981. Identifiability as a deterrent to social loafing: Two cheering experiments. Journal of Experimental Social Psychology 40: 303-311.
251. Williams, K.D., and K.L. Sommer. 1997. Social ostracism by coworkers: Does rejection lead to loafing or compensation? Personality and Social Psychology Bulletin 23 (7): 693-706.
252. Wilson, T.D. 2005. Evolution in information behavior modelling: Wilson’s model. In Theories of Information behavior, ed. K.E. Fisher, S. Erdelez, and L. McKechnie. Medford, NJ: Information Today.
253. Cagan, R. 2013. The San Francisco declaration on research assessment. Disease Models and Mechanisms 6: 869-870.
254. Hicks, D., P.F. Wouters, L. Waltman, S. De Rijcke, and I. Rafols. 2015. The Leiden Manifesto for research metrics. Nature 520 (7548): 429.
255. Lamont, M. 2009. How professors think: Inside the curious world of academic judgement. Cambridge, MA: Harvard University Press.
256. Stern, N. 2016. Building on success and learning from experience: An independent review of the Research Excellence Framework. Department for Business, Energy & Industrial Strategy.
257. Wilsdon, J. 2016. The metric tide: Independent review of the role of metrics in research assessment and management. London: SAGE.