'Your comments are meaner than your score': Score calibration talk influences intra- and inter-panel variability during scientific grant peer review

Research Evaluation

Volumn 26, Issue 1, 2017, Pages 1-14

  Pier, Elizabeth L ^a   Raclaw, Joshua ^a,b   Kaatz, Anna ^a   Brauer, Markus ^a   Carnes, Molly ^a   Nathan, Mitchell J ^a   Ford, Cecilia E ^a  

^a UNIVERSITY OF WISCONSIN MADISON   (United States)

^b WEST CHESTER UNIVERSITY   (United States)

Author keywords

Collaboration; Decision making; Discourse analysis; Peer review

Indexed keywords

EID: 85018209939     PISSN: 09582029     EISSN: 14715449     Source Type: Journal    
DOI: 10.1093/reseval/rvw025     Document Type: Article

References (70)

1. Aronson, E. et al. (1978) The Jigsaw Classroom. Beverly Hills, CA: Sage Publishing Company.
2. Asch, S. E. (1951) 'Effects of Group Pressure on the Modification and Distortion of Judgments'. In: Guetzkow H (ed.) Groups, Leadership and Men, pp. 177-90. Pittsburgh, PA: Carnegie Press.
3. Asch, S. E. (1955) 'Opinions and Social Pressure', Scientific American, 193: 33-5.
4. Baker, M. et al. (1999) 'The Role of Grounding in Collaborative Learning Tasks.' In: Dillenbourg P (ed.) Collaborative Learning: Cognitive and Computational Approaches, pp. 31-63. Oxford: Pergamon.
5. Barron, B. (2000) 'Achieving Coordination in Collaborative Problem-Solving Groups', The Journal of the Learning Sciences, 9/4: 403-36.
6. Bernardin, H. J., and Hennessey., H. W., Jr, and Peyrefitte, J. S. (1995) 'Age, Racial, and Gender Bias as a Function of Criterion Specificity: A Test of Expert Testimony', Human Resource Management Review, 5: 63-77.
7. Bornstein, B. H. (1999) 'The Ecological Validity of Jury Simulations: Is the Jury Still Out?', Law and Human Behavior, 23/1: 75-91.
8. Brown, A. L. et al. (1993) 'Distributed Expertise in the Classroom'. In: Salomon G (ed.) Distributed Cognitions: Psychological and Educational Considerations, pp. 188-228. Cambridge: Cambridge University Press.
9. Chubin, D. E., and Hackett, E. J. (1990) Peerless Science: Peer Review and U. S. Science Policy. Albany, NY: State University of New York Press.
10. Cialdini, R. B., and Goldstein, N. J. (2004) 'Social Influence: Compliance and Conformity', Annual Review of Psychology, 55: 591-621.
11. Cicchetti, D. V. (1991) 'The Reliability of Peer Review for Manuscript and Grant Submissions: A Cross-Disciplinary Investigation', Behavioral and Brain Sciences, 14: 119-35.
12. Clark, H. H., and Brennan, S. A. (1991) 'Grounding in Communication'. In: Levine JM, and Levine LB, Teasley SD (eds.) Perspectives on Socially Shared Cognition, pp. 127-49. Washington, DC: American Psychological Association.
13. Cole, J., and Cole, S. (1981) Peer Review in the National Science Foundation: Phase II of a Study. Washington, DC: National Academy of Sciences.
14. Cole, S., and Cole., J. R., and Simon, G. A. (1981) 'Chance and Consensus in Peer Review', Science, 214/4523: 881-6.
15. Cole, S., Rubin, L., and Cole, J. R. (1978) Peer Review in the National Science Foundation: Phase I of a Study. Washington, DC: National Academy of Sciences.
16. Creswell, J. W., and Plano Clark, V. L. (2011) Designing and Conducting Mixed Methods Research 2nd edn. Thousand Oaks, CA: SAGE Publications, Inc.
17. Crosbie-Burnett, M., and Eisen, M. (1992) 'Simulated Divorced and Remarried Families: An Experiential Teaching Technique', Family Relations, 41/1: 54-8.
18. Deutsch, M., and Gerard, H. B. (1955) 'A Study of Normative and Informative Social Influences upon Individual Judgment', Journal of Abnormal Social Psychology, 51: 629-36.
19. Dotger, B. H., Dotger, S. C, and Maher, M. J. (2010) 'From Medicine to Teaching: The Evolution of the Simulated Interaction Model', Innovative Higher Education, 35: 129-41.
20. Fogelholm, M. et al. (2012) 'Panel Discussion Does Not Improve Reliability of Peer Review for Medical Research Grant Proposals', Journal of Clinical Epidemiology, 65/1: 47-52.
21. Foss, R. D. (1976) 'Group Decision Processes in the Simulated Trial Jury', Sociometry, 39/4: 305-16.
22. Gallo, S. A., and Carpenter., A. S., and Glisson, S. R. (2013) 'Teleconferenec versus Face-to-Face Scientific Peer Review of Grant Application: Effects on Review Outcomes', PLoS ONE, 8/8: e71693.
23. Gillespie, G. W., Jr, Chubin, D. E., and Kurzon, G. M. (1985) 'Experience with NIH Peer Review: Researchers' Cynicism and Desire for Change', Science, Technology, & Human Values, 10/3: 44-54.
24. Ginther, D. K. et al. (2011) 'Race, Ethnicity, and NIH Research Awards', Science, 333/6045: 1015-9.
25. Ginther, D. K. et al. (2012) 'Are Race, Ethnicity, and Medical School Affiliation Associated with NIH R01 Type Award Probability for Physician Investigators?', Academic Medicine, 87/11: 1516-24.
26. Hackett, E. J. (1987) 'Funding and Academic Research in the Life Sciences: Results of an Exploratory Study', Science & Technology Studies, 5/3-4: 134-47.
27. Hayes, A. F., and Krippendorff, K. (2007) 'Answering the Call for a Standard Reliability Measure for Coding Data', Communication Methods and Measures, 1/1: 77-89.
28. Heilman, M. E., and Haynes, M. C. (2008) 'Subjectivity in the Appraisal Process: A Facilitator of Gender Bias in Work Settings'. In: Borgida E and Fiske ST (eds.) Beyond Common Sense: Psychological Science in Court. Oxford: Blackwell Publishing, Ltd.
29. Helitzer, D. L. et al. (2011) 'A Randomized Controlled Trial of Communication Training with Primary Care Providers to Improve Patient-Centeredness and Health Risk Communication', Patient Education and Counseling, 82: 21-9.
30. Heritage, J. C., and Watson, D. R. (1979) 'Formulations as Conversational Objects'. In: Psathas G (ed.) Everyday Language: Studies in Ethnomethodology, pp. 123-62. New York: Irvington.
31. Kaatz, A. et al. (2015) 'A Quantitative Linguistic Analysis of National Institutes of Health R01 Application Critiques from Investigators at One Institution', Academic Medicine, 90/1: 69-75.
32. Kane, M. N. (2003) 'Teaching Direct Practice Techniques for Work with Elders with Alzheimer's Disease: A Simulated Group Experience', Educational Gerontology, 29: 777-94.
33. Kerr, N. L., and Nerenz., D. R., and Herrick, D. (1979) 'Role Playing and the Study of Jury Behavior', Sociological Methods & Research, 7/3: 337-5.
34. Klahr, D. (1985) 'Insiders, Outsiders, and Efficiency in a National Science Foundation Panel', American Psychologist, 40/2: 148-54.
35. Koshik, I. (2002) 'A Conversation-Analytic Study of Yes/No questions Which Convey Reversed Polarity Assertions', Journal of Pragmatics, 34: 1851-77.
36. Krippendorff, K. (2004) Content Analysis: An Introduction to its Methodology, 2nd edn. Thousand Oaks, CA: Sage Publishing.
37. Krippendorff, K. (2008) 'Systematic and Random Disagreement and the Reliability of Nominal Data', Communication Methods and Measure, 2/4: 323-38.
38. Krippendorff, K. (2013) Content Analysis: An Introduction to its Methodology, 3rd edn. Thousand Oaks, CA: Sage Publishing.
39. Kruijver, I. P. M. et al. (2001) 'Communication Between Nurses and Simulated Patients with Cancer: Evaluation of a Communication Training Programme', European Journal of Oncology Nursing, 5/3: 140-50.
40. Lamont, M. (2009). How Professors Think: Inside the Curious World of Academic Judgment. Cambridge, MA: Harvard University Press.
41. Langfeldt, L. (2001) 'The Decision-Making Constraints and Processes of Grant Peer Review, and Their Effects on the Review Outcome', Social Studies of Science, 31/6: 820-41.
42. Lerner, J. S., and Tetlock, P. E. (1999) 'Accounting for the Effects of Accountability', Psychological Bulletin, 125: 255-75.
43. Levinson, S. (1992) 'Activity Types and Language'. In: Drew P and Heritage J (eds.) Talk at Work: Interaction in Institutional Settings, pp. 66-100. Cambridge: Cambridge University Press.
44. Ley, T. J., and Hamilton, B. H. (2008) 'The Gender Gap in NIH Grant Applications', Science, 322/5907: 1472-4.
45. Marsh, H. W., and Jayasinghe., U. W., and Bond, N. W. (2008) 'Improving the Peer Review Process for Grant Applications: Reliability, Validity, Bias and Generalizability', American Psychologist, 63/3: 160-8.
46. McCullough, J. (1989) 'First Comprehensive Survey of NSF Applicants Focuses on Their Concerns About Proposal Review', Science, Technology, & Human Values, 14/1: 78-88.
47. McQuiston-Surrett, D., and Saks, M. J. (2009) 'The Testimony of Forensic Identification Science: What Expert Witnesses Say and What Factfinders Hear', Law and Human Behavior, 33/5: 436-53.
48. Moyer, S. M. (2016) 'Large Group Simulation: Using Combined Teaching Strategies to Connect Classroom and Clinical Learning', Teaching and Learning in Nursing, 11: 67-73.
49. National Institutes of Health (NIH). (2013) Enhancing Peer Review Survey Results Report, accessed 28 July 2016.
50. National Institutes of Health (NIH). (2015) Scoring System and Procedure, accessed 27 April 2016.
51. National Institutes of Health (NIH). (2016) NIH Budget, accessed 24 March 2016.
52. Obrecht, M., Tibelius, K., and D'Aloisio, G. (2007) 'Examining the Value Added by Committee Discussion in the Review of Applications for Research Awards', Research Evaluation, 16/2: 70-91.
53. Olbrecht, M., and Bornmann, L. (2010) 'Panel Peer Review of Grant Applications: What Do We Know From Research in Social Psychology on Judgment and Decision-Making in Groups?', Research Evaluation, 19/4: 293-304.
54. Paulus, T. M. (2009) 'Online but Off-Topic: Negotiating Common Ground in Small Learning Groups', Instructional Science, 37: 227-45.
55. Pennington, J., and Schlenker, B. R. (1999) 'Accountability for Consequential Decisions: Justifying Ethical Judgments to Audiences', Personality and Social Psychology Bulletin, 25: 1067-81.
56. Pier, E. L. et al. (2015). Studying the study section: How group decision making in person and via videoconferencing affects the grant peer review process (WCER Working Paper No. 2015-6). Retrieved from University of Wisconsin-Madison, Wisconsin Center for Education Research website:
57. Pohlhaus, J. R. et al. (2011) 'Sex Differences in Application, Success, and Funding Rates for NIH Extramural Programs', Academic Medicine, 86/6: 759-67.
58. Porter, A. L., and Rossini, F. A. (1985) 'Peer Review of Interdisciplinary Research Proposals', Science, Technology, & Human Values, 10/3: 33-8.
59. Quinn, A., and Schlenker, B. R. (2002) 'Can Accountability Produce Independence? Goals as Determinants of the Impact of Accountability on Conformity', Personality and Social Psychology Bulletin, 28: 472-83.
60. Raclaw, J., and Ford, C. E. (2015) 'Laughter as a resource for managing delicate matters in peer review meetings'. Paper presented at the Conference on Culture, Language, and Social Practice, Boulder, CO.
61. Resnick, L. B. et al. (1993) 'Reasoning in Conversation', Cognition and Instruction, 11: 347-64.
62. Romano, J. L. (1998) 'Simulated Group Counseling: An Experiential Training Model for Group Work', The Journal for Specialists in Group Work, 23/2: 119-32.
63. Romano, J. L., and Sullivan, B. A. (2000) 'Simulated Group Counseling for Group Work Training: A Four-Year Research Study of Group Development', The Journal for Specialists in Group Work, 25/4: 366-75.
64. Schegloff, E. A. (2007) Sequence Organization in Interaction: A Primer in Conversation Analysis, Vol 1. Cambridge: Cambridge University Press.
65. Sigelman, L., and Scioli, F. P. Jr (1987) 'Retreading Familiar Terrain. Bias, Peer Review, and the NSF Political Science Program', Political Science & Politics, 20/1: 62-9.
66. Stroben, F. et al. (2016) 'A Simulated Night Shift in the Emergency Room Increases Students' Self-Efficacy Independent of Role Taking Over During Simulation', BMC Medical Education, 16: 177-83.
67. Vygotsky, L. (1978) Mind in Society. London: Harvard University Press.
68. Wenneras, C., and Wold, A. (1997) 'Nepotism and Sexism in Peer-Review', Nature, 387: 341-3.
69. Wessely, S. (1998) 'Peer Review of Grant Applications: What Do We Know?', Lancet, 352: 301-5.
70. Young, O. M., and Parviainen, K. (2014) 'Training Obstetrics and Gynecology Residents to be Effective Communicators in the Era of the 80-Hour Workweek: A Pilot Study', BMC Research Notes, 7: 455-60.