The comprehensive assessment of team member effectiveness: Development of a behaviorally anchored rating scale for self- and peer evaluation

Academy of Management Learning and Education

Volumn 11, Issue 4, 2012, Pages 609-630

  Ohland, Matthew W ^a   Loughry, Misty L ^b   Woehr, David J ^c   Bullard, Lisa G ^d   Felder, Richard M ^d   Finelli, Cynthia J ^e   Layton, Richard A ^f   Pomeranz, Hal R ^g   Schmucker, Douglas G ^h  

^a PURDUE UNIVERSITY   (United States)

^b GEORGIA SOUTHERN UNIVERSITY   (United States)

^c UNIVERSITY OF NORTH CAROLINA AT CHARLOTTE   (United States)

^d NORTH CAROLINA STATE UNIVERSITY   (United States)

^e UNIVERSITY OF MICHIGAN   (United States)

^f ROSE HULMAN INSTITUTE OF TECHNOLOGY   (United States)

^g Deer Run Asociates ^*  (United States)

^h STANFORD UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84879470456     PISSN: 1537260X     EISSN: None     Source Type: Journal    
DOI: 10.5465/amle.2010.0177     Document Type: Article

References (96)

1. ABET. 2000. Criteria for Accrediting Engineering Programs. Accreditation Board for Engineering and Technology [Abet], Baltimore, MD.
2. Aggarwal, P., & O'Brien, C. L. 2008. Social loafing on group projects structural antecedents and effect on student satisfaction. Journal of Marketing Education, 30: 255-264.
3. Alsop, R. 2002. Playing well with others. The Wall Street Journal, September 9, 2002: R11.
4. Bacon, D. R., Stewart, K. A., & Silver, W. S. 1999. Lessons from the best and worst student team experiences: How a teacher can make the difference. Journal of Management Education, 23: 467-488.
5. Baker, D. F. 2008. Peer assessment in small groups: A comparison of methods. Journal of Management Education, 33: 183-209.
6. Bamberger, P. A., Erev, I., Kimmel, M., & Oref-Chen, T. 2005. Peer assessment, individual performance, and contribution to group processes: The impact of rater anonymity. Group and Organization Management, 30: 344-377.
7. Barrick, M. R., Stewart, G. L., Neubert, M. J., & Mount, M. K. 1998. Rating member ability and personality to work-team processes and team effectiveness. Journal of Applied Psychology, 83: 377-391.
8. Bell, M. P. 2010. From the editors: What if . . . ? Diversity, scholarship, and impact. Academy of Management Learning and Education, 9: 5-10.
9. Bettenhausen, K. L., & Fedor, D. B. 1997. Peer and upward appraisals: A comparison of their benefits and problems. Group and Organization Management, 22: 236-263.
10. Boni, A. A., Weingart, L. R., & Evenson, S. 2009. Innovation in an academic setting: Designing and leading a business through market-focused, interdisciplinary teams. Academy of Management Learning and Education, 8: 407-417.
11. Brennan, R. L. 1994. Variance components in generalizability theory. In C. R. Reynolds (Ed.), Cognitive assessment: A multidisciplinary perspective: 175-207. New York: Plenum Press.
12. Brennan, R. L. 2000. Performance assessments from the perspective of generalizability theory. Applied Psychological Measurement, 24: 339-353.
13. Brooks, C. M., & Ammons, J. L. 2003. Free riding in group projects and the effects of timing, frequency, and specificity of criteria in peer assessments. Journal of Education for Business, 78: 268-272.
14. Brutus, S., & Donia, M. B. 2010. Improving the effectiveness of students in groups with a centralized peer evaluation system. Academy of Management Learning and Education, 9: 652-662.
15. Brutus, S., Donia, M., & Ronen, S. In press. Can business students learn to evaluate better? Evidence from repeated exposure to a peer evaluation process. Academy of Management Learning and Education.
16. Burdett, J. 2003. Making groups work: University students' perceptions. International Education Journal, 4: 177-190.
17. Burdett, J., & Hastie, B. 2009. Predicting satisfaction with group work assignments. Journal of University Teaching and Learning Practice, 6(1): 62-71.
18. Calloway School of Business and Accountancy of Wake Forest University. A Report on Recruiters' Perceptions of Undergraduate Business Schools and Students. February 2004.
19. Campbell, J. P., Dunnette, M. D., Arvey, R. D., & Hellervik, L. V. 1973. The development and evaluation of behaviorally based rating scales. Journal of Applied Psychology, 57: 15-22.
20. Chapman, K. J., & van Auken, S. 2001. Creating positive group project experiences: An examination of the role of the instructor on students' perceptions of group projects. Journal of Marketing Education, 23: 117-127.
21. Chen, G., Donahue, L. M., & Klimoski, R. J. 2004. Training undergraduates to work in organizational teams. Academy of Management Learning and Education, 3: 27-40.
22. Chen, Y., & Lou, H. 2004. Students' perceptions of peer evaluation: An expectancy perspective. Journal of Education for Business, 79: 275-282.
23. Conway, J. M., & Huffcutt, A. I. 1997. Psychometric properties of multisource performance ratings: A meta-analysis of subordinate, supervisory, peer, and self-ratings. Human Performance, 10: 331-360.
24. Cronbach, L. J., Glesser, G. C., Nanda, H., & Rajaratnam, N. 1972. The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: John Wiley & Sons.
25. Davis, D., Trevisan, M., Gerlick, R., Davis, H., McCormack, J., Beyerlein, S., Thompson, P., Howe, S., Leiffer, P., & Brackin, P. 2010. Assessing team member citizenship in capstone engineering design courses. International Journal of Engineering Education, 26: 1-13.
26. Dochy, F., Segers, M., & Sluijsmans, D. 1999. The use of self-, peer and co-assessment in higher education: A review. Studies in Higher Education, 24: 331-350.
27. Dominick, P. G., Reilly, R. R., & McGourty, J. W. 1997. The effects of peer feedback on team member behavior. Group and Organization Management, 22: 508-520.
28. Druskat, V. U., & Wolff, S. B. 1999. Effects and timing of developmental peer appraisals in self-managing work groups. Journal of Applied Psychology, 84: 58-74.
29. Erez, A., LePine, J. A., & Elms, H. 2002. Effects of rotated leadership and peer evaluation on the functioning and effectiveness of self-managed teams: A quasi-experiment. Personnel Psychology, 55: 929-248.
30. Felder, R. M., & Brent, R. 2007. Cooperative learning. In P. A. Mabrouk, ed., Active learning: Models from the analytical sciences, ACS Symposium Series, 970: 34-53. Washington, DC: American Chemical Society.
31. Fellenz, M. R. 2006. Toward fairness in assessing student groupwork: A protocol for peer evaluation of individual contributions. Journal of Management Education, 30: 570-591.
32. Gatfield, T. 1999. Examining student satisfaction with group projects and peer assessment. Assessment and Evaluation in Higher Education, 24: 365-377.
33. Gueldenzoph, L. E., & May, G. L. 2002. Collaborative peer evaluation: Best practices for group member assessments. Business Communication Quarterly, 65: 9-20.
34. Hackman, J. R. 1987. The design of work teams. In Handbook of organizational behavior: 315-342. Englewood Cliffs, NJ: Prentice Hall.
35. Hansen, R. S. 2006. Benefits and problems with student teams: Suggestions for improving team projects. Journal of Education for Business, 82: 11-19.
36. Harris, M. M., & Schaubroeck, J. 1988. A meta-analysis of selfsupervisor, self-peer, and peer-supervisor ratings. Personnel Psychology, 41: 43-62.
37. Harris, T. C., & Barnes-Farrell, J. L. 1997. Components of teamwork: Impact on evaluations of contributions to work team effectiveness. Journal of Applied Social Psychology, 27: 1694-1715.
38. Hedge, J. W., Bruskiewicz, K. T., Logan, K. K., Hanson, M. A., & Buck, D. 1999. Crew resource management team and individual job analysis and rating scale development for air force tanker crews. Technical Report: 336. Minneapolis, MN: Personnel Decisions Research Institutes, Inc.
39. Hess, P. W. 2007. Enhancing leadership skill development by creating practice/feedback opportunities in the classroom. Journal of Management Education, 31: 195-213.
40. Hooijberg, R., & Lane, N. 2009. Using multisource feedback coaching effectively in executive education. Academy of Management Learning and Education, 8: 483-493.
41. Hughes, R. L., & Jones, S. K. 2011. Developing and assessing college student teamwork skills. New Directions for Institutional Research, 2011: 53-64.
42. Inderrieden, E. J., Allen, R. E., & Keaveny, T. J. 2004. Managerial discretion in the use of self-ratings in an appraisal system: The antecedents and consequences. Journal of Managerial Issues, 16: 460-482.
43. Jassawalla, A., Sashittal, H., & Malshe, A. 2009. Students' perceptions of social loafing: It's antecedents and consequences in undergraduate business classroom teams. Academy of Management Learning and Education, 8: 42-54.
44. Jehn, K. A., & Mannix, E. A. 2001. The dynamic nature of conflict: A longitudinal study of intragroup conflict and group performance. Academy of Management Journal, 44: 238-251.
45. Johnston, L., & Miles, L. 2004. Assessing contributions to group assignments. Assessment and Evaluation in Higher Education, 29: 751-768.
46. Kaufman, D. B., Felder, R. M., & Fuller, H. 2000. Accounting for individual effort in cooperative learning teams. Journal of Engineering Education, 89: 133-140.
47. Kolb, A. Y., & Kolb, D. A. 2005. Learning styles and learning spaces: Enhancing experiential learning in higher education. Academy of Management Learning and Education, 4: 193-212.
48. Kruger, J., & Dunning, D. 1999. Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77: 1121-1134.
49. Layton, R. A., Loughry, M. L., Ohland, M. W., & Ricco, G. D. 2010. Design and validation of a web-based system for assigning members to teams using instructor-specified criteria. Advances in Engineering Education, 2: 1-28.
50. London, M., Smither, J. W., & Adsit, D. J. 1997. Accountability: The Achilles' heel of multisource feedback. Group and Organization Management, 22: 162-184.
51. Loughry, M. L., Ohland, M. W., & Moore, D. D. 2007. Development of a theory-based assessment of team member effectiveness. Educational Psychological Measurement, 67: 505-524.
52. Loyd, D. L., Kern, M. C., & Thompson, L. 2005. Classroom research: Bridging the ivory divide. Academy of Management Learning and Education, 4: 8-21.
53. MacDonald, H. A., & Sulsky, L. M. 2009. Rating formats and rater training redux: A context-specific approach for enhancing the effectiveness of performance management. Canadian Journal of Behavioural Science, 41: 227-240.
54. McCorkle, D. E., Reardon, J., Alexander, J. F., Kling, N. D., Harris, R. C., & Iyer, R. V. 1999. Undergraduate marketing students, group projects, and teamwork: The good, the bad, and the ugly? Journal of Marketing Education, 21: 106-117.
55. McGourty, J., & De Meuse, K. P. 2001. The team developer: An assessment and skill building program. Student guidebook. New York: John Wiley & Sons.
56. Mero, N. P., Guidice, R. M., & Brownlee, A. L. 2007. Accountability in a performance appraisal context: The effect of audience and form of accounting on rater response and behavior. Journal of Management, 33: 223-252.
57. Michaelsen, L. K., Knight, A. B., & Fink, L. D. (Eds.). 2004. Teambased learning: A transformative use of small groups in college teaching. Sterling, VA: Stylus Publishing.
58. Millis, B. J., & Cottell, P.G., Jr. 1998. Cooperative Learning for Higher Education Faculty. Phoenix, AZ: American Council on Education/Oryx Press.
59. Neuman, G. A., & Wright, J. 1999. Team effectiveness: Beyond skills and cognitive ability. Journal of Applied Psychology, 84: 376-389.
60. Oakley, B., Felder, R. M., Brent, R., & Elhajj, I. 2004. Turning student groups into effective teams. Journal of Student-Centered Learning, 2: 9-34.
61. Offerman, L. R., Bailey, J. R., Vasilopoulos, N. L., Seal, C., & Sass, M. 2004. The relative contribution of emotional competence and cognitive ability to individual and team performance. Human Performance, 17: 219-243.
62. Ohland, M. W., Layton, R. A., Loughry, M. L., & Yuhasz, A. G. 2005. Effects of behavioral anchors on peer evaluation reliability. Journal of Engineering Education, 94: 319-326.
63. Paswan, A. K., & Gollakota, K. 2004. Dimensions of peer evaluation, overall satisfaction, and overall evaluation: An investigation in a group task environment. Journal of Education for Business, 79: 225-231.
64. Pfaff, E., & Huddleston, P. 2003. Does it matter if I hate teamwork? What impacts student attitudes toward teamwork. Journal of Marketing Education, 25: 37-45.
65. Pope, N. K. L. 2005. The impact of stress in self- and peer assessment. Assessment & Evaluation in Higher Education, 30: 51-63.
66. Raelin, J. 2006. Does action learning promote collaborative leadership? Academy of Management Learning and Education, 5: 152-168.
67. Rosenstein, R., & Dickinson, T. L. (1996, August). The teamwork components model: An analysis using structural equation modeling. In Rosenstein, R., (Chair), Advances in definitional team research. Symposium conducted at the annual meeting of the American Psychological Association, Toronto.
68. Saavedra, R. & Kwun, S. K. 1993. Peer evaluation in selfmanaging work groups. Journal of Applied Psychology, 78: 450-462.
69. Salas, E., Sims, D. E., & Burke, C. S. 2005. Is there a "big five" in teamwork? Small Group Research, 36, 555-599.
70. Schmitt, N., Gooding, R. Z., Noe, R. A., & Kirsch, M. 1984. Metaanalyses of validity studies published between 1964 and 1982 and the investigation of study characteristics. Personnel Psychology, 37: 407-422.
71. Sharma, S., & Weathers, D. 2003. Assessing generalizability of scales used in cross-national research. Research in Marketing, 20: 287-295.
72. Shavelson, R. J., & Webb, N. M. 1991. Generalizability Theory. Newbury Park: Sage Publications, Inc.
73. Sheppard, S., Chen, H. L., Schaeffer, E., Steinbeck, R., Neumann, H., & Ko, P. 2004. Peer assessment of student collaborative processes in undergraduate engineering education. Final Report to the National Science Foundation, Award Number 0206820, NSF Program 7431 CCLI-ASA.
74. Shore, T. H., Shore, L. M., & Thornton, G. C., III 1992. Construct validity of self-and peer evaluations of performance dimensions in an assessment center. Journal of Applied Psychology, 77: 42-54.
75. Smith, P. C., & Kendall, L. M. 1963. Retranslation of expectations: An approach to the construction of unambiguous anchors for rating scales. Journal of Applied Psychology, 47: 149-155.
76. Sonnetag, S., & Volmer, J. 2009. Individual-level predictors of task-related teamwork processes: The role of expertise and self-efficacy in team meetings. Group & Organization Management, 34: 37-66.
77. Stevens, M. J., & Campion, M. A. 1999. Staffing work teams: Development and validation of a selection test for teamwork settings. Journal of Management, 25: 207-228.
78. Stevens, M. J., & Campion, M. A. 1994. The knowledge, skill, and ability requirements for teamwork: Implications for human resource management. Journal of Management, 20: 503-530.
79. Taggar, S., & Brown, T. C. 2006. Interpersonal affect and peer rating bias in teams. Small Group Research, 37: 87-111.
80. Taggar, S., & Brown, T. C. 2001. Problem-solving team behaviors: Development and validation of BOS and a hierarchical factor structure. Small Group Research, 32: 698-726.
81. The Wall Street Journal/Harris Interactive Survey. 2002. The rating criteria. The Wall Street Journal, September 9, 2002: R5.
82. Thomas, G., Martin, D., & Pleasants, K. 2011. Using self- and peer-assessment to enhance students' future-learning in higher education. Journal of University Teaching & Learning Practice, 8(1): article 5.
83. Ulloa, B. C. R., & Adams, S. G. 2004. Attitude toward teamwork and effective teaming. Team Performance Management, 10: 145-151.
84. Van Duzer, E., & McMartin, F. 2000. Methods to improve the validity and sensitivity of a self/peer assessment instrument. IEEE Transactions on Education, 43: 153-158.
85. Verzat, C., Byrne, J., & Fayolle, A. 2009. Tangling with spaghetti: Pedagogical lessons from games. Academy of Management Learning and Education, 8: 356-369.
86. Viswesvaran, C., Ones, D. S., & Schmidt, F. L. 1996. Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81: 557-574.
87. Viswesvaran, C., Schmidt, F. L., & Ones, D. S. 2005. Is there a general factor in ratings of job performance? A metaanalytic framework for disentangling substantive and error influences. Journal of Applied Psychology, 90: 108-131.
88. Walker, A. 2001. British psychology students' perceptions of group-work and peer assessment. Psychology Learning and Teaching, 1: 28-36.
89. Wang, L., MacCann, C., Zhuang, X., Liu, O. L., & Roberts, R. D. 2009. Assessing teamwork and collaboration in high school students: A multimethod approach. Canadian Journal of School Psychology, 24, 108-124.
90. Willcoxson, L. E. 2006. "It's not fair!": Assessing the dynamics and resourcing of teamwork. Journal of Management Education, 30: 798-808.
91. Woehr, D. J. 1994. Understanding frame-of-reference training: The impact of training on the recall of performance information. Journal of Applied Psychology, 79: 525-534.
92. Woehr, D. J., & Huffcutt, A. I. 1994. Rater training for performance appraisal: A quantitative review. Journal of Occupational and Organizational Psychology, 67: 189-206.
93. Young, C. B., & Henquinet, J. A. 2000. A conceptual framework for designing group projects. Journal of Education for Business, 76: 56-60.
94. Zantow, K., Knowlton, D. S., & Sharp, D. C. 2005. More than fun and games: Reconsidering the virtues of strategic management simulations. Academy of Management Learning and Education, 4: 451-458.
95. Zhang, B., & Ohland, M.W. 2009. How to assign individualized scores on a group project: An empirical evaluation. Applied Measurement in Education, 22: 290-308.
96. Zhu, J., Chen, S., & Lu, Q. 2010. Measuring member performance in multi-functional R&D teams: An empirical study with GAHP analysis. 2010 IEEE International Conference on Industrial Engineering and Engineering Management IEEM): 507-5111.