Effects of behavioral anchors on peer evaluation reliability

Journal of Engineering Education

Volumn 94, Issue 3, 2005, Pages 319-326

  Ohland, Matthew W ^a,d   Layton, Richard A ^b   Loughry, Misty L ^c,d   Yuhasz, Amy G ^a,d  

^a AMRL   (United States)

^b ROSE HULMAN INSTITUTE OF TECHNOLOGY   (United States)

^c CLEMSON UNIVERSITY   (United States)

^d 104 Holtzendorff Hall ^*  (United States)

Author keywords

Assessment; Behaviorally anchored rating scale; Peer evaluation

Indexed keywords

BEHAVIORAL RESEARCH; ENGINEERING EDUCATION; EVALUATION; PERSONNEL; RELIABILITY; STUDENTS;

ASSESSMENT; BEHAVIORALLY ANCHORED RATING SCALE; ENGINEERING STUDENTS; PEER EVALUATION;

SOCIAL ASPECTS;

EID: 23844535793     PISSN: 10694730     EISSN: None     Source Type: Journal    
DOI: 10.1002/j.2168-9830.2005.tb00856.x     Document Type: Article

References (26)

1. Criteria for Accrediting Engineering Programs, published by ABET, Inc., Baltimore, Maryland, www.abet.org/images/Criteria/E001 04-05 EAC Criteria 11-20-03.pdf, 2004.
2. Johnson, D.W., Johnson, R.T., and Smith, K.A., Active Learning: Cooperation in the College Classroom, Edina, Minn.: Interaction Book Co., 1998.
3. Millis, B.J., and Cottell, Jr., P.G., Cooperative Learning for Higher Education Faculty, Phoenix, Ariz.: American Council on Education/Oryx Press, 1998, p. 194.
4. McGourty, J., and De Meuse, K., The Team Developer: An Assessment and Skill Building Program, New York, New York: J. Wiley and Company, 2000.
5. Kaufman, D.B., Felder, R.M., and Fuller, H., "Accounting for Individual Effort in Cooperative Learning Teams," Journal of Engineering Education Vol. 89, No. 2, 2000, pp. 133-140.
6. Moran, L., Musselwhite, E., Zenger, J.H., Keeping Teams on Track What to Do When the Going Gets Rough, Chicago, 111.: Irwin Professional Publishing, 1996, pp. 271-274 and 280-283.
7. Williams, K., Harkins, S., and Latané, B., "Identifiability as a Deterrent to Social Loafing: Two Cheering Experiments," Journal of Personality and Social Psychology, Vol. 40, 1981, pp. 303-311.
8. Conway, J.M., and Huffcutt, A.I., "Psychometric Properties of Multisource Performance Ratings: A Meta-analysis of Subordinate, Supervisory, Peer, and Self-ratings," Human Performance, Vol. 10, 1997, pp. 331-360.
9. Viswesvaran, C., Ones, D.S., and Schmidt, F.L., "Comparative Analysis of the Reliability of Job Performance Ratings," Journal of Applied Psychology, Vol. 81, 1996, pp. 557-574.
10. Harris, M.M., and Schaubroeck, J., "A Meta-analysis of Self-supervisor, Self-peer, and Peer-supervisor Ratings," Personnel Psychology, Vol. 41, 1988, pp. 43-62.
11. Schmitt, N., Gooding, R.Z., Noe, R.A., and Kirsch, M., "Metaanalyses of Validity Studies Published between 1964 and 1982 and the Investigation of Study Characteristics," Personnel Psychology, Vol. 37, 1984, pp. 407-422.
12. Brown, R.W., "Autorating: Getting Individual Marks from Team Marks and Enhancing Teamwork," Proceedings, 1995 Frontiers in Education Conference, IEEE/ASEE, Pittsburgh, November, 1995.
13. Sheppard, S., Chen, H.L., Schaeffer, E., Steinbeck, R., Neumann, H., and Ko, P., Peer Assessment of Student Collaborative Processes in Under-graduate Engineering Education, Final Report to the National Science Foundation, Award Number 0206820, NSF Program 7431 CCLI-ASA, 2004.
14. Van Duzer, E., and McMartin, F., "Methods to Improve the Validity and Sensitivity of a Self/Peer Assessment Instrument," IEEE Transactions on Education, Vol. 43, No. 2, May 2000, pp. 153-158.
15. Finelli, C.J., "Assessing Improvement in Students' Team Skills and Using a Learning Style Inventory to Increase It," Proceedings, 2001 Frontiers in Education Conference, IEEE/ASEE, Reno, Nevada, October, 2001.
16. Ohland, M.W., and Finelli, C.J., "Peer Evaluation in a Mandatory Cooperative Education Environment," Proceedings, 2001 American Society of Engineering Education Conference and Exposition, Albuquerque, New Mexico, June, 2001.
17. Layton, R.A., and Ohland, M.W., "Peer Evaluations in Teams of Predominantly Minority Students," Proceedings, 2000 American Society of Engineering Education Conference and Exposition, Washington, D.C., 2000.
18. Ohland, M.W., and Layton, R.A., "Comparing the Reliability of Two Peer Evaluation Instruments," Proceedings, 2000 American Society of Engineering Education Conference and Exposition, Washington, D.C., 2000.
19. Layton, R.A., and Ohland, M.W., "Peer Evaluations Revisited: Focus on Teamwork, Not Ability," Proceedings, 2001 American Society of Engineering Education Conference and Exposition, Albuquerque, New Mexico, June, 2001.
20. Ohland, M.W., Loughry, M.L., Carter, R.L., Bullard, L.F., Felder, R.M., Finelli, C.J., Layton, R.A., and Schmucker, D.G., "Developing a Peer Evaluation Instrument that is Simple, Reliable, and Valid," in press, Proceedings, 2005 American Society of Engineering Education Conference and Exposition, Portland, Oregon, June, 2005.
21. Crocker, L., and Algina, J., Introduction to Classical and Modern Test Theory, Chicago, Ill.: Holt, Rinehart and Winston, Inc., 1986, p. 143, 157ff.
22. Kubiszyn, T., and Borich, G., Educational Testing and Measurement: Classroom Application and Practice, 4th Ed., Harper Collins, New York, 1993, Chapter 15.
23. Ohland, M.W., Loughry, M.L., Carter, R.L., and Yuhasz, A.G., "Designing a Peer Evaluation Instrument that is Simple, Reliable, and Valid" Proceedings, 2004 American Society of Engineering Education Conference and Exposition, Salt Lake City, Utah, June 2004.
24. Lejk, M., and Wyvill, M., "The Effect of the Inclusion of Self-assessment with Peer Assessment of Contributions to a Group Project: A Quantitative Study of Secret and Agreed Assessments," Assessment & Evaluation in Higher Education, Vol. 26, 2001, pp. 551-561.
25. Thompson, R.S., "Relative Validity of Peer and Self-evaluations in Self-directed Interdependent Work teams," Proceedings, 2001 Frontiers in Education Conference, IEEE/ASEE, pp. T4A/9-T4A/14.
26. Smith, P.C., and Kendall, L.M., "Retranslation of Expectations: An Approach to the Construction of Unambiguous Anchors for Rating Scales," Journal of Applied Psychology, Vol. 47, No. 2, 1963, pp. 149-155.