Do self-assessments work to detect workshop success? An analysis of argument and recommendation by D'Eon et al

American Journal of Evaluation

Volumn 30, Issue 1, 2009, Pages 93-105

  Lam, Tony C M ^a  

^a UNIVERSITY OF TORONTO   (Canada)

Author keywords

Outcome evaluation; Response bias; Self assessment; Training evaluation

Indexed keywords

EID: 61349163052     PISSN: 10982140     EISSN: 15570878     Source Type: Journal    
DOI: 10.1177/1098214008327931     Document Type: Article

References (39)

1. Alvarez, K., Salas, E., & Garofano, C.M. (2004). An integrated model of training evaluation and effectiveness. Human Resource Development Review, 3, 385-616.
2. American Educational Research Association (AERA), American Psychological Association (APA) and National Council on Measurement in Education (NCME). (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
3. Bamberger, M., Rugh, J., Church, M., & Fort, L. (2004). Shoestring evaluation: Designing impact evaluations under budget, time and data constraints. American Journal of Evaluation, 25, 5-37.
4. Bardella, I.J., Janosky, J., Elnicki, D.M., Ploof, D., & Kolarik, R. (2005). Observed versus reported precepting skills: Teaching behaviors in a community ambulatory clerkship. Medical Education, 39, 1039-1044.
5. Boud, D., & Falchikov, N. (1989). Quantitative studies in student self-assessment in higher education: A critical analysis of findings. Higher Education, 18, 529-549.
6. Burstein, L. (1980). The analysis of multilevel data in educational research and evaluation. Review of Research in Education, 8, 158-233.
7. Campbell, D.T., & Fiske, D.W. (1959). Convergent and discriminant validation by multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
8. Cassidy, S. (2007). Assessing "inexperienced" students' ability to self-assess: Exploring links with learning style and academic personal control. Assessment & Evaluation in Higher Education, 32, 313-330.
9. Cronbach, L.J. (1976). Research on classrooms and schools: Formulation of questions, design and analysis. Occasional Paper of the Stanford Evaluation Consortium. Stanford, CA: Stanford University.
10. Davis, D.A., Mazmanian, P.E., Fordis, M., Harrison, R.V., Thorpe, K.E., & Perrier, L. (2006). Accuracy of physician self-assessment compared with observed measures of competence: A systematic review. JAMA, 296, 1094-1102.
11. D'Eon, M., Sadownik, L., Harrison, A., & Nation, J. (2008). Using self-assessments to detect workshop success: Do they work? American Journal of Evaluation, 29, 92-98.
12. Eckert, W.A. (2000). Situational enhancement of design validity: The case of training evaluation at the World Bank Institute. American Journal of Evaluation, 21, 185-193.
13. Eva, K.W., Cunnington, J.P.W., Reiter, H.I., Keane, D.R., & Norman, G.R. (2004). How can I know what I don't know? Poor self-assessment in a well-defined domain. Advances in Health Sciences Education, 9, 211-224.
14. Galtung, J. (1969). Theory and Methods of Social Research. New York: Columbia University Press.
15. Haccoun, R.R., & Hamtiaux, T. (1994). Optimizing knowledge tests for inferring learning acquisition levels in single group training evaluation designs: The internal referencing strategy. Personnel Psychology, 47, 593-604.
16. Hayman, J., Rayder, N., & Stenner, A.J. & Madey, D.L. (1979). On aggregation, generalization, and utility in educational evaluation. Educational Evaluation and Policy Analysis, 1, 31-39.
17. Hopkins, K.D. (1982). The unit of analysis: Group means versus individual observations. American Educational Research Journal, 19, 5-18.
18. Hox, J. (2002). Multilevel analysis: Techniques and applications. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
19. Keren, G. (1991). Calibration and probability judgments: Conceptual and methodological issues. Acta Psychologica, 77, 217-273.
20. Knapp, T.R. (1977). The unit-of-analysis problem in applications of simple correlation analysis to educational research. Journal of Educational Statistics, 2, 171-186.
21. Kraiger, K., Ford, K., & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78, 311-328.
22. Kreft, I., & Leeuw, J.D. (1998). Introducing multilevel modeling. Thousand Oaks, CA: Sage.
23. Lam, T.C.M. (1990, October). A two-stage approach to assessing program effects. Paper presented at the annual meeting of the American Evaluation Association, Washington, DC.
24. Lam, T.C.M. (2006). The seven deadly sins in program outcome evaluation. Paper presented at the annual meeting of the American Evaluation Association, Washington, DC.
25. Mabe, P.A., III. & West, S.G. (1982). Validity of self-evaluation of ability: A review and meta-analysis. Journal of Applied Psychology, 67, 280-296.
26. Mayne, J. (2001). Addressing attribution through contribution analysis: Using performance measures sensibly. The Canadian Journal of Program Evaluation, 16, 1-24.
27. Miller, P.J. (2003). The effect of scoring criteria specificity on peer and self-assessment. Assessment & Evaluation in Higher Education, 28, 383-394.
28. Pallier, G., Wilkinson, R., Danthiir, V., Kleitman, S., Knezevic, G., Stankov, L., et al (2002). The role of individual differences in the accuracy of confidence judgments. The Journal of General Psychology, 129, 257-299.
29. Phillips, J.J., & Stone, R.D. (2002). How to measure training results. New York: McGraw-Hill.
30. Poyner, H. (1974). Selecting units of analysis. In G. Borich (Ed.), Evaluating educational programs and products. Englewood Cliffs, NJ: Educational Technology Press.
31. Raudenbush, S.W., & Bryk, A.S. (2002). Hierarchical linear models: Applications and data analysis methods (2 nd ed.). Thousand Oaks, CA: Sage.
32. Raudenbush, S.W., Bryk, A.S., Cheong, Y.F., & Congdon, R. (2004). HLM 6: Hierarchical linear and nonlinear modeling. Lincolnwood, IL Scientific Software International. Lincolnwood, Inc.
33. Robinson, W.S. (1950). Ecological correlations and the behavior of individuals. American Sociological Review, 15, 351-357.
34. Shadish, W.R., Cook, T.D., & Campbell, D.T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin Company.
35. Sirotnik, K.A. (1980). Psychometric implications of the unit-of-analysis problem (with examples from the measurement of organizational climate). Journal of Educational Measurement, 17, 245-282.
36. Thorndike, E.L. (1939). On the fallacy of imputing the correlations found for groups to the individuals or smaller groups composing them. American Journal of Psychology, 52, 122-124.
37. Vnuk, A., Owen, H., & Plummer, J. (2006). Assessing proficiency in adult basic life support: Student and expert assessment and the impact of video recording. Medical Teacher, 28, 429-434.
38. Ward, M.L., Gruppen, L., & Regehr, G. (2002). Measuring self-assessment: Current state of the art. Advances in Health Sciences Education, 7, 63-80.
39. Wayne, D.B., Butter, J., Siddall, V.J., Fudala, M.J., Wade, L.D., Feinglass, J., et al (2006). Graduating internal medicine residents' self-assessment and performance of advanced cardiac life support skills. Medical Teacher, 28, 365-369.