Measures of program effectiveness based on retrospective pretest data: Are all created equal?

American Journal of Evaluation

Volumn 32, Issue 1, 2011, Pages 8-28

  Nimon, Kim ^a   Zigarmi, Drea ^b   Allen, Jeff ^a  

^a UNIVERSITY OF NORTH TEXAS   (United States)

^b Ken Blanchard Co   (United States)

Author keywords

experience limitation; implicit theory of change; pretest sensitization; retrospective pretest

Indexed keywords

EID: 79551622902     PISSN: 10982140     EISSN: 15570878     Source Type: Journal    
DOI: 10.1177/1098214010378354     Document Type: Article

References (43)

1. Aiken, L.S., & West, S.G. (1990). Invalidity of true experiments: Self-report pretest biases. Evaluation Review, 14, 374-390.
2. Betz, D.L., & Hill, L.G. (2006). Real world evaluation. Journal of Extension, 44. Retrieved November 20, 2009, from http://www.joe.org/joe/2006april/rb9.php.
3. Blanchard, K.H., Zigarmi, D., & Zigarmi, P. (1985). Leadership and the one minute manager. New York, NY: Morrow Publishers.
4. Campbell, D.T., & Stanley, J.C. (1963). Experimental and quasi-experimental designs for research on teaching. In N. L. Gage (Ed.), Handbook of research on teaching. Chicago, IL: RandMcNally.
5. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
6. Cronbach, L.J., & Furby, L. (1970). How should we measure "change" or should we? Psychological Bulletin, 74, 68-80.
7. Deutsch, M., & Collins, M.E. (1951). Interracial housing: A psychological evaluation of a social experiment. Minneapolis, MN: University of Minnesota Press.
8. Dunlap, W.P., Cortina, J.M., Vaslow, J.B., & Burke, M.J. (1996). Meta-analysis of experiments with matched groups or repeated measures. Psychological Methods, 1, 170-177.
9. Grice, H.P. (1975). Logic and conversation. In P. Cole & J. L. Morgan (Eds.), Syntax and semantics (pp. 41-48). New York, NY: Academic Press.
10. Haccoun, R.R., & Hamtiaux, T. (1994). Optimizing knowledge tests for inferring learning acquisition levels in single group training evaluation designs: The internal referencing strategy. Personnel Psychology, 47, 593-604.
11. Hill, L.G., & Betz, D.L. (2005). Revisiting the retrospective pretest. American Journal of Evaluation, 26, 501-507.
12. Hinkle, D.E., Wiersma, W., & Jurs, S.G. (2003). Applied statistics for the behavioral sciences (5th ed.). Boston: Houghton Mifflin.
13. Hoogstraten, J. (1985). Influence of objective measures on self-reports in a retrospective pretest-posttest design. Journal of Experimental Education, 53, 207-210.
14. Howard, G.S. (1980). Response-shift bias: A problem in evaluating programs with pre/post self-reports. Evaluation Review, 4, 93-106.
15. Howard, G.S., Ralph, K.M., Gulanick, N.A., Maxwell, S.E., Nance, D.W., & Gerber, S.K. (1979). Internal validity in pretest-posttest self-report evaluations and a re-evaluation of retrospective pretests. Applied Psychological Measurement, 3, 1-23.
16. Howard, G.S., Schmeck, R.R., & Bray, J.H. (1979). Internal validity in studies employing self-report instruments: A suggested remedy. Journal of Educational Measurement, 16, 129-135.
17. Klatt, J., & Taylor-Powell, E. (2005, October). Synthesis of literature relative to the retrospective pretest design. Paper presented at the 2005 Joint CES/AEA Conference, Toronto, Canada.
18. Lam, T.C.M., & Bengo, P. (2003). A comparison of three retrospective self-reporting methods of measuring change in instructional practice. American Journal of Evaluation, 24, 65-80.
19. Lamb, T.A., & Tschillard, R. (2005, Spring). Evaluating learning in professional development workshops: Using the retrospective pretest. Journal of Research in Professional Learning. Retrieved October 19, 20005, from http://www.nsdc.org/library/publications/research/lamb.pdf.
20. Linn, R.L., & Slinde, J.A. (1977). The determination of the significance of change between pre- and post testing periods. Review of Educational Research, 47, 121-150.
21. May, H. (2004). Making statistics more meaningful for policy research and program evaluation. American Journal of Evaluation, 25, 525-540.
22. Mezoff, B. (1981). How to get accurate self-reports of training outcomes. Training and Development Journal, 35, 56-61.
23. Nisbett, R.E., & Wilson, T.D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84, 231-259.
24. Pearson, R.W., Ross, M., & Dawes, R.M. (1992). Personal recall and the limits of retrospective questions in surveys. In J. M. Tanur (Ed.), Questions and survey questions, Meaning, memory, expression, and social interactions in survey (pp. 65-94). New York, NY: Russell Sage.
25. Pelfrey, W.V., & Sr., & Pelfrey, W. V. Jr. (2009). Curriculum evaluation and revision in a nascent field: The utility of the retrospective pretest-posttest model in a homeland security program of study. Evaluation Review, 1, 54-82.
26. Pitts, S.C., West, S.G., & Tein, J. (1996). Longitudinal measurement models in evaluation research: Examining stability and change. Evaluation and Program Planning, 19, 333-350.
27. Pratt, C.C., McGuigan, W.M., & Katzev, A.R. (2000). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21, 341-349.
28. Raidl, M., Johnson, S., Gardiner, K., Denham, M., Spain, K., & Lanting, R. (2004). Use retrospective surveys to obtain complete data sets and measure impact in extension programs. Journal of Extension, 42. Retrieved October 19, 2005, from http://www.joe.org/joe/2004april/rb2.shtml.
29. Ross, M. (1989). Relation of implicit theories to the construction of personal histories. Psychological Review, 96, 341-357.
30. Schwarz, N. (1996). Cognition and communication, Judgmental biases, research methods, and the logic of conversation. Mahwah, NJ: Lawrence Erlbaum.
31. Schwarz, N. (1999). Self-reports: How the questions shape the answers. American Psychologist, 54, 93-105.
32. Sears, R.R., Maccoby, E.E., & Levin, H. (1957). Patterns of child rearing. Evanston, IL: Row Peterson.
33. Shipper, F. (1995). A study of the psychometric properties of the managerial skill scales of the survey of management practices. Educational and Psychological Measurement, 55, 468-479.
34. Sprangers, M., & Hoogstraten, J. (1989). Pretesting effects in retrospective pretest-posttest designs. Journal of Applied Psychology, 74, 265-272.
35. Taylor, P.J., Russ-Eft, D.F., & Taylor, H. (2009). Gilding the outcome by tarnishing the past: Inflationary biases in retrospective pretest. American Journal of Evaluation, 30, 31-34.
36. Terborg, J.R., & Davis, G.A. (1982). Evaluation of a new method for assessing change to planned job redesign as applied to Hackman and Oldham's job characteristic model. Organizational Behavior and Human Performance, 29, 112-128.
37. Thorndike, R.L. (1949). Personnel selection. New York, NY: John Wiley.
38. Umble, K., Upshaw, V., Orton, S., & Matthews, K. (2000, June). Using the post-then method to assess learner change. Presentation at the AAHE Assessment Conference, Charlotte, NC, USA.
39. Walk, R.D. (1956). Self-ratings of fear in a fear-invoking situation. Journal of Abnormal and Social Psychology, 52, 171-178.
40. Willis, G. (2005). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: Sage.
41. Wilson, C.L. (1978). The Wilson multi-level management surveys: Refinement and replication of the scales. JSAS: Catalog of Selected Documents in Psychology, 8, 1707. Washington, DC: American Psychology Association.
42. Wilson, C.L. (2006). The Clark Wilson group surveys: Management practices. Silver Spring, MD: The Clark Wilson Group.
43. Zigarmi, D., Edeburn, C., & Blanchard, K. (1997). Getting to know the LBAII®, Research, validity and reliability of the self and other forms (4th ed.). Escondido, CA: Blanchard Training and Development, Inc.