A comparison of three retrospective self-reporting methods of measuring change in instructional practice

American Journal of Evaluation

Volumn 24, Issue 1, 2003, Pages 65-80

  Lam, Tony C M ^a   Bengo, Priscilla ^a  

^a UNIVERSITY OF TORONTO   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0037338734     PISSN: 10982140     EISSN: None     Source Type: Journal    
DOI: 10.1016/S1098-2140(02)00273-4     Document Type: Article

References (47)

1. Benjamin, E. R. (1982). Using the 'post-then' method of evaluation. Training/HRD, 72.
2. Blair, E. A., & Ganesh, G. K. (1991). Characteristics of interval-based estimates of autobiographical frequencies. Applied Cognitive Psychology, 5, 237-250.
3. Bracht, G. H., & Glass, G. V. (1968). The external validity of experiments. American Educational Research Journal, 5, 437-474.
4. Bray, J. H., & Howard, G. S. (1980). Methodological considerations in the evaluation of a teacher-training program. Journal of Educational Psychology, 72, 62-70.
5. Bray, J. H., Maxwell, S. E., & Howard, G. S. (1984). Methods of analysis with response-shift bias. Educational & Psychological Measurement, 44, 781-804.
6. Burstein, L., McDonnell, L. M., Van Winkle, J., Ormseth, T., Mirocha, J., & Guitton, G. (1995). Validating national curriculum indicators. Sanata Monica, CA: Rand McNally College Publishing Company.
7. Conway, M., & Ross, M. (1984). Getting what you want by revising what you had. Journal of Personality and Social Psychology, 47, 738-748.
8. Cook, T. D., & Campbell, D. T. (1979). Quasi-experimental design and analysis issues for field settings. Chicago: Rand McNally College Publishing Company.
9. Cowan, G., & Komorita, S. S. (1971). The effects of forewarning and pretesting on attitude change. Educational & Psychological Measurement, 31, 431-439.
10. Eckert, W. L. (2000). Situational enhancement of design validity: The case of training evaluation at the World Bank institute. American Journal of Evaluation, 21, 185-193.
11. Firestone, W. A., Mayrowetz, D., & Fairman, J. (1998). Performance-based assessment and instructional change: The effects of testing in Maine and Maryland. Educational Evaluation and Policy Analysis, 20(2), 95-113.
12. Goedhart, H., & Hoogstraten, J. (1992). The retrospective pretest and the role of pretest information in evaluative studies. Psychological Reports, 70, 699-704.
13. Green, M. C., Krosnick, J. A., & Holbrook, A. L. (2001). The survey response process in telephone and face-to-face surveys: Differences in respondent satisficing and social desirability response bias. Paper presented at the 2000 meeting of the American Association for Public Opinion Research in Montreal, Canada.
14. Gutek, B. A., & Winter, S. J. (1992). Consistency of job satisfaction across situations: Fact or framing artifact? Journal of Vocational Behavior, 41, 61-78.
15. Hawkins, S. A., & Hastie, R. (1990). Hindsight: Biased judgments of past events after the outcomes are known. Psychological Bulletin, 107, 311-327.
16. Herold, E. S., & Thomas, R. E. (1981). Perceived versus computed change: Can perceived measures tell us something that computed measures cannot? Educational & Psychological Measurement, 41, 701-707.
17. Hoogstraten, J. (1979). Pretesting as determinant of attitude change in evaluation research. Applied Psychological Measurement, 3, 25-30.
18. Hoogstraten, J. (1980). The reactive effect of pretesting in attitude change research: General or specific? Applied Psychological Measurement, 4, 39-42.
19. Hoogstraten, J. (1982). The retrospective pretest in an educational training context. Journal of Experimental Education, 50, 200-204.
20. Hoogstraten, J. (1985). Influence of objective measures on self-reports in a retrospective pretest-posttest design. Journal of Experimental Education, 53, 207-210.
21. Howard, G. S. (1980). Response-shift bias: A problem in evaluating interventions with pre/post self-reports. Evaluation Review, 4, 93-106.
22. Howard, G. S., & Dailey, P. R. (1979). Response-shift bias: A source of contamination of self-report measures. Journal of Applied Psychology, 64, 144-150.
23. Howard, G. S., Dailey, P. R., & Gulanick, N. A. (1979c). The feasibility of informed pretests in attenuating response-shift bias. Applied Psychological Measurement, 3, 481-494.
24. Howard, G. S., Millham, J., Slaten, S., & O'Donnell, L. (1981). Influence of subject response style effects on retrospective measures. Applied Psychological Measurement, 5, 89-100.
25. Howard, G. S., Ralph, K. M., Gulanick, N. A., Maxwell, S. E., Nance, D. W., & Gerber, S. K. (1979b). Internal invalidity in pretest-posttest self-report evaluations and a re-evaluation of retrospective pretests. Applied Psychological Measurement, 3, 1-23.
26. Howard, G. S., Schmeck, R. R., & Bray, J. H. (1979a). Internal invalidity in studies employing self-report instruments: A suggested remedy. Journal of Educational Measurement, 16, 129-135.
27. Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5, 213-236.
28. Krosnick, J. A., Narayan, S., & Smith, W. R. (1996). Satisficing in surveys: Initial evidence. In M. T. Braveman, J. K. Slater (Eds.), Advances in survey research. New directions for evaluation (no. 70, pp. 29-44). San Francisco: Jossey Bass.
29. Linn, R. L. (1981). Measuring pretest-posttest performance change. In R. A. Berk (Ed.), Educational evaluation methodology: The state of the art (pp. 84-109). Baltimore, Maryland: The Johns Hopkins University Press.
30. Lyons, T. F., & Dickinson, T. L. (1973). A comparison of perceived and computed-change measures over a three-year period. Journal of Applied Psychology, 58, 318-321.
31. Maltz, M. D., Gordon, A. C., McDowall, D., & McCleary, R. (1980). An artifact in pretest-posttest designs: How it can mistakenly make delinquency programs look effective. Evaluation Review, 4, 225-240.
32. Mayer, D. P. (1999). Measuring instructional practice: Can policymakers trust survey data. Educational Evaluation and Policy Analysis, 21, 29-35.
33. Paulhus, D. L. (1984). Two-component models of socially desirable responding. Journal of Personality and Social Psychology, 46, 598-609.
34. Pearson, R.W., Ross, M., & Dawes, R. M. (1992). Personal recall and the limits of retrospective questions in surveys. In J. M. Tanur (Ed.), Questions about questions: Inquiries into the cognitive bases of surveys. New York: Russell Sage.
35. Pohl, N. F. (1982). Using retrospective pre-ratings to counteract response-shift confounding. Journal of Experimental Education, 50, 211-214.
36. Pohlmann, J. T., & Beggs, D. L. (1974). A study of the validity of self-reported measures of academic growth. Journal of Educational Measurement, 11, 115-119.
37. Pomplun, M. (1997). State assessment and instructional change: A path model analysis. Applied Measurement in Education, 10, 217-234.
38. Pratt, C. C., McGuigan, W. M., & Katzev, A. R. (2000). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21, 341-349.
39. Rosnow, R. L., Holper, H. M., & Gitter, A. G. (1973). More on the reactive effects of pretesting in attitude research: Demand characteristics or subject commitment? Educational & Psychological Measurement, 33, 7-17.
40. Schwarz, N., & Oyserman, D. (2001). Asking questions about behavior: Cognition, communication, and questionnaire construction. American Journal of Evaluation, 22(2), 127-160.
41. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized casual inference. Boston: Houghton-Mifflin.
42. Sprangers, M. (1987). Validity threats in retrospective pretest-posttest designs. Paper presented at the annual meeting of the American Educational Research Association, Washington, DC.
43. Sprangers, M., & Hoogstraten, J. (1989). Pretesting effects in retrospective pretest-posttest designs. Journal of Applied Psychology, 74, 265-272.
44. Stecher, B. M., Barron, S., Kaganoff, T., & Goodwin, J. (1998). The effects of standards-based assessment on classroom practices: Results of the 1996-1997 RAND survey of Kentucky teachers of mathematics and writing (CSE Technical Report 482). Los Angeles, CA: National Center for Research on Evaluation, Standards and Student Testing.
45. Taylor, C. S., Lairon, M. (1993). The influence of high stakes performance-based assessments on classroom teachers' attitudes and behaviors. Paper presented at the 1993 annual meeting of the National Council on Measurement in Education, Atlanta, Georgia.
46. Welch, W. W., & Walberg, H. J. (1970). Pretest and sensitization effects in curriculum evaluation. American Educational Research Journal, 7, 605-614.
47. Willson, V. L., & Putnam, R. R. (1982). A meta-analysis of pretest sensitization effects in experimental design. American Educational Research Journal, 19(2), 249-258.