Evaluating methods for estimating program effects

American Journal of Evaluation

Volumn 32, Issue 2, 2011, Pages 246-272

  Reichardt, Charles S ^a  

^a UNIVERSITY OF DENVER   (United States)

Author keywords

bias; program effects; quasi experiments; randomized experiments

Indexed keywords

EID: 79955420801     PISSN: 10982140     EISSN: 15570878     Source Type: Journal    
DOI: 10.1177/1098214011398954     Document Type: Article

References (78)

1. Astbury, B., & Leeuw, F.L. (2010). Unpacking black boxes: Mechanisms and theory building in evaluation. American Journal of Evaluation, 31, 363-381.
2. Bamberger, M., Rugh, J., Church, M., & Fort, L. (2004). Shoestring evaluation: Designing impact evaluations under budget, time and data constraints. American Journal of Evaluation, 25, 5-37.
3. Barnette, J.J., & Wallis, A.B. (2005). The missing treatment design element: Continuity of treatment when multiple postobservations are used in time-series and repeated measures study designs. American Journal of Evaluation, 26, 106-123.
4. Box, G.E.P., & Tiao, G.C. (1975). Intervention analysis with applications to economic and environmental problems. Journal of the American Statistical Association, 70, 70-92.
5. Brand, M. (Ed.). (1976). The nature of causation (pp. 252-281). Urbana, IL: University of Illinois Press.
6. Brand, M. (1979). Causality. In P. D. Asquith & H. E. Kyburg, Jr. (Eds.), Current research in philosophy of science. East Lansing, MI: Philosophy of Science Association.
7. Brandon, P.R., Taum, A.K.H., Young, D.B., Pottenger, F.M., III, & Speitel, T.W. (2008). The complexity of measuring the quality of program implementation with observations: The case of middle school inquiry-based science. American Journal of Evaluation, 29, 235-250.
8. Braucht, G.N., & Reichardt, C.S. (1993). A computerized approach to trickle-process, random assignment. Evaluation Review, 17, 79-90.
9. Bryk, A.S., & Raudenbush, S.W. (1992). Hierarchical linear models: Applications and data analysis methods. Newbury Park, CA: Sage.
10. Campbell, D.T., & Stanley, J.C. (1966). Experimental and quasi-experimental designs for research. Chicago, IL: Rand McNally.
11. Cappelleri, J.C., Darlington, R.B., & Trochim, W.M.K. (1994). Power analysis of cutoff-based randomized clinical trials. Evaluation Review, 18, 141-152.
12. Century, J., Rudnick, M., & Freeman, C. (2010). A framework for measuring fidelity of implementation: A foundation for shared language and accumulation of knowledge. American Journal of Evaluation, 31, 199-218.
13. Cook, T.D. (2002). Randomized experiments in educational policy research: A critical examination of the reasons the educational evaluation community has offered for not doing them. Educational Evaluation and Policy Analysis, 24, 175-199.
14. Cook, T.D., & Campbell, D.T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago, IL: Rand McNally.
15. Cook, T. D., & Reichardt, C. S. (Eds.). (1979). Qualitative and quantitative methods in evaluation research. Newbury Park, CA: Sage.
16. Cook, T.D., Scriven, M., Coryn, C.L.S., & Evergreen, S. D. H. (2010). Contemporary thinking about causation in evaluation: A dialogue with Tom Cook and Michael Scriven. American Journal of Evaluation, 31, 105-117.
17. Coryn, C.L.S., Schroter, D.C., & Hanssen, C.E. (2009). Adding a time-series design element to the success case method to improve methodological rigor: An application for nonprofit program evaluation. American Journal of Evaluation, 30, 80-92.
18. Dawid, A.P. (2000). Causal inference without counterfactuals. Journal of the American Statistical Association, 95, 407-424.
19. Eckert, W.A. (2000). Situational enhancement of design validity: The case of training evaluation at the World Bank Institute. American Journal of Evaluation, 21, 185-193.
20. Fetterman, D.M. (1982). Ibsen's baths: Reactivity and insensitivity. Educational Evaluation and Policy Analysis, 4, 261-279.
21. Fetterman, D.M. (1992). In response to Lee Sechrest's 1991 AEA presidential address: "Roots: Back to our first generations," February 1991, 1-7. American Journal of Evaluation, 13, 171-172.
22. Gargani, J. (2010). A welcome change from debate to dialogue about causality. American Journal of Evaluation, 31, 131-132.
23. Gilovich, T. (1991). How we know what isn't so: The fallibility of human reason in everyday life. New York, NY: Free Press.
24. Goldberger, A.S. (1972). Selection bias in evaluating treatment effects: Some formal illustrations. Discussion Paper 123-72. Madison, WI: Institute for Research on Poverty, University of Wisconsin.
25. Harris, P.M. (1998). Attrition revisited. American Journal of Evaluation, 19, 293-305.
26. Holland, P.W. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81, 945-970.
27. Imbens, G.W., & Lemieux, T. (2008). Regression discontinuity designs: A guide to practice. Journal of Econometrics, 142, 615-635.
28. Jagannathan, R., Camasso, M.J., & Killingsworth, M.R. (2004). Do family caps on welfare affect births among welfare recipients? Reconciling efficacy and effectiveness estimates of impact through a blended design strategy. American Journal of Evaluation, 25, 295-319.
29. Kraemer, H.C., & Thiemann, S. (1987). How many subjects? Statistical power analysis in research. Newbury Park, CA: Sage.
30. Lam, J.A., Hartwell, S.W., & Jekel, J.F. (1994). "I prayed real hard, so I know I'll get in": Living with randomization. In K. J. Conrad (Ed.), Critically evaluating the role of experiments. New directions for program evaluation, No. 63(pp. 55-66). San Francisco, CA: Jossey-Bass.
31. Langer, E.J., & Rodin, J. (1976). The effects of choice and enhanced personal responsibility for the aged: A field experiment in an institutional setting. Journal of Personality and Social Psychology, 34, 191-198.
32. Leow, C., Marcus, S., Zanutto, E., & Boruch, R. (2004). Effects of advanced course-taking on math and science achievement: Addressing selection bias using propensity scores. American Journal of Evaluation, 25, 461-478.
33. Lincoln, Y.S., & Guba, E.G. (1992). In response to Lee Sechrest's 1991 AEA presidential address: "Roots: Back to our first generations," February 1991, 1-7. American Journal of Evaluation, 13, 165-169.
34. Lincoln, Y.S., & Guba, E.G. (1994). RSVP: We are pleased to accept your invitation. American Journal of Evaluation, 15, 179-192.
35. Lipsey, M.W. (1990). Design sensitivity: Statistical power for experimental research. Newbury Park, CA: Sage.
36. Little, R.J., & Yau, L. (1998). Statistical techniques for analyzing data from prevention tirals: Treatment of no-shows using Rubin's causal model. Psychological Methods, 3, 247-259.
37. Marcantonio, R.J., & Cook, T.D. (1994). Convincing quasi-experiments: The interrupted time series and regression-discontinuity designs. In J. S. Wholey, H. P. Hatry, & K. E. Newcomer (Eds.), Handbook of practical program evaluation (pp. 133-154). San Francisco, CA: Jossey-Bass.
38. Mark, M.M., & Reichardt, C.S. (2004). Quasi-experimental and correlational designs: Methods for the real world when random assignment isn't feasible. In C. Sansone, C. C. Morf, & A. T. Panter (Eds.), Handbook of methods in social psychology (pp. 265-286). Thousand Oaks, CA: Sage.
39. Mastoroudes, C.C. (1993). Selecting projects for ex post evaluation. American Journal of Evaluation, 14, 253-258.
40. McConney, A., Rudd, A., & Ayres, R. (2002). Getting to the bottom line: A method for synthesizing findings within mixed-method program evaluations. American Journal of Evaluation, 23, 121-140.
41. Meier, S.T. (2004). Improving design sensitivity through intervention-sensitive measures. American Journal of Evaluation, 25, 321-334.
42. Mohr, L.B. (1999). The qualitative method of impact analysis. American Journal of Evaluation, 20, 69-84.
43. Mowbray, C.T., Bybee, D., Holter, M., & Lewandowski, L. (2006). Validation of a fidelity rating instrument for consumer-operated services. American Journal of Evaluation, 27, 9-27.
44. Newton, X.A., & Llosa, L. (2010). Toward a more nuanced approach to program effectiveness assessment: Hierarchical linear models in K-12 program evaluation. American Journal of Evaluation, 31, 162-179.
45. Ong-Dean, C., Hofstetter, C.H., & Strick, B.R. (2011). Challenges and dilemmas in implementing random assignment in educational research. American Journal of Evaluation, 32, 29-49.
46. Patton, M.Q. (2008). Advocacy impact evaluation. Journal of Multidisciplinary Evaluation, 5, 1-10.
47. Peck, L.R. (2007). What are the effects of welfare sanction policies? Or, using propensity scores as a subgroup indicator to learn more from social experiments. American Journal of Evaluation, 28, 256-274.
48. Raudenbush, S.W., & Bryk, A.S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.
49. Reichardt, C.S. (1979). The statistical analysis of data from nonequivalent group designs. In T. D. Cook & D. T. Campbell (Eds.), Quasi-experimentation: Design and analysis issues for field settings (pp. 147-205). Chicago, IL: Rand McNally.
50. Reichardt, C.S. (2000). A typology of strategies for ruling out threats to validity. In L. Bickman (Ed.), Research design: Donald Campbell's legacy (Vol. 2, pp. 89-115). Thousand Oaks, CA: Sage.
51. Reichardt, C.S. (2006). The principle of parallelism in the design of studies to estimate treatment effects. Psychological Methods, 11, 1-18.
52. Reichardt, C.S. (2009). Quasi-experimental design. In R. E. Millsap & A. Maydeu-Olivares (Eds.), The SAGE handbook of quantitative methods in psychology (pp. 46-71). Thousand Oaks, CA: Sage.
53. Reichardt, C.S., & Cook, T.D. (1979). Beyond qualitative versus quantitative methods. In T. D. Cook & C. S. Reichardt (Eds.), Qualitative and quantitative methods in evaluation research (pp. 7-32). Newbury Park, CA: Sage.
54. Reichardt, C.S., & Gollob, H.F. (1989). Ruling out threats to validity. Evaluation Review, 13, 3-17.
55. Reichardt, C.S., & Henry, G.T. (2012). Regression-discontinuity designs. In H. Cooper (Ed.), The handbook of research methods in psychology. Washington, DC: American Psychological Association.
56. Reichardt, C.S., & Mark, M.M. (1998). Quasi-experimentation. In L. Bickman & D. J. Rog (Eds.), Handbook of applied social research methods (pp. 193-228). Thousand Oaks, CA: Sage.
57. Reynolds, A.J. (1998). Confirmatory program evaluation: A method for strengthening causal inference. American Journal of Evaluation, 19, 203-221.
58. Rosenbaum, P.R., & Rubin, D.B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41-55.
59. Rubin, D.B. (2005). Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100, 322-331.
60. Sadler, D.R. (1981). Intuitive data processing as a potential source of bias in naturalistic evaluations. Educational Evaluation and Policy Analysis, 3, 25-31.
61. Schafer, J.L., & Graham, J.W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7, 147-177.
62. Scriven, M. (1968). In defense of all causes. Issues in Criminology, 4, 79-81.
63. Scriven, M. (1975). Causation as explanation. Nous, 9, 3-16.
64. Scriven, M. (1976). Maximizing the power of causal investigations: The modus operandi method. In G. V. Glass (Ed.), Evaluation studies review annual (Vol. 1, pp. 101-118). Beverly Hills, CA: Sage.
65. Scriven, M. (2003). Scriven on randomized control groups. Retrieved fromhttp://interversity.org/lists/arn-l/archives/Nov2003/msg00309.html.
66. Scriven, M. (2005, December 8). Can we infer causation from cross-sectional data? Paper presented at a Symposium on the Use of School-Level Data for Evaluating Federal Education Programs, Washington, DC: The Board on Testing and Assessment, The National Academies.
67. Scriven, M. (2008). A summative evaluation of RCT methodology: And an alternative approach to causal research. Journal of Multidisciplinary Evaluation, 5, 11-24.
68. Scriven, M. (2009). Demythologizing causation and evidence. In S. I. Donaldson, C. A. Christie, & M. M. Mark (Eds.), What counts as credible evidence in applied research and evaluation practice? (pp. 134-152). Thousand Oaks, CA: Sage.
69. Sechrest, L. (1992). Roots: Back to our first generations. American Journal of Evaluation, 13, 1-7.
70. Sechrest, L., Babcock, J., & Smith, B. (1993). An invitation to methodological pluralism. American Journal of Evaluation, 14, 227-235.
71. Shadish, W.R., & Cook, T.D. (1999). Comment-design rules: More steps toward a complete theory of quasi-experimentation. Statistical Science, 14, 294-300.
72. Shadish, W.R., Cook, T.D., & Campbell, D.T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
73. Smith, N.L., Brandon, P.R., Lawton, B.E., & Krohn-Ching, V. (2010). Evaluation exemplar: Exemplary aspects of a small group-randomized local educational program evaluation. American Journal of Evaluation, 31, 254-265.
74. Trochim, W.M.K. (1984). Research designs for program evaluation: The regression-discontinuity approach. Newbury Park, CA: Sage.
75. Veney, J.E. (1993). Evaluation applications of regression analysis with time-series data. American Journal of Evaluation, 14, 259-274.
76. West, S.G., Biesanz, J.C., & Pitts, S.C. (2000). Causal inference and generalization in field settings: Experimental and quasi-experimental designs. In H. T. Reis & C. M. Judd (Eds.), Handbook of research methods in social psychology (pp. 40-84). New York, NY: Cambridge University Press.
77. West, S.G., & Thoemmes, F. (2010). Campbell's and Rubin's perspectives on causal inference. Psychological Methods, 15, 18-37.
78. Zvoch, K. (2009). Treatment fidelity in multisite evaluation: A multilevel longitudinal examination of provider adherence status and change. American Journal of Evaluation, 30, 44-61.