Case Matching and the Reduction of Selection Bias in Quasi-Experiments: The Relative Importance of Pretest Measures of Outcome, of Unreliable Measurement, and of Mode of Data Analysis

Psychological Methods

Volumn 15, Issue 1, 2010, Pages 56-68

  Cook, Thomas D ^a   Steiner, Peter M ^a  

^a NORTHWESTERN UNIVERSITY   (United States)

Author keywords

bias reduction; multiple regression; pretests; unreliability propensity scores

Indexed keywords

ARTICLE; EPIDEMIOLOGY; HUMAN; METHODOLOGY; PSYCHOLOGICAL THEORY; PSYCHOLOGY; STATISTICS;

CAUSALITY; HUMANS; PSYCHOLOGICAL THEORY; PSYCHOLOGY;

EID: 77949623528     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0018536     Document Type: Article

References (66)

1. Arceneaux K., Gerber A.S., and Green D.P. Comparing experimental and matching methods using a large-scale voter mobilization experiment. Political Analysis 14 (2005) 37-62
2. Battistin E., and Rettore E. Ineligibles and eligible non-participants as a double comparison group in regression-discontinuity designs. Journal of Econometrics 142 (2008) 715-730
3. Blalock H.M. Causal inferences in non-experimental research (1964), University of North Carolina Press, Chapel Hill, NC
4. Bloom H.S., Michalopoulos C., and Hill C.J. Using experiments to assess nonexperimental comparison-group methods for measuring program effects. In: Bloom H.S. (Ed). Learning more from social experiments (2005), Russell Sage Foundation, New York, NY 173-235
5. Campbell D.T. Factors relevant to the validity of experiments in social settings. Psychological Bulletin 54 (1957) 297-312
6. Campbell D.T. Pattern matching as an essential in distal knowing. In: Hammond K.R. (Ed). The psychology of Egon Brunswick (1966), Holt, Rinehart and Winston, New York, NY 81-106
7. Campbell D.T. Prospective: Artifact and control. In: Rosenthal R., and Rosnow R.L. (Eds). Artifact in behavioral research (1969), Academic Press, New York, NY 351-382
8. Campbell D.T. Reforms as experiments. American Psychologist 24 (1969) 409-429
9. Campbell D.T. Evolutionary epistemology. In: Schilpp P.A. (Ed). The philosophy of Karl Popper (1974), Open Court, Montreal, Quebec, Canada 413-463
10. Campbell D.T. Focal local indicators for social program evaluation. Social Indicators Research 3 (1976) 237-256
11. Campbell D.T. Can we be scientific in applied social science?. In: Conner R., Altman D.G., and Trochim W.M.K. (Eds). Research design for program evaluation (1984), Sage, Beverly Hills, CA 15-43
12. Campbell D.T. Science's social system of validity-enhancing collective belief change and the problems of the social sciences. In: Fiske D.W., and Shweder R.A. (Eds). Metatheory in social science: Pluralisms and subjectivities (1986), University of Chicago Press, Chicago, IL 108-135
13. Campbell D.T. Methodology and epistemology for social science: Selected papers (1988), University of Chicago Press, Chicago, IL
14. Campbell D.T., and Boruch R.F. Making the case for randomized assignment to treatments by considering the alternatives: Six ways in which quasi-experimental evaluations in compensatory education tend to underestimate effects. In: Bennett C.A., and Lumsdaine A.A. (Eds). Evaluation and experiments: Some critical issues in assessing social programs (1975), Academic Press, New York, NY 195-296
15. Campbell D.T., and Erlebacher A.E. How regression artifacts can mistakenly make compensatory education programs look harmful. In: Hellmuth J. (Ed). The disadvantaged child: Vol. 3, Compensatory education: A national debate (1970), Brunner/Mazel, New York, NY 185-210
16. Campbell D.T., and Fiske D.W. Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin 56 (1959) 81-105
17. Campbell D.T., and Kenny D.A. A primer on regression artifacts (1999), Guilford Press, New York, NY
18. Campbell D.T., and Stanley J.C. Experimental and quasi-experimental designs for research (1963), Rand McNally, Chicago, IL
19. Cartwright N. Hunting causes and using them (2007), Cambridge University Press, Cambridge, England
20. Cochran W.G., and Rubin D.B. Controlling bias in observational studies: A review. The Indian Journal of Statistics 35 A (1973) 417-466
21. Collingwood R. An essay on metaphysics (1940), Clarendon Press, Oxford, England
22. Cook T.D., and Campbell D.T. Quasi-experimentation: Design and analysis issues for field settings (1979), Rand McNally, Chicago, IL
23. Cook T.D., Shadish W.R., and Wong V.C. Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management 27 (2008) 724-750
24. Corrin W.J., and Cook T.D. Design elements of quasi-experimentation. Advances in Educational Productivity 7 (1998) 35-57
25. Cronbach L.J. Designing evaluations of educational and social programs (1982), Jossey-Bass, San Francisco, CA
26. Diaz J.J., and Handa S. An assessment of propensity score matching as a nonexperimental impact estimator. The Journal of Human Resources 41 (2006) 319-345
27. Fisher R.A. Statistical methods for research workers (1925), Oliver and Boyd, Edinburgh, Scotland
28. Fraker T., and Maynard R. The adequacy of comparison group designs for evaluations of employment-related programs (1986) (Available from Mathematica Policy Research, P. O. Box 2393, Princeton, NJ 08543-2393)
29. Glazerman S., Levy D.M., and Myers D. Nonexperimental versus experimental estimates of earnings impacts. Annals of the American Academy 589 (2003) 63-93
30. Goldberger A.S. Selection bias in evaluating treatment effects: Some formal illustrations (1972), Institute for Research on Poverty, Madison, WI
31. Granger C. Investigating causal relations by econometric models and cross-spectral methods. Econometrica 37 (1969) 424-438
32. Hahn J., Todd P., and van der Klaauw W. Identification and estimation of treatment effects with a regression-discontinuity design. Econometrica 69 (2001) 201-209
33. Heckman J., Ichimura H., and Todd P.E. Matching as an econometric evaluation estimator: Evidence from evaluating a job training program. Review of Economic Studies 64 (1997) 605-654
34. Heckman J., Ichimura H., and Todd P.E. Matching as an econometric evaluation estimator. Review of Economic Studies 65 (1998) 261-294
35. Hill J.L., Reiter J.P., and Zanutto E.L. A comparison of experimental and observational data analyses. In: Gelman A., and Meng X.L. (Eds). Applied Bayesian and causal inference from an incomplete data perspective (2004), Wiley, New York, NY 49-60
36. Imbens G.W. Nonparametric estimation of average treatment effects under exogeneity: A review. The Review of Economics and Statistics 86 (2004) 4-29
37. Imbens G., and Lemieux T. Regression discontinuity designs: A guide to practice. Journal of Econometrics 142 (2008) 615-635
38. LaLonde R. Evaluating the econometric evaluations of training with experimental data. American Economic Review 76 (1986) 604-620
39. Lipsey M.W., and Wilson D.B. The efficacy of psychological, educational, and behavioral treatment. American Psychologist 48 (1993) 1181-1209
40. Lord F.M. Large-scale covariance analysis when the control variable is fallible. Journal of the American Statistical Association 55 (1960) 307-331
41. Magidson J. Toward a causal model approach for adjusting for preexisting differences in the nonequivalent control group situation. Evaluation Quarterly 1 (1977) 399-402
42. Pearl J. Causality: Models, reasoning, and inference. 2nd ed. (2009), Cambridge University Press, Cambridge, England
43. Pohl S., Steiner P.M., Eisermann J., Soellner R., and Cook T.D. Unbiased causal inference from an observational study: Results of a within-study comparison. Educational Evaluation and Policy Analysis 31 (2009) 463-479
44. Popper K.R. The logic of scientific discovery (1959), Basic Books, New York, NY
45. Rosenbaum P.R. Observational studies. 2nd ed. (2002), Springer, New York, NY
46. Rosenbaum P.R., and Rubin D.B. Assessing sensitivity to an unobserved binary covariate in an observational study with binary outcome. Journal of the Royal Statistical Society 45 B (1983) 212-218
47. Rosenbaum P.R., and Rubin D.B. The central role of the propensity score in observational studies for causal effects. Biometrika 70 (1983) 41-55
48. Rosenbaum P.R., and Rubin D.B. Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association 79 (1984) 516-524
49. Rubin D.B. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology 66 (1974) 688-701
50. Rubin D.B. Assignment to treatment group on the basis of a covariate. Journal of Educational Statistics 2 (1977) 1-26
51. Rubin D.B. William G. Cochran's contribution to the design, analysis, and evaluation of observational studies. In: Rao P.S.R.S., and Sedransk J. (Eds). W. G. Cochran's impact on statistics (1984), Wiley, New York, NY 37-69
52. Rubin D.B. Practical implications of modes of statistical inference for causal effects and the critical role of the assignment mechanism. Biometrics 4 (1991) 1213-1234
53. Rubin D.B. Using propensity scores to help design observational studies: Application to the tobacco litigation. Health Services and Outcomes Research Methodology 2 (2001) 169-188
54. Rubin D.B. Matched sampling for causal effects (2006), Cambridge University Press, Cambridge, England
55. Rubin D.B. The design versus the analysis of observational studies for causal effects: Parallels with the design of randomized trials. Statistics in Medicine 26 (2007) 20-36
56. Rubin D.B. For objective causal inference, design trumps analysis. The Annals of Applied Statistics 2 (2008) 808-840
57. Rubin D.B., and Thomas N. Matching using estimated propensity scores: Relating theory to practice. Biometrics 52 (1996) 249-264
58. Schafer J.L., and Kang J. Average causal effects from non-randomized studies: A practical guide and simulated example. Psychological Methods 13 (2008) 279-313
59. Shadish W.R. Campbell and Rubin: A primer and comparison of their approaches to causal inference in field settings. Psychological Methods 15 (2010) 3-17
60. Shadish W.R., Clark M.H., and Steiner P.M. Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random to nonrandom assignment. Journal of the American Statistical Association 103 (2008) 1334-1344
61. Shadish W.R., Cook T.D., and Campbell D.T. Experimental and quasi-experimental designs for generalized causal inference (2002), Houghton-Mifflin, Boston, MA
62. Steiner, P. M., Cook, T. D., & Shadish, W. R. (in press). On the importance of reliable covariate measurement in selection bias adjustments using propensity scores. Journal of Educational and Behavioral Statistics
63. Steiner, P. M., Cook, T. D., Shadish, W. R., & Clark, M. H. (2009). The importance of covariate selection in controlling for selection bias in observational studies. Manuscript submitted for publication.
64. Stuart E.A., and Rubin D.B. Best practices in quasi-experimental designs: Matching methods for causal inference. In: Osborne J. (Ed). Best practices in quantitative methods (2007), Sage, Thousand Oaks, CA 155-176
65. Thistlethwaite D.L., and Campbell D.T. Regression-discontinuity analysis: An alternative to the ex-post facto experiment. Journal of Educational Psychology 51 (1960) 309-317
66. West S.G., and Thoemmes F. Campbell's and Rubin's perspectives on causal inference. Psychological Methods 15 (2010) 18-37