Campbell and Rubin: A Primer and Comparison of Their Approaches to Causal Inference in Field Settings

Psychological Methods

Volumn 15, Issue 1, 2010, Pages 3-17

  Shadish, William R ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

causal inference; propensity score; Rubin's causal model; SUTVA; validity

Indexed keywords

ARTICLE; COMPARATIVE STUDY; EPIDEMIOLOGY; HUMAN; METHODOLOGY; PSYCHOLOGICAL THEORY; PSYCHOLOGY;

CAUSALITY; HUMANS; PSYCHOLOGICAL THEORY; PSYCHOLOGY;

EID: 77949611999     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0015916     Document Type: Article

References (137)

1. Aiken L.S., West S.G., and Millsap R.E. Doctoral training in statistics, measurement, and methodology: Replication and extension of Aiken, West, Sechrest, and Reno's (1990) survey of PhD programs in North America. American Psychologist 63 (2008) 32-50
2. Angrist J.D., Imbens G.W., and Rubin D.B. Identification of causal effects using instrumental variables. Journal of the American Statistical Association 91 (1996) 444-455
3. Angrist J.D., and Lavy V. Using Maimonides' rule to identify the effects of class size on scholastic achievement. Quarterly Journal of Economics 114 (1999) 533-575
4. Berk R.A. Regression analysis: A constructive critique (2004), Sage, Thousand Oaks, CA
5. Bickman L. Validity and social experimentation: Donald Campbell's legacy (Vol. 1 & 2) (2000), Sage, Thousand Oaks, CA
6. Blitstein J.L., Murray D.M., Hannan P.J., and Shadish W.R. Increasing the degrees of freedom in existing group randomized trials: The df* approach. Evaluation Review 29 (2005) 241-267
7. Boruch R.F., and Mosteller F. Evidence matters: Randomized trials in education research (2002), Brookings Institution Press, Washington, DC
8. Boruch R.F., and Wolins L. A procedure for estimation of trait, method, and error variance attributable to a measure. Educational and Psychological Measurement 30 (1970) 547-574
9. Braver M.C.W., and Braver S.L. Statistical treatment of the Solomon four-group design: A meta-analytic approach. Psychological Bulletin 104 (1988) 150-154
10. Brewer M.B., and Collins B.E. Scientific inquiry and the social sciences: A volume in honor of Donald T. Campbell (1981), Jossey-Bass, San Francisco
11. Campbell D.T. Factors relevant to the validity of experiments in social settings. Psychological Bulletin 54 (1957) 297-312
12. Campbell D.T. Evolutionary epistemology. In: Schilpp P.A. (Ed). The philosophy of Karl Popper (1974), Open Court, La Salle, IL
13. Campbell D.T. "Degrees of freedom" and the case study. Comparative Political Studies 8 (1975) 178-193
14. Campbell D.T. Toward an epistemologically relevant sociology of science. Science, Technology, and Human Values 10 (1984) 38-48
15. Campbell D.T. Relabeling internal and external validity for applied social scientists. In: Trochim W.M.K. (Ed). Advances in quasi-experimental design and analysis (1986), Jossey-Bass, San Francisco 67-77
16. Campbell D.T. Methodology and epistemology for social science: Selected papers (1988), University of Illinois Press, Chicago
17. Campbell D.T. Retrospective and prospective on program impact assessment. American Journal of Evaluation 15 (1994) 291-298
18. Campbell D.T., and Erlebacher A.E. How regression artifacts can mistakenly make compensatory education programs look harmful. In: Hellmuth J. (Ed). The disadvantaged child: Vol. 3. Compensatory education: A national debate (1970), Bruner/Mazel, New York 185-210
19. Campbell D.T., and Fiske D.W. Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin 56 (1959) 81-105
20. Campbell D.T., and Stanley J.C. Experimental and quasi-experimental designs for research on teaching. In: Gage N.L. (Ed). Handbook of research on teaching (1963), Rand McNally, Chicago 171-246
21. Campbell D.T., and Stanley J.C. Experimental and quasi-experimental designs for research (1966), Rand McNally, Chicago
22. Chapin F.S. The advantages of experimental sociology in the study of family group patterns. Social Forces 11 (1932) 200-207
23. Chapin F.S. Experimental designs in sociological research (1947), Harper, New York
24. Cicirelli V.G., and Associates. The impact of Head Start: An evaluation of the effects of Head Start on children's cognitive and affective development (Vol. 1 & 2). A report to the Office of Economic Opportunity (1969), Ohio University and Westinghouse Learning Corporation, Athens, OH
25. Cochran W.G. The planning of observational studies in human populations. Journal of the Royal Statistical Society: Series A 128 (1965) 134-155
26. Collins J., Hall E., and Paul L. Causation and counterfactuals (2004), MIT Press, Cambridge, MA
27. Cook T.D. Postpositivist critical multiplism. In: Shotland L., and Mark M.M. (Eds). Social science and social policy (1985), Sage, Newbury Park, CA 21-62
28. Cook T.D. The generalization of causal connections: Multiple theories in search of clear practice. In: Sechrest L., Perrin E., and Bunker J. (Eds). Research methodology: Strengthening causal interpretations of nonexperimental data (1990), Department of Health and Human Services, Rockville MD 9-31 DHHS Publication No. [PHS] 90-3454
29. Cook T.D. Clarifying the warrant for generalized causal inferences in quasi-experimentation. In: McLaughlin M.W., and Phillips D.C. (Eds). Evaluation and education: At quarter-century (1991), National Society for the Study of Education, Chicago 115-144
30. Cook T.D. Causal generalization: How Campbell and Cronbach influenced my theoretical thinking on this topic, including in Shadish, Cook, and Campbell. In: Alkin M. (Ed). Evaluation roots: Tracing theorists' views and influences (2004), Sage, Thousand Oaks, CA 88-113
31. Cook T.D. "Waiting for life to arrive": A history of the regression-discontinuity design in psychology, statistics and economics (2007), Institute for Policy Research, Northwestern University, Evanston, IL. http://www.northwestern.edu/ipr/publications/papers/2007/wp0703.pdf Retrieved April 17, 2009, from http://www.northwestern.edu/ipr/publications/papers/2007/wp0703.pdf
32. Cook T.D., and Campbell D.T. Quasi-experimentation: Design and analysis issues for field settings (1979), Rand-McNally, Chicago
33. Cook T.D., and Campbell D.T. The causal assumptions of quasi-experimental practice. Synthese 68 (1986) 141-180
34. Cook T.D., and Reichardt C.S. Qualitative and quantitative methods in evaluation research (1979), Sage, Newbury Park, CA
35. Cook T.D., Shadish W.R., and Wong V.C. Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management 27 (2008) 724-750
36. Cordray D.W. Quasi-experimental analysis: A mixture of methods and judgment. In: Trochim W.M.K. (Ed). Advances in quasi-experimental design and analysis (1986), Jossey-Bass, San Francisco 9-27
37. Cronbach L.J. Designing evaluations of educational and social programs (1982), Jossey-Bass, San Francisco
38. D'Agostino Jr. R., and Rubin D.B. Estimation and use of propensity scores with incomplete data. Journal of the American Statistical Association 95 (1999) 749-759
39. Dawid A.P. Causal inference without counterfactuals. Journal of the American Statistical Association 95 (2000) 407-448
40. Dehejia R., and Wahba S. Causal effects in non-experimental studies: Re-evaluating the evaluation of training programs. Journal of the American Statistical Association 94 (1999) 1053-1062
41. Delucchi K.L. Methods for the analysis of binary outcome results in the presence of missing data. Journal of Consulting and Clinical Psychology 62 (1994) 569-575
42. Duckart J.P. An evaluation of the Baltimore Community Lead Education and Reduction Corp (CLEARCorps) Program. Evaluation Review 22 (1998) 373-402
43. Fisher R.A. Statistical methods for research workers. 1st ed. (1925), Oliver & Boyd, London
44. Fisher R.A. The arrangement of field experiments. Journal of the Ministry of Agriculture 33 (1926) 503-513
45. Folkman J. Fighting cancer by attacking its blood supply. Scientific American 275 (1996) 150-154
46. Frangakis C., and Rubin D.B. Principal stratification in causal inference. Biometrics 58 (2002) 21-29
47. Frangakis C., Rubin D.B., and Zhou X.-H. Clustered encouragement designs with individual noncompliance: Bayesian inference with randomization, and applications to advanced directive forms. Biostatistics 3 (2002) 147-177
48. Glazerman S., Levy D.M., and Myers D. Nonexperimental versus experimental estimates of earnings impacts. Annals of the American Academy of Political and Social Science 589 (2003) 63-93
49. Goldberger A.S. Selection bias in evaluating treatment effects: Some formal illustrations (Discussion Paper 123-72) (1972), University of Wisconsin, Institute for Research on Poverty, Madison
50. Hahn J., Todd P., and Van der Klaauw W. Identification and estimation of treatment effects with a regression-discontinuity design. Econometrica 69 (2001) 201-209
51. Haviland A., Nagin D.S., and Rosenbaum P.R. Combining propensity score matching and group-based trajectory analysis in an observational study. Psychological Methods 12 (2007) 247-267
52. Heider F. The psychology of interpersonal relations (1958), Wiley, New York
53. Henry G., and Mark M.M. Beyond use: Understanding evaluation's influence on attitudes and actions. American Journal of Evaluation 24 (2003) 293-314
54. Holland P.W. Statistics and causal inference. Journal of the American Statistical Association 81 (1986) 945-970
55. Holland P.W. Causal inference, path analysis, and recursive structural equations models. Sociological Methodology 18 (1988) 449-484
56. Holland P.W., and Rubin D.B. Causal inference in retrospective studies. Evaluation Review 12 (1988) 203-231
57. Hong G., and Raudenbush S.W. Effects of kindergarten retention policy on children's cognitive growth in reading and mathematics. Educational Evaluation and Policy Analysis 27 (2005) 205-224
58. Hume D. An enquiry concerning human understanding (1963), Open Court Press, LaSalle, IL (Original work published 1748)
59. Kuhn T.S. The structure of scientific revolutions (1962), University of Chicago Press, Chicago
60. Kurth T., Walker A.M., Glynn R.J., Chan K.A., Gaziano J.M., Berger K., and Robins J. Results of multivariable logistic regression, propensity matching, propensity adjustment, and propensity-based weighting under conditions of nonuniform effect. American Journal of Epidemiology 163 (2005) 262-270
61. Lazarsfeld P.F. The use of panels in social research. Proceedings of the American Philosophical Society 92 (1948) 405-410
62. Lewis D. Counterfactuals (1973), Blackwell, Cambridge, MA
63. Light R.J., Singer J.D., and Willett J.B. By design: Planning research in higher education (1990), Harvard University Press, Cambridge, MA
64. Lipsey M.W. Design sensitivity: Statistical power for experimental research (1990), Sage, Thousand Oaks, CA
65. Little R.J.A., and Rubin D.B. Causal effects in clinical and epidemiological studies via potential outcomes: Concepts and analytic approaches. Annual Review of Public Health 21 (2000) 121-145
66. Little R.J.A., and Rubin D.B. Statistical analysis with missing data. 2nd ed. (2002), Wiley, New York
67. Lo Y., Matthysse S., Rubin D.B., and Holzman P.S. Permutation tests for detecting and estimating mixtures in task performance within groups. Statistics in Medicine 21 (2002) 1937-1953
68. Luellen J.K., Shadish W.R., and Clark M.H. Propensity scores: An introduction and experimental test. Evaluation Review 29 (2005) 530-558
69. Mackie J.L. The cement of the universe: A study of causation (1974), Oxford University Press, Oxford, England
70. Magidson J. Toward a causal model approach for adjusting for preexisting differences in the nonequivalent control group situation. Evaluation Quarterly 1 (1977) 399-420
71. Maxwell S.E., and Delaney H.D. Designing experiments and analyzing data: A model comparison approach. 2nd ed. (2003), Brooks/Cole, Pacific Grove, CA
72. McCall W.A. How to experiment in education (1923), Macmillan, New York
73. McKillip J. Research without control groups: A control construct design. In: Bryant F.B., Edwards J., Tindale R.S., Posavac E.J., Heath L., and Henderson E. (Eds). Methodological issues in applied psychology (1992), Plenum, New York 159-175
74. McKnight P.E., McKnight K.M., Sidani S., and Figueredo A.J. Missing data: A gentle introduction (2007), Guilford, New York
75. Meyer B.D. Natural and quasi-experiments in economics. Journal of Business and Economic Statistics 13 (1995) 151-161
76. Meyer B.D., Viscusi W.K., and Durbin D.L. Workers' compensation and injury duration: Evidence from a natural experiment. American Economic Review 85 (1995) 322-340
77. Millsap M.A., Chase A., Obeidallah D., Perez-Smith A.P., Brigham N., and Johnston K. Evaluation of Detroit's Comer Schools and Families Initiative: Final report (2000), Abt Associates, Cambridge, MA
78. Morgan S.L., and Winship C. Counterfactuals and causal inference (2007), Cambridge University Press, Cambridge, England
79. Neustrom M.W., and Norton W.M. The impact of drunk driving legislation in Louisiana. Journal of Safety Research 24 (1993) 107-121
80. Neyman J. On the application of probability theory to agricultural experiments: Essay on principles, Section 9. Statistical Science 5 (1990) 465-480 (Original work published 1923)
81. Oakes J.M. The (mis)estimation of neighborhood effects: Causal inference for a practicable social epidemiology. Social Science and Medicine 58 (2004) 1929-1952
82. Pearl J. Causality: Models, reasoning, and inference (2000), Cambridge University Press, Cambridge, England
83. Pohl S., Steyer R., and Kraus K. Modeling method effects as individual causal effects. Journal of the Royal Statistical Society: Series A 171 (2008) 41-63
84. Pollack A. Judah Folkman, Researcher, Dies at 74. New York Times (2008, January 16). http://www.nytimes.com/2008/01/16/us/16folkman.html?_r=1&scp=1&sq=Judah% 20Folkman%20&st=cse Retrieved April 19, 2009, from http://www.nytimes.com/2008/01/16/us/16folkman.html?_r=1&scp=1&s q=Judah%20Folkman%20&st=cse
85. Popper K.R. The logic of scientific discovery (1959), Basic Books, New York
86. Potvin L., and Campbell D.T. The case-control method as a quasi-experimental design. In: Campbell D.T., and Russo M.J. (Eds). Social experimentation (1999), Sage, Thousand Oaks, CA 363-374
87. Reichardt C.S. A typology of strategies for ruling out threats to validity. In: Bickman L. (Ed). Research design: Donald Campbell's legacy 2 (2000), Sage, Thousand Oaks, CA 89-115
88. Reichardt C.S. The principle of parallelism in the design of studies to estimate treatment effects. Psychological Methods 11 (2006) 1-18
89. Reichardt C.S., and Gollob H.F. Satisfying the constraints of causal modeling. In: Trochim W.M.K. (Ed). Advances in quasi-experimental design and analysis (1986), Jossey-Bass, San Francisco 91-107
90. Reichardt C.S., and Gollob H.F. Taking uncertainty into account when estimating effects. In: Mark M.M., and Shotland R.L. (Eds). Multiple methods for program evaluation (1987), Jossey-Bass, San Francisco 7-22
91. Reskin B.F. Including mechanisms in our models of ascriptive inequality. American Sociological Review 68 (2003) 1-21
92. Reynolds K.D., and West S.G. A multiplist strategy for strengthening nonequivalent control group designs. Evaluation Review 11 (1987) 691-714
93. Rogosa D. A critique of cross-lagged correlation. Psychological Bulletin 88 (1980) 245-258
94. Rosenbaum P.R. Discussing hidden bias in observational studies. Annals of Internal Medicine 115 (1991) 901-905
95. Rosenbaum P.R. Sensitivity analysis for matched case-control studies. Biometrics 47 (1991) 87-100
96. Rosenbaum P.R. Observational studies. 2nd ed. (2002), Springer-Verlag, New York
97. Rosenthal R., and Rosnow R.L. Essentials of behavioral research: Methods and data analysis (1991), McGraw-Hill, New York
98. Rosnow R.L., Rosenthal R., and Rubin D.B. Contrasts and correlations in effect size estimation. Psychological Science 11 (2000) 446-453
99. Rowe P.M. What is all the hullabaloo about endostatin?. Lancet 353 (1999) 732
100. Rubin D.B. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology 66 (1974) 688-701
101. Rubin D.B. Assignment to treatment group on the basis of a covariate. Journal of Educational Statistics 2 (1977) 1-26
102. Rubin D.B. Multiple imputation for nonresponse in surveys (1987), Wiley, New York
103. Rubin D.B. A new perspective. In: Wachter K.W., and Straf M.L. (Eds). The future of meta-analysis (1990), Sage Foundation, New York 155-165
104. Rubin D.B. Meta-analysis: Literature synthesis or effect-size surface estimation?. Journal of Educational Statistics 17 (1992) 363-374
105. Rubin D.B. Using propensity scores to help design observational studies: Application to the tobacco litigation. Health Services and Outcomes Research Methodology 2 (2001) 169-188
106. Rubin D.B. Teaching statistical inference for causal effects in experiments and observational studies. Journal of Educational and Behavioral Statistics 29 (2004) 343-367
107. Rubin D.B. Causal inference using potential outcomes. Journal of the American Statistical Association 100 (2005) 322-331
108. Rubin D.B. Matched sampling for causal effects (2006), Cambridge University Press, New York
109. Rubin D.B., Stuart E.A., and Zanutto E.L. A potential outcomes view of value-added assessment in education. Journal of Educational and Behavioral Statistics 24 (2004) 103-116
110. Rubin D.B., and Thomas N. Characterizing the effect of matching using linear propensity score methods with normal covariates. Biometrika 79 (1992) 797-809
111. Sackett D.L. Bias in analytic research. Journal of Chronic Diseases 32 (1979) 51-63
112. Shadish W.R. The empirical program of quasi-experimentation. In: Bickman L. (Ed). Validity and social experimentation: Donald Campbell's legacy (2000), Sage, Thousand Oaks, CA 13-35
113. Shadish W.R., Clark M.H., and Steiner P.M. Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random to nonrandom assignment. Journal of the American Statistical Association 103 (2008) 1334-1343
114. Shadish W.R., and Cook T.D. Design rules: More steps towards a complete theory of quasi-experimentation. Statistical Science 14 (1999) 294-300
115. Shadish W.R., and Cook T.D. The renaissance of field experimentation in evaluating interventions. Annual Review of Psychology 60 (2009) 607-629
116. Shadish W.R., Cook T.D., and Campbell D.T. Experimental and quasi-experimental design for generalized causal inference (2002), Houghton Mifflin, Boston
117. Shadish W.R., Cook T.D., and Leviton L.C. Foundations of program evaluation: Theories of practice (1991), Sage, Newbury Park, CA
118. Shadish W.R., and Fuller S. The social psychology of science (1994), Guilford, New York
119. Shadish W.R., Hu X., Glaser R.R., Kownacki R.J., and Wong T. A method for exploring the effects of attrition in randomized experiments with dichotomous outcomes. Psychological Methods 3 (1998) 3-22
120. Shadish W.R., Luellen J.K., and Clark M.H. Propensity scores and quasi-experimentation. In: Bootzin R.R., and McKnight P. (Eds). Strengthening research methodology: Psychological measurement and evaluation (2006), American Psychological Association, Washington, DC 143-157
121. Shadish W.R., Matt G.E., Navarro A.M., and Phillips G. The effects of psychological therapies under clinically representative conditions: A meta-analysis. Psychological Bulletin 126 (2000) 512-529
122. Shadish W.R., Phillips G., and Clark M.H. Content and context: The impact of Campbell and Stanley. In: Sternberg R.J. (Ed). The anatomy of impact: What makes the great works of psychology great? (2003), American Psychological Association, Washington, DC 161-176
123. Shadish W.R., and Rindskopf D.M. Methods for evidence-based practice: Quantitative synthesis of single-subject designs. In: Julnes G., and Rog D.J. (Eds). Informing federal policies on evaluation method: Building the evidence base for method choice in government sponsored evaluation (2007), Jossey-Bass, San Francisco 95-109
124. Shih W.J., and Quan H. Testing for treatment differences with dropouts present in clinical trials-A composite approach. Statistics in Medicine 16 (1997) 1225-1239
125. Sobel M.E. Identification of causal parameters in randomized studies with mediating variables. Journal of Educational and Behavioral Statistics 33 (2008) 230-251
126. Solomon R.L. An extension of control group design. Psychological Bulletin 46 (1949) 137-150
127. Stanley T.D., and Jarrell S.B. Gender wage discrimination bias: A meta-regression analysis. Journal of Human Resources 33 (1998) 947-973
128. Steyer R. Analyzing individual and average causal effects via structural equation models. Methodology: European Journal of Research Methods in the Behavioral and Social Sciences 1 (2005) 39-54
129. Thistlewaite D.L., and Campbell D.T. Regression-discontinuity analysis: Alternative to the ex post facto experiment. Journal of Educational Psychology 51 (1960) 309-317
130. Trochim W.M.K., and Cappelleri J.C. Cutoff assignment strategies for enhancing randomized clinical trials. Controlled Clinical Trials 13 (1992) 190-212
131. Vanhonacker W.R. Meta-analysis and response surface extrapolation: A least squares approach. American Statistician 50 (1996) 294-299
132. Verbeke G., and Molenberghs G. Linear mixed models for longitudinal data (2000), Springer, New York
133. West S.G., Biesanz J., and Pitts S.C. Causal inference and generalization in field settings: Experimental and quasi-experimental designs. In: Reis H.T., and Judd C.M. (Eds). Handbook of research methods in social psychology (2000), Cambridge University Press, New York 40-84
134. White P.A. Causal attribution and Mill's methods of experimental inquiry: Past, present and prospect. British Journal of Social Psychology 39 (2000) 429-447
135. Winship C., and Morgan S.L. The estimation of causal effects from observational data. Annual Review of Psychology 25 (1999) 659-706
136. Woodward J. Making things happen: A theory of causal explanation (2003), Oxford University Press, New York
137. Yeaton W.H., Wortman P.M., and Langberg N. Differential attrition: Estimating the effect of crossovers on the evaluation of a medical technology. Evaluation Review 7 (1983) 831-840