Examining the Internal Validity and Statistical Precision of the Comparative Interrupted Time Series Design by Comparison With a Randomized Experiment

American Journal of Evaluation

Volumn 35, Issue 3, 2014, Pages 311-327

  St Clair, Travis ^a   Cook, Thomas D ^b,c   Hallberg, Kelly ^d  

^a UNIVERSITY OF MARYLAND   (United States)

^b NORTHWESTERN UNIVERSITY   (United States)

^c MATHEMATICA POLICY RESEARCH   (United States)

^d AMERICAN INSTITUTES FOR RESEARCH   (United States)

Author keywords

educational evaluation; interrupted time series; randomized clinical trial; within study comparison

Indexed keywords

EID: 84905181563     PISSN: 10982140     EISSN: 15570878     Source Type: Journal    
DOI: 10.1177/1098214014527337     Document Type: Article

References (26)

1. Ballart X.,Riba C.Impact of legislation requiring moped and motorbike riders to wear helmets.Evaluation and Program Planning. 1995;18:311-320.
2. Campbell D. T.,Erlebacher A. E.The disadvantaged child: vol. 3, Compensatory education: A national debate. Hellmuth J., ed. New York, NY: Brunner/Mazel; 1970:185-210.
3. Campbell D. T.,Ross H. L.The Connecticut crackdown on speeding: Time-series data in quasi-experimental analysis.Law & Society Review. 1968;3:33-54.
4. Cook T. D.,Campbell D. T.Quasi-experimentation: Design and analysis issues for field settings. Boston, MA: Houghton Mifflin; 1979:.
5. Cook T. D.,Shadish W. R.,Wong V. C.Three conditions under which experiments and observational studies often produce comparable causal estimates: New findings from within-study comparisons.Journal of Policy Analysis and Management. 2008;27:724-750.
6. Cook T. D.,Steiner P. M.Case matching and the reduction of selection bias in quasi-experiments: The relative importance of covariate choice, unreliable measurement and mode of data analysis.Psychological Methods. 2010;15:56-68.
7. Diaz J. J.,Handa S.An assessment of propensity score matching as a nonexperimental impact estimator: Evidence from Mexico's PROGRESA program.Journal of Human Resources. 2006;41:319-345.
8. Fretheim A.,Soumerai S. B.,Zhang F.,Oxman A. D.,Ross-Degnan D.Interrupted time-series analysis yielded an effect estimate concordant with the cluster randomized controlled-trial result.Journal of Clinical Epidemiology. 2013;66:883-887.
9. Heckman J.,Ichimura H.,Smith J. C.,Todd P.Characterizing selection bias.Econometrica. 1998;66:1107-1098.
10. Heckman J.,Ichimura H.,Todd P. E.Matching as an econometric evaluation estimator: Evidence from evaluating a job training program.Review of Economic Studies. 1997;64:605-654.
11. Heckman J.,Ichimura H.,Todd P. E.Matching as an econometric evaluation estimator.Review of Economic Studies. 1998;65:261-294.
12. KratochwillT. R.HitchcockJ.HornerR. H.LevinJ. R.OdomS. L.RindskopfD. M.ShadishW. (2010). Single-case designs technical documentation. Retrieved fromhttp://ies.ed.gov/ncee/wwc/documentsum.aspx?sid=229.
13. Lalonde R.Evaluating the econometric evaluations of training with experimental data.American Economic Review. 1986;76:604-620.
14. Michalopoulos C.,Bloom H. S.,Hill C. J.Can propensity-score methods match the findings from a random assignment evaluation of mandatory welfare-to-work programs?.Review of Economics and Statistics. 2004;86:156-179.
15. Mulford H. A.,Ledolter J.,Fitzgerald J. L.Alcohol availability and consumption: Iowa sales data revisited.Journal of Studies on Alcohol. 1992;53:487-494.
16. Pohl S.,Steiner P. M.,Eisermann J.,Soellner R.,Cook T. D.Unbiased causal inference from an observational study: Results of a within-study comparison.Educational Evaluation and Policy Analysis. 2009;31:463-479.
17. Roifman C. M.,Levison H.,Gelfand E. W.High-dose versus low-dose intravenous immunoglobulin in hypogammaglobulinaemia and chronic lung disease.The Lancet. 1987;329:1075-1077.
18. Rubin D.The design and analysis of gold standard randomized experiments. Comment on "Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random to nonrandom assignment" by Shadish, Clark, and Steiner.The Journal of the American Statistical Association. 2008;103:1350-1353.
19. Schneeweiss S.,Maclure M.,Carleton B.,Glynn R. J.,Avorn J.Clinical and economic consequences of a reimbursement restriction of nebulised respiratory therapy in adults: Direct comparison of randomised and observational evaluations.British Medical Journal. 2004;328:560.
20. Shadish W. R.,Clark M. H.,Steiner P. M.Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random to nonrandom assignment.Journal of the American Statistical Association. 2008;103:1334-1343.
21. Shadish W. R.,Cook T. D.,Campbell D. T.Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin; 2002:.
22. Shadish W. R.,Galindo R.,Wong V.,Steiner P.,Cook T.A randomized experiment comparing random and cutoff-based assignment.Psychological Methods. 2011;16:179-191.
23. Shadish W. R.,Rindskopf D. M.,Boyajian J. G.A comparison of results from single-case designs to a randomized experiment. n.d.;:.
24. Smith J. C.,Todd P.Does matching overcome LaLonde's critique of nonexperimental estimators.Journal of Econometrics. 2005;125:305-353.
25. Somers M.,Zhu P.,Jacob R.,Bloom H.The validity and precision of the comparative interrupted time series design and the difference-in-difference design in educational evaluation. MDRC: New York, NY; 2013:.
26. Steiner P.,Cook T. D.,Shadish W.On the importance of reliable covariate measurement in selection bias adjustments using propensity scores.Journal of Educational and Behavioral Statistics. 2011;36:213-236.