Analysis and meta-analysis of single-case designs with a standardized mean difference statistic: A primer and applications

Journal of School Psychology

Volumn 52, Issue 2, 2014, Pages 123-147

  Shadish, William R ^a   Hedges, Larry V ^b   Pustejovsky, James E ^c  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b NORTHWESTERN UNIVERSITY   (United States)

^c UNIVERSITY OF TEXAS AT AUSTIN   (United States)

Author keywords

Analysis; D Statistic; Meta analysis; Single case designs

Indexed keywords

HUMAN; META ANALYSIS (TOPIC); METHODOLOGY; STANDARDS; STATISTICAL ANALYSIS;

DATA INTERPRETATION, STATISTICAL; HUMANS; META-ANALYSIS AS TOPIC; RESEARCH DESIGN;

EID: 84897590833     PISSN: 00224405     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jsp.2013.11.005     Document Type: Article

References (80)

1. Allison D.B., Gorman B.S. "Making things as simple as possible, but no simpler". A rejoinder to Scruggs and Mastropieri. Behaviour Research and Therapy 1994, 32:885-890.
2. Arthur W., Bennett W., Huffcutt A.I. Conducting meta-analysis using SAS 2001, Psychology Press, New York, NY.
3. Baer D.M. Perhaps it would be better not to know everything. Journal of Applied Behavior Analysis 1977, 10:167-172.
4. 2-change effect size indices for single-subject experimental designs. Evidence-Based Communication Assessment and Intervention 2008, 2:120-128.
5. Borenstein M., Hedges L.V., Higgins J.P.T., Rothstein H.R. Comprehensive meta-analysis, version 2 2005, Biostat, Englewood, NJ.
6. Borenstein M., Hedges L.V., Higgins J.P.T., Rothstein H.R. Introduction to meta-analysis 2009, Wiley, West Sussex, UK.
7. Busk P.L., Serlin R.C. Meta-analysis for single-case research. Single-case research design and analysis: New directions for psychology and education 1992, 187-212. Erlbaum, Hillsdale, NJ. T.R. Kratochwill, J.R. Levin (Eds.).
8. Chen D.-G., Peace K.E. Applied meta-analysis with R 2013, CRC Press, Boca Raton, FL.
9. Cicerone K.D., Dahlberg C., Kalmar K., Langenbahn D.M., Malec J.F., Bergquist T.F., et al. Evidence-based cognitive rehabilitation: Recommendations for clinical practice. Archives of Physical Rehabilitation Medicine 2000, 81:1596-1615.
10. Cicerone K.D., Dahlbert C., Malec J.F., Langenbahn D.M., Felicetti T., Kneipp S., et al. Evidence-based cognitive rehabilitation: Update of the literature from 1998 through 2002. Archives of Physical Rehabilitation Medicine 2005, 86:1681-1692. 10.1016/j.apmr.2005.03.024.
11. Cicerone K.D., Langenbahn D.M., Braden C., Malec J.F., Kalmar K., Fraas M., et al. Evidence-based cognitive rehabilitation: Updated review of the literature from 2003 through 2008. Archives of Physical Rehabilitation Medicine 2011, 92:519-530.
12. Citkowicz M., Vevea J.L. A parsimonious weight function for modeling publication bias. Paper presented at the annual meeting of the Society for Research Synthesis Methodology, Ottawa, Canada 2011.
13. Cook T.D., Shadish W.R., Wong V.C. Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management 2008, 27:724-750.
14. Duval S.J. The trim and fill method. Publication bias in meta-analysis: Prevention, assessment, and adjustments 2005, 127-144. Wiley, Chichester, England. H.R. Rothstein, A.J. Sutton, M. Borenstein (Eds.).
15. Duval S.J., Tweedie R.L. Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 2000, 56:455-463.
16. Duval S.J., Tweedie R.L. A nonparametric "trim and fill" method of accounting for publication bias in meta-analysis. Journal of the American Statistical Association 2000, 95(449):89-98.
17. Eikeseth S. Outcome of comprehensive psycho-educational interventions for young children with autism. Research in Developmental Disabilities 2009, 30(1):158-178.
18. Fabiano G.A., Pelham W.E., Coles E.K., Gnagy E.M., Chronis-Tuscano A., O'Connor B.C. A meta-analysis of behavioral treatments for attention-deficit/hyperactivity disorder. Clinical Psychology Review 2009, 29:129-140.
19. Field A.P., Gillett R. How to do a meta-analysis. British Journal of Mathematical and Statistical Psychology 2010, 63:665-694. 10.1348/000711010X502733.
20. Fox J., Weisberg S. An R companion to applied regression 2011, Sage, Thousand Oaks, CA.
21. Galbraith R.F. Graphical display of estimates having differing standard errors. Technometrics 1988, 30:271-281.
22. Galbraith R.F. A note on graphical presentation of estimated odds ratios from several clinical trials. Statistics in Medicine 1988, 7:889-894.
23. Galbraith R.F. Some applications of radial plots. Journal of the American Statistical Association 1994, 89:1232-1242.
24. Gorman-Smith D., Matson J.L. A review of treatment research for self-injurious and stereotyped responding. Journal of Mental Deficiency Research 1985, 29:295-308.
25. Hedges L.V. Distribution theory for Glass's estimator of effect size and related estimators. Journal of Educational Statistics 1981, 6:107-128.
26. Hedges L.G., Pustejovsky J.E., Shadish W.R. A standardized mean difference effect size for single-case designs. Research Synthesis Methods 2012, 3:224-239.
27. Hedges L.G., Pustejovsky J.E., Shadish W.R. A standardized mean difference effect size for multiple baseline designs across individuals. Research Synthesis Methods 2013, 10.1002/jrsm.1086.
28. Hedges L.V., Shadish W.R. Power analysis for single case designs 2013, (Manuscript in preparation).
29. Hedges L.V., Shadish W.R. Power analysis for multiple baseline designs 2013, (Manuscript in preparation).
30. Hedges L.V., Vevea J.L. Estimating effect size under publication bias: Small sample properties and robustness of a random effects selection model. Journal of Educational and Behavioral Statistics 1996, 21:299-333.
31. Hedges L.V., Vevea J.L. Fixed- and random-effects models in meta-analysis. Psychological Methods 1998, 3:486-504.
32. Kennedy M., Coehlo C., Turkstra L., Ylvisaker M., Sohlberg M.M., Yorkston K., et al. Intervention for executive functions after traumatic brain injury: a systematic review, meta-analysis and clinical recommendations. Neuropsychological Rehabilitation 2008, 18(3):257-299.
33. Knapp G., Hartung J. Improved tests for a random effects meta-regression with a single covariate. Statistics in Medicine 2003, 22(17):2693-2710.
34. Kratochwill T.R., Hitchcock J., Horner R.H., Levin J.R., Odom S.L., Rindskopf D.M., et al. Single-case designs technical documentation Retrieved from What Works Clearinghouse website:. http://ies.ed.gov/ncee/wwc/pdf/wwc_scd.pdf.
35. Kratochwill T.R., Hitchcock J., Horner R.H., Levin J.R., Odom S.L., Rindskopf D.M., et al. Single-case intervention research design standards. Remedial and Special Education 2013, 34:26-38. 10.1177/0741932512452794.
36. Lambert M.C., Cartledge G., Heward W.L., Lo Y. Effects of response cards on disruptive behavior and academic responding during math lessons by fourth-grade urban students. Journal of Positive Behavior Interventions 2006, 8e:88-99.
37. Laski K.E., Charlop M.H., Schreibman L. Training parents to use the natural language paradigm to increase their autistic children's speech. Journal of Applied Behavior Analysis 1988, 21:391-400.
38. LeBlanc L.A., Geiger K.B., Sautter R.A., Sidener T.M. Using the natural language paradigm (NLP) to increase vocalizations of older adults with cognitive impairments. Research in Developmental Disabilities 2007, 28:437-444.
39. Lewis S., Clarke M. Forest plots: Trying to see the wood and the trees. British Medical Journal 2001, 322:1279-1480.
40. Lipsey M.W., Wilson D.B. The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist 1993, 48:1181-1209.
41. Lipsey M.W., Wilson D.B. Practical meta-analysis 2000, Sage, Newbury Park, CA.
42. Maggin D.M., O'Keeffe B.V., Johnson A.H. A quantitative synthesis of methodology in the meta-analysis of single-subject research for students with disabilities: 1985-2009. Exceptionality 2011, 19:109-135. 10.1080/09362835.2011.565725.
43. Maggin D.M., Swaminathan H., Rogers H.J., O'Keefe B.V., Sugai G., Horner R.H. A generalized least squares regression approach for computing effect sizes in single-case research: Application examples. Journal of School Psychology 2011, 49:301-321. 10.1016/j.jsp.2011.03.004.
44. Mahoney M. Publication prejudices: An experimental study of confirmatory bias on the peer review system. Cognitive Therapy and Research 1977, 1:161-175.
45. Moeyaert M., Ferron J., Beretvas S.N., Van den Noortgate W. From a single-level analysis to a multilevel analysis of single-case experimental designs. Journal of School Psychology 2013, 52:191-211. (this issue).
46. Parker R.I., Hagan-Burke S., Vannest K. Percentage of all non-overlapping data (PAND): An alternative to PND. Journal of Special Education 2007, 40:194-204.
47. Parker R.I., Vannest K. An improved effect size for single-case research: Nonoverlap of all pairs. Behavior Therapy 2009, 40:357-367.
48. Parker R.I., Vannest K.J., Davis J.L. Effect size in single case research: A review of nine nonoverlap techniques. Behavior Modification 2011, 35:303-322. 10.1177/0145445511399147.
49. Parker R.I., Vannest K.J., Davis J.L., Sauber S.B. Combining nonoverlap and trend for single-case research: Tau-U. Behavior Therapy 2011, 42:284-299.
50. Perone M. Statistical inference in behavior analysis: Experimental control is better. The Behavior Analyst 1999, 22:109-116.
51. Pustejovsky J.E. Operationally comparable effect sizes for meta-analysis of single-case research 2013, (Unpublished doctoral dissertation), Northwestern University, Evanston, IL.
52. R Development Core Team R: A language and environment for statistical computing 2012, R Foundation for Statistical Computing, Vienna, Austria, (URL http://www.R-project.org/).
53. Rao P., Beidel D., Murray M. Social skills interventions for children with Asperger's syndrome or high-functioning autism: A review and recommendations. Journal of Autism and Developmental Disorders 2008, 38:353-361.
54. Ringdahl J.E., Call N.A., Mews J.B., Boelter E.W., Christensen T.J. Assessment and treatment of aggressive behavior without a clear social function. Research in Developmental Disabilities 2008, 29:351-362. 10.1016/j.ridd.2007.06.003.
55. Rogers L.A., Graham S. A meta-analysis of single subject design writing intervention research. Journal of Educational Psychology 2008, 100:879-906.
56. Publication bias in meta-analysis: Prevention, assessment and adjustments 2005, Wiley, New York, NY. H. Rothstein, A.J. Sutton, M. Borenstein (Eds.).
57. Schreibman L., Stahmer A.C., Barlett V.C., Dufek S. Brief report: toward refinement of a predictive behavioral profile for treatment outcome in children with autism. Research in Autism Spectrum Disorders 2009, 3:163-172.
58. Scruggs T.E., Mastropieri M.A. PND at 25: Past, present, and future trends in summarizing single-subject research. Remedial and Special Education 2013, 34:9-19. 10.1177/0741932512440730.
59. Scruggs T.E., Mastropieri M.A., Casto G. The quantitative synthesis of single-case research: Methodology and validation. Remedial and Special Education 1987, 8(2):24-33.
60. Shadish W.R. Analysis and meta-analysis of single-case designs: An introduction. Journal of School Psychology 2013, 52:109-122. (this issue).
61. Shadish W.R., Clark M.H., Steiner P.M. Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random to nonrandom assignment. Journal of the American Statistical Association 2008, 103:1334-1343.
62. Shadish W.R., Galindo R.G., Steiner P.M., Wong V.C., Cook T.D. A randomized experiment comparing random to cutoff-based assignment. Psychological Methods 2011, 16:179-191.
63. Shadish W.R., Haddock C.K. Combining estimates of effect size. The handbook of research synthesis and meta-analysis 2009, 257-277. Sage, New York, NY. 2nd ed. H.M. Cooper, L.V. Hedges, J.C. Valentine (Eds.).
64. Shadish W.R., Hedges L.V., Pustejovsky J.E., Boyajian J.G., Sullivan K.J., Andrade A., et al. A d-statistic for single-case designs that is equivalent to the usual between-groups d-statistic. Neuropsychological Rehabilitation 2013, 10.1080/09602011.2013.819021.
65. Shadish W.R., Hedges L.V., Pustejovsky J., Rindskopf D.M., Boyajian J.G., Sullivan K.J. Analyzing single-case designs: d, G, multilevel models, Bayesian estimators, generalized additive models, and the hopes and fears of researchers about analyses. Single-case intervention research: Methodological and data-analysis advances 2013, American Psychological Association, Washington, DC, (in press). T.R. Kratochwill, J.R. Levin (Eds.).
66. Shadish W.R., Kyse E.N., Rindskopf D.M. Analyzing data from single-case designs using multilevel models: New applications and some agenda items for future research. Psychological Methods 2013, 18:385-405. 10.1037/a0032964.
67. Shadish W.R., Marso D.M. SPSS macros for effect sizes and power in single-case (N-of-1) designs. Paper presented at the annual meeting of the Society for Research Synthesis Methodology, Providence, Rhode Island 2013.
68. Shadish W.R., Rindskopf D.M. Methods for evidence-based practice: Quantitative synthesis of single-subject designs. Informing federal policies on evaluation method: Building the evidence base for method choice in government sponsored evaluation 2007, 95-109. Jossey-Bass, San Francisco, CA. G. Julnes, D.J. Rog (Eds.).
69. Shadish W.R., Sullivan K.J. Characteristics of single-case designs used to assess intervention effects in 2008. Behavior Research Methods 2011, 43:971-980. (ERIC #ED530280). 10.3758/s13428-011-0111-y.
70. Shadish W.R., Zuur A.F., Sullivan K.J. Using generalized additive (mixed) models to analyze single case designs. Journal of School Psychology 2013, 52:149-178. (this issue).
71. Meta-analysis in Stata: An updated collection from the Stata Journal 2009, Stata Press, College Station, TX. J.A.C. Sterne (Ed.).
72. Swaminathan H., Rogers H.J., Horner R.H. An effect size measure and Bayesian analysis of single-case designs. Journal of School Psychology 2013, 52:213-230. (this issue).
73. Thorp D.M., Stahmer A.C., Schreibman L. The effects of sociodramatic play training on children with autism. Journal of Autism and Developmental Disorders 1995, 25:265-282.
74. Van den Noortgate W., Onghena P. A multilevel meta-analysis of single-subject experimental design studies. Evidence-Based Communication Assessment and Intervention 2008, 2:142-151.
75. Vevea J.L., Hedges L.V. A general linear model for estimating effect size in the presences of publication bias. Psychometrika 1995, 60:419-435.
76. Vevea J.L., Woods C.M. Publication bias in research synthesis: Sensitivity analysis using a priori weight functions. Psychological Methods 2005, 10:428-443. 10.1037/1082-989X.10.4.428.
77. Viechtbauer W. Conducting meta-analyses in R with the metafor package. Journal of Statistical Software 2010, 36(3):1-48. (http://www.jstatsoft.org/v36/i03/).
78. Viechtbauer W., Cheung M.W.-L. Outlier and influence diagnostics for meta-analysis. Research Synthesis Methods 2010, 1:112-125.
79. Wilkinson L., the Task Force on Statistical Inference Statistical methods in psychology journals: Guidelines and explanations. American Psychologist 1999, 54:594-604.
80. Wilson D.B. Meta-analysis macros for SAS, SPSS, and Stata Retrieved May 22, 2013, from. http://mason.gmu.edu/~dwilsonb/ma.html.