Procedural sensitivities of effect sizes for single-case designs with directly observed behavioral outcome measures

Psychological Methods

Volumn 24, Issue 2, 2019, Pages 217-235

  Pustejovsky, James E ^a  

^a UNIVERSITY OF TEXAS AT AUSTIN   (United States)

Author keywords

Alternating renewal process; Behavioral observation; Effect sizes; Meta analysis; Single case research

Indexed keywords

ARTICLE; BEHAVIORAL OBSERVATION; EFFECT SIZE; HUMAN; META ANALYSIS; MONTE CARLO METHOD; OUTCOME ASSESSMENT; SAMPLE SIZE; BEHAVIORAL RESEARCH; METHODOLOGY; STANDARDS; STATISTICAL ANALYSIS; STATISTICAL MODEL;

BEHAVIORAL RESEARCH; DATA INTERPRETATION, STATISTICAL; HUMANS; MODELS, STATISTICAL; OUTCOME ASSESSMENT (HEALTH CARE); RESEARCH DESIGN;

EID: 85048712117     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/met0000179     Document Type: Article

References (73)

1. Adamson, R. M., & Wachsmuth, S. T. (2014). A review of direct observation research within the past decade in the field of emotional and behavioral disorders. Behavioral Disorders, 39, 181-189
2. Allison, D. B., & Gorman, B. S. (1993). Calculating effect sizes for meta-analysis: The case of the single case. Behaviour Research and Therapy, 31, 621-631
3. Allison, D. B., & Gorman, B. S. (1994). "Make things as simple as possible, but no simpler." A rejoinder to Scruggs and Mastropieri. Behaviour Research and Therapy, 32, 885-890. http://dx.doi.org/10 .1016/0005-7967(94)90170-8
4. Ayres, K., & Gast, D. L. (2010). Dependent measures and measurement procedures. In D. L. Gast (Ed.), Single subject research methodology in behavioral sciences (pp. 129-165). New York, NY: Routledge
5. Borenstein, M. (2009). Effect sizes for continuous data. In H. M. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (pp. 221-236). New York, NY: Russell Sage Foundation
6. Busk, P. L., & Serlin, R. C. (1992). Meta-analysis for single-case research. In T. R. Kratochwill & J. R. Levin (Eds.), Single-case research design and analysis: New directions for psychology and education (pp. 187-212). Hillsdale, NJ: Erlbaum
7. Campbell, J. M. (2003). Efficacy of behavioral interventions for reducing problem behavior in persons with autism: A quantitative synthesis of single-subject research. Research in Developmental Disabilities, 24, 120-138. http://dx.doi.org/10.1016/S0891-4222(03)00014-3
8. Campbell, J. M., & Herzinger, C. V. (2010). Statistics and single subject research methodology. In D. L. Gast (Ed.), Single subject research methodology in behavioral sciences (pp. 417-450). New York, NY: Routledge
9. Center, B. A., Skiba, R. J., & Casey, A. (1985-86). A methodology for the quantitative synthesis of intra-subject design research. The Journal of Special Education, 19, 387-400. http://dx.doi.org/10.1177/0022466 98501900404
10. Conroy, M. A., Dunlap, G., Clarke, S., & Alter, P. J. (2005). A descriptive analysis of positive behavioral intervention research With young children with challenging behavior. Topics in Early Childhood Special Education, 25, 157-166. http://dx.doi.org/10.1177/02711214050 250030301
11. DiCarlo, C. F., Reid, D. H., & Stricklin, S. B. (2003). Increasing toy play among toddlers with multiple disabilities in an inclusive classroom: A more-to-less, child-directed intervention continuum. Research in Developmental Disabilities, 24, 195-209. http://dx.doi.org/10.1016/S0891-4222(03)00025-8
12. Duan, N., Kravitz, R. L., & Schmid, C. H. (2013). Single-patient (n-of-1) trials: A pragmatic clinical decision methodology for patient-centered comparative effectiveness research. Journal of Clinical Epidemiology, 66(Suppl. 8), S21-S28. http://dx.doi.org/10.1016/j.jclinepi.2013.04.006
13. Evans, J. J., Gast, D. L., Perdices, M., & Manolov, R. (2014). Single case experimental designs: Introduction to a special issue of Neuropsychological Rehabilitation. Neuropsychological Rehabilitation, 24, 305-314. http://dx.doi.org/10.1080/09602011.2014.903198
14. Gabler, N. B., Duan, N., Vohra, S., & Kravitz, R. L. (2011). N-of-1 trials in the medical literature: A systematic review. Medical Care, 49, 761-768. http://dx.doi.org/10.1097/MLR.0b013e318215d90d
15. Gage, N. A., Lewis, T. J., & Stichter, J. P. (2012). Functional behavioral assessment-based interventions for students with or at risk for emotional and/or behavioral disorders in school: A hierarchical linear modeling meta-analysis. Behavioral Disorders, 37, 55-77
16. Gast, D. L. (Ed.). (2010). Applied research in education and behavioral sciences. Single subject research methodology in behavioral sciences (pp. 1-19). New York, NY: Routledge
17. Gingerich, W. J. (1984). Meta-analysis of applied time-series data. Journal of Applied Behavioral Science, 20, 71-79. http://dx.doi.org/10.1177/002188638402000113
18. Harrington, M., & Velicer, W. F. (2015). Comparing visual and statistical analysis in single-case studies using published studies. Multivariate Behavioral Research, 50, 162-183. http://dx.doi.org/10.1080/00273171 .2014.973989
19. Heath, A. K., Ganz, J. B., Parker, R. I., Burke, M., & Ninci, J. (2015). A meta-analytic review of functional communication training across mode of communication, age, and disability. Review Journal of Autism and Developmental Disorders, 2, 155-166. http://dx.doi.org/10.1007/s40489-014-0044-3
20. Hedges, L. V. (1981). Distribution theory for Glass's estimator of effect size and related estimators. Journal of Educational Statistics, 6, 107-128. http://dx.doi.org/10.3102/10769986006002107
21. Hedges, L. V. (2008). What are effect sizes and why do we need them? Child Development Perspectives, 2, 167-171. http://dx.doi.org/10.1111/j.1750-8606.2008.00060.x
22. Hedges, L. V., Gurevitch, J., & Curtis, P. (1999). The meta-analysis of response ratios in experimental ecology. Ecology, 80, 1150-1156
23. Heyvaert, M., Saenen, L., Campbell, J. M., Maes, B., & Onghena, P. (2014). Efficacy of behavioral interventions for reducing problem behavior in persons with autism: An updated quantitative synthesis of single-subject research. Research in Developmental Disabilities, 35, 2463-2476. http://dx.doi.org/10.1016/j.ridd.2014.06.017
24. Horner, R. H., Carr, E. G., Halle, J., McGee, G., Odom, S. L., & Wolery, M. (2005). The use of single-subject research to identify evidence-based practice in special education. Exceptional Children, 71, 165-179. http://dx.doi.org/10.1177/001440290507100203
25. Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. Thousand Oaks, CA: Sage
26. Kraemer, H. C. (1979). One-zero sampling in the study of primate behavior. Primates, 20, 237-244. http://dx.doi.org/10.1007/BF02373376
27. Kratochwill, T. R., Levin, J. R., Horner, R. H., & Swoboda, C. M. (2014). Visual analysis of single-case intervention research: Conceptual and methodological issues. In T. R. Kratochwill & J. R. Levin (Eds.), Single-case intervention research: Methodological and statistical advances (pp. 91-125). Washington, DC: American Psychological Association
28. Ledford, J. R. (2015). The effects of interval-based measurement on the estimation of effect sizes in single case research. In J. Ganz (Ed.), Issues in and application of meta-analyses and syntheses of single-case experimental research in autism and developmental disabilities (pp. 39-64). San Antonio, TX: Association for Behavior Analysis International Conference. Retrieved from http://hdl.handle.net/1969.1/154270
29. Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage
30. Ma, H.-H. (2006). An alternative method for quantitative synthesis of single-subject researches: Percentage of data points exceeding the median. Behavior Modification, 30, 598-617. http://dx.doi.org/10.1177/0145445504272974
31. Maggin, D. M., & Chafouleas, S. M. (2013). Introduction to the special series: Issues and advances of synthesizing single-case research. Remedial and Special Education, 34, 3-8. http://dx.doi.org/10.1177/0741932512466269
32. Maggin, D. M., Chafouleas, S. M., Goddard, K. M., & Johnson, A. H. (2011). A systematic evaluation of token economies as a classroom management tool for students with challenging behavior. Journal of School Psychology, 49, 529-554. http://dx.doi.org/10.1016/j.jsp.2011 .05.001
33. Maggin, D. M., O'Keeffe, B. V., & Johnson, A. H. (2011). A quantitative synthesis of methodology in the meta-analysis of single-subject research for students with disabilities: 1985-2009. Exceptionality, 19, 109-135. http://dx.doi.org/10.1080/09362835.2011.565725
34. Maggin, D. M., Swaminathan, H., Rogers, H. J., O'Keeffe, B. V., Sugai, G., & Horner, R. H. (2011). A generalized least squares regression approach for computing effect sizes in single-case research: Application examples. Journal of School Psychology, 49, 301-321. http://dx.doi.org/10.1016/j.jsp.2011.03.004
35. Maggin, D. M., Zurheide, J., Pickett, K. C., & Baillie, S. J. (2015). A systematic evidence review of the check-in/check-out program for reducing student challenging behaviors. Journal of Positive Behavior Interventions, 17, 197-208. http://dx.doi.org/10.1177/109830071 5573630
36. Manolov, R., & Solanas, A. (2009). Percentage of nonoverlapping corrected data. Behavior Research Methods, 41, 1262-1271. http://dx.doi .org/10.3758/BRM.41.4.1262
37. Marquis, J. G., Horner, R. H., Carr, E. G., Turnbull, A. P., Thompson, M., Behrens, G. A.,. Doolabh, A. (2000). A meta-analysis of positive behavior support. In R. Gersten, E. P. Schiller, & S. Vaughan (Eds.), Contemporary special education research: Syntheses of the knowledge base on critical instructional issues (pp. 137-178). Mahwah, NJ: Erlbaum
38. McDonald, S., Quinn, F., Vieira, R., O'Brien, N., White, M., Johnston, D. W., & Sniehotta, F. F. (2017). The state of the art and future opportunities for using longitudinal n-of-1 methods in health behaviour research: A systematic literature overview. Health Psychology Review, 7199, 1-17. http://dx.doi.org/10.1080/17437199.2017.1316672
39. Mudford, O. C., Locke, J. M., & Jeffrey, K. (2011). Rates of responding measured by continuous recording in applied behavioral research. Behavioral Interventions, 26, 41-49. http://dx.doi.org/10.1002/bin.323
40. Mudford, O. C., Taylor, S. A., & Martin, N. T. (2009). Continuous recording and interobserver agreement algorithms reported in the Journal of Applied Behavior Analysis (1995-2005). Journal of Applied Behavior Analysis, 42, 165-169. http://dx.doi.org/10.1901/jaba.2009.42-165
41. Parker, R. I., Hagan-Burke, S., & Vannest, K. J. (2007). Percentage of all non-overlapping data (PAND): An alternative to PND. The Journal of Special Education, 40, 194-204. http://dx.doi.org/10.1177/00224 669070400040101
42. Parker, R. I., & Vannest, K. J. (2009). An improved effect size for single-case research: Nonoverlap of all pairs. Behavior Therapy, 40, 357-367. http://dx.doi.org/10.1016/j.beth.2008.10.006
43. Parker, R. I., Vannest, K. J., & Brown, L. (2009). The improvement rate difference for single-case research. Exceptional Children, 75, 135-150. http://dx.doi.org/10.1177/001440290907500201
44. Parker, R. I., Vannest, K. J., & Davis, J. L. (2011). Effect size in singlecase research: A review of nine nonoverlap techniques. Behavior Modification, 35, 303-322. http://dx.doi.org/10.1177/0145445511399147
45. Parker, R. I., Vannest, K. J., & Davis, J. L. (2014). Non-overlap analysis for single-case research. In T. R. Kratochwill & J. R. Levin (Eds.), Single-case intervention research: Methodological and statistical advances (pp. 127-151). Washington, DC: American Psychological Association. http://dx.doi.org/10.1037/14376-005
46. Parker, R. I., Vannest, K. J., Davis, J. L., & Sauber, S. B. (2011). Combining nonoverlap and trend for single-case research: Tau-U. Behavior Therapy, 42, 284-299. http://dx.doi.org/10.1016/j.beth.2010.08 .006
47. Petersen-Brown, S., Karich, A. C., & Symons, F. J. (2012). Examining estimates of effect using non-overlap of all pairs in multiple baseline studies of academic intervention. Journal of Behavioral Education, 21, 203-216. http://dx.doi.org/10.1007/s10864-012-9154-0
48. Pustejovsky, J. E. (2015). Measurement-comparable effect sizes for singlecase studies of free-operant behavior. Psychological Methods, 20, 342-359. http://dx.doi.org/10.1037/met0000019
49. Pustejovsky, J. E. (2016). ARPobservation: Simulating recording procedures for direct observation of behavior. R package Version 1.1. Retrieved from http://cran.r-project.org/web/packages/ARPobservation
50. Pustejovsky, J. E. (2018). Using response ratios for meta-analyzing singlecase designs with behavioral outcomes. Journal of School Psychology, 68, 99-112. http://dx.doi.org/10.1016/j.jsp.2018.02.003
51. Pustejovsky, J. E., & Ferron, J. M. (2016). Research synthesis and metaanalysis of single-case designs. In J. M. Kaufmann, D. P. Hallahan, & P. C. Pullen (Eds.), Handbook of special education, 2nd ed. (pp. 168-186). New York, NY: Routledge
52. Pustejovsky, J. E., & Runyon, C. (2014). Alternating renewal process models for behavioral observation: Simulation methods, software, and validity illustrations. Behavioral Disorders, 39, 211-227
53. Pustejovsky, J. E., & Swan, D. M. (2015). Four methods for analyzing partial interval recording data, with application to single-case research. Multivariate Behavioral Research, 50, 365-380. http://dx.doi.org/10 .1080/00273171.2015.1014879
54. Rogosa, D., & Ghandour, G. (1991). Statistical models for behavioral observations. Journal of Educational Statistics, 16, 157-252. http://dx .doi.org/10.2307/1165191
55. Scruggs, T. E., & Mastropieri, M. A. (1998). Summarizing single-subject research: Issues and applications. Behavior Modification, 22, 221-242. http://dx.doi.org/10.1177/01454455980223001
56. Scruggs, T. E., & Mastropieri, M. A. (2013). PND at 25: Past, present, and future trends in summarizing single-subject research. Remedial and Special Education, 34, 9-19. http://dx.doi.org/10.1177/074193251 2440730
57. Scruggs, T. E., Mastropieri, M. A., & Casto, G. (1987). The quantitative synthesis of single-subject research: Methodology and validation. Remedial and Special Education, 8, 24-43. http://dx.doi.org/10.1177/074193258700800206
58. Shadish, W. R. (2014). Analysis and meta-analysis of single-case designs: An introduction. Journal of School Psychology, 52, 109-122. http://dx .doi.org/10.1016/j.jsp.2013.11.009
59. Shadish, W. R. (2014). Statistical analyses of single-case designs: The shape of things to come. Current Directions in Psychological Science, 23, 139-146. http://dx.doi.org/10.1177/0963721414524773
60. Shadish, W. R., Hedges, L. V., & Pustejovsky, J. E. (2014). Analysis and meta-analysis of single-case designs with a standardized mean difference statistic: A primer and applications. Journal of School Psychology, 52, 123-147. http://dx.doi.org/10.1016/j.jsp.2013.11.005
61. Shadish, W. R., Rindskopf, D. M., & Hedges, L. V. (2008). The state of the science in the meta-analysis of single-case experimental designs. Evidence-Based Communication Assessment and Intervention, 2, 188-196. http://dx.doi.org/10.1080/17489530802581603
62. Shadish, W. R., & Sullivan, K. J. (2011). Characteristics of single-case designs used to assess intervention effects in 2008. Behavior Research Methods, 43, 971-980. http://dx.doi.org/10.3758/s13428-011-0111-y
63. Smith, J. D. (2012). Single-case experimental designs: A systematic review of published research and current standards. Psychological Methods, 17, 510-550. http://dx.doi.org/10.1037/a0029312
64. Solomon, B. G., Howard, T. K., & Stein, B. L. (2015). Critical assumptions and distribution features pertaining to contemporary single-case effect sizes. Journal of Behavioral Education, 24, 438-458. http://dx.doi.org/10.1007/s10864-015-9221-4
65. Tarlow, K. R. (2017). An improved rank correlation effect size statistic for single-case designs: Baseline corrected Tau. Behavior Modification, 41, 427-467. http://dx.doi.org/10.1177/0145445516676750
66. Tate, R. L., Perdices, M., McDonald, S., Togher, L., & Rosenkoetter, U. (2014). The design, conduct and report of single-case research: Resources to improve the quality of the neurorehabilitation literature. Neuropsychological Rehabilitation, 24, 315-331. http://dx.doi.org/10 .1080/09602011.2013.875043
67. Van den Noortgate, W., & Onghena, P. (2008). A multilevel meta-analysis of single-subject experimental design studies. Evidence-Based Communication Assessment and Intervention, 2, 142-151. http://dx.doi.org/10 .1080/17489530802505362
68. Vargha, A., & Delaney, H. D. (2000). A critique and improvement of the "CL" common language effect size statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics, 25, 101-132. http://dx.doi.org/10.2307/1165329
69. White, D. M., Rusch, F. R., Kazdin, A. E., & Hartmann, D. P. (1989). Applications of meta analysis in individual-subject research. Behavioral Assessment, 11, 281-296
70. White, O. R. (1987). Some comments concerning "The quantitative synthesis of single-subject research." Remedial and Special Education, 8, 34-39. http://dx.doi.org/10.1177/074193258700800207
71. Wirth, O., Slaven, J., & Taylor, M. A. (2014). Interval sampling methods and measurement error: A computer simulation. Journal of Applied Behavior Analysis, 47, 83-100. http://dx.doi.org/10.1002/jaba.93
72. Wolery, M., Busick, M., Reichow, B., & Barton, E. E. (2010). Comparison of overlap methods for quantitatively synthesizing single-subject data. The Journal of Special Education, 44, 18-28. http://dx.doi.org/10.1177/0022466908328009
73. Wong, C., Odom, S. L., Hume, K. A., Cox, A. W., Fettig, A., Kucharczyk, S.,. Schultz, T. R. (2015). Evidence-based practices for children, youth, and young adults with autism spectrum disorder: A comprehensive review. Journal of Autism and Developmental Disorders, 45, 1951-1966. http://dx.doi.org/10.1007/s10803-014-2351-z