Randomization tests for single-case experiments: State of the art, state of the science, and state of the application

Journal of Contextual Behavioral Science

Volumn 3, Issue 1, 2014, Pages 51-64

  Heyvaert, Mieke ^a,b   Onghena, Patrick ^a  

^a UNIVERSITY OF LEUVEN   (Belgium)

^b RESEARCH FOUNDATION FLANDERS FWO   (Belgium)

Author keywords

Alternation design; Phase design; Randomization test; Single subject experimental design; Statistical software

Indexed keywords

EID: 84894664039     PISSN: 22121447     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jcbs.2013.10.002     Document Type: Article

References (111)

1. Allison D.B., Gorman B.S. Calculating effect sizes for meta-analysis: The case of the single case. Behaviour Research and Therapy 1993, 31:621-631. 10.1016/0005-7967(93)90115-B.
2. Altman D.G., Schulz K.F., Moher D., Egger M., Davidoff F., Elbourne D., et al. The revised CONSORT statement for reporting randomized trials: Explanation and elaboration. Annals of Internal Medicine 2001, 134:663-694.
3. American Psychological Association Publication manual of the American Psychological Association 2010, American Psychological Association, Washington, DC. 6th ed.
4. Barlow D.H., Hayes S.C. Alternating treatments design: One strategy for comparing the effects of two treatments in a single subject. Journal of Applied Behavior Analysis 1979, 12:199-210. 10.1901/jaba.1979.12-199.
5. Barlow D.H., Hayes S.C., Nelson R.O. The scientist practitioner: research and accountability in clinical and educational settings 1985, Pergamon, Oxford, UK.
6. Barlow D.H., Nock M.K., Hersen M. Single case experimental designs: Strategies for studying behavior change 2009, Allyn & Bacon, Boston, MA. 3rd ed.
7. Beretvas S.N., Chung H. A review of meta-analyses of single-subject experimental designs: Methodological issues and practice. Evidence-Based Communication Assessment and Intervention 2008, 2:129-141. 10.1080/17489530802446302.
8. Blampied N.M. A legacy neglected: Restating the case for single-case research in cognitive-behaviour therapy. Behaviour Change 1999, 16:89-104. 10.1375/bech.16.2.89.
9. Borckardt J.J., Murphy M.D., Nash M.R., Shaw D. An empirical examination of visual analysis procedures for clinical practice evaluation. Journal of Social Service Research 2004, 30:55-73. 10.1300/J079v30n03_04.
10. Box G.E.P., Jenkins G.M. Time series analysis: forecasting and control 1970, Holden Day, San Francisco, CA.
11. Bulté I., Onghena P. An R package for single-case randomization tests. Behavior Research Methods 2008, 40:467-478. 10.3758/BRM.40.2.467.
12. Bulté I., Onghena P. Randomization tests for multiple baseline designs: An extension of the SCRT-R package. Behavior Research Methods 2009, 41:477-485. 10.3758/BRM.41.2.477.
13. Bulté I., Onghena P. When the truth hits you between the eyes. A software tool for the visual analysis of single-case experimental data. Methodology 2012, 8:104-114. 10.1027/1614-2241/a000042.
14. Bulté, I., Van Den Noortgate, W., & Onghena, P. (May 2010). An R package for the nonparametric meta-analysis of small-n educational studies. Poster presented at the annual meeting of the American Educational Research Association (AERA). Denver, Colorado (USA).
15. Busk P.L., Marascuilo L.A. Statistical analysis in single-case research: Issues, procedures, and recommendations, with applications to multiple behaviors. Single-case research design and analysis: new directions for psychology and education 1992, 159-185. Lawrence Erlbaum, Hillsdale, NJ. T.R. Kratochwill, J.R. Levin (Eds.).
16. Busk P.L., Serlin R.C. Meta-analysis for single-case research. Single-case research design and analysis: new directions for psychology and education 1992, 187-212. Lawrence Erlbaum, Hillsdale, NJ. T.R. Kratochwill, J.R. Levin (Eds.).
17. Busse R.T., Kratochwill T.R., Elliott S.N. Meta-analysis for single-case consultation outcomes: Applications to research and practice. Journal of School Psychology 1995, 33:269-285. 10.1016/0022-4405(95)00014-D.
18. Center B.A., Skiba R.J., Casey A. A methodology for the quantitative synthesis of intra-subject design research. Journal of Special Education 1985-1986, 19:387-400. 10.1177/002246698501900404.
19. Christ T.J. Experimental control and threats to internal validity of concurrent and nonconcurrent multiple baseline designs. Psychology in the Schools 2007, 44:451-459. 10.1002/pits.20237.
20. DeProspero A., Cohen S. Inconsistent visual analysis of intrasubject data. Journal of Applied Behavior Analysis 1979, 12:285-296. 10.1901/jaba.1979.12-573.
21. Dugard P., File P., Todman J. Single-case and small-n experimental designs: a practical guide to randomization tests 2012, Routledge Academic, New York, NY. 2nd ed.
22. Edelson M. Hypothesis and evidence in psychoanalysis 1985, University of Chicago Press, Chicago, IL.
23. Edgington E., Onghena P. Randomization tests 2007, Chapman & Hall/CRC, Boca Raton, FL. 4th ed.
24. Edgington E.S. Statistical inference from N=1 experiments. Journal of Psychology 1967, 65:195-199.
25. Edgington E.S. An additive method for combining probability values from independent experiments. Journal of Psychology 1972, 80:351-363. 10.1080/00223980.1972.9924813.
26. Edgington E.S. A normal curve method for combining probability values from independent experiments. Journal of Psychology 1972, 82:85-89. 10.1080/00223980.1972.9916971.
27. Edgington E.S. Randomization tests for one-subject operant experiments. Journal of Psychology 1975, 90:57-68.
28. Edgington E.S. Overcoming obstacles to single-subject experimentation. Journal of Educational Statistics 1980, 5:261-267.
29. Edgington E.S. Randomized single-subject experiments and statistical tests. Journal of Counseling Psychology 1987, 34:437-442. 10.1037/0022-0167.34.4.437.
30. Edgington E.S. Randomized single-subject experimental designs. Behaviour Research and Therapy 1996, 34:567-574. 10.1016/0005-7967(96)00012-5.
31. Ferron J., Foster-Johnson L., Kromrey J. The functioning of single-case randomization tests with and without random assignment. Journal of Experimental Education 2003, 71:267-288. 10.1080/00220970309602066.
32. Ferron J., Jones P.K. Tests for the visual analysis of response-guided multiple-baseline data. Journal of Experimental Education 2006, 75:66-81. 10.3200/JEXE.75.1.66-81.
33. Ferron J., Onghena P. The power of randomization tests for single-case phase designs. Journal of Experimental Education 1996, 64:231-239.
34. Ferron J., Sentovich C. Statistical power of randomization tests used with multiple-baseline designs. Journal of Experimental Education 2002, 70:165-178. 10.1080/00220970209599504.
35. Ferron J., Ware W. Analyzing single-case data: The power of randomization tests. Journal of Experimental Education 1995, 63:167-178. 10.1080/00220973.1995.9943820.
36. Design and analysis of single-case research. 1996, Lawrence Erlbaum, Mahwah, NJ. R.D. Franklin, D.B. Allison, B.S. Gorman (Eds.).
37. Gage N.A., Lewis T.J. Analysis of effect for single-case design research. Journal of Applied Sport Psychology 2013, 25:46-60. 10.1080/10413200.2012.660673.
38. Gentile J.R., Roden A.H., Klein R.D. An analysis-of-variance model for the intrasubject replication design. Journal of Applied Behavior Analysis 1972, 5:193-198. 10.1901/jaba.1972.5-193.
39. Glass G.V., Willson V.L., Gottman J.M. Design and analysis of time-series experiments 1975, Colorado Associated University Press, Boulder, CO.
40. Gottman J.M., Glass G.V. Analysis of interrupted time series analysis. Single Subject Research 1978, Academic Press, New York, NY. T.R. Kratochwill (Ed.).
41. Greenhalgh T. Papers that summarise other papers (systematic reviews and meta-analyses). British Medical Journal 1997, 315:672-675. 10.1136/bmj.315.7109.672.
42. Hacker B. Single subject research strategies in occupational therapy, Part 1. American Journal of Occupational Therapy 1980, 34:103-108.
43. Harbst K.B., Ottenbacher K.J., Harris S.R. Interrater reliability of therapists' judgments of graphed data. Physical Therapy 1991, 71:107-115.
44. Hartmann D.P., Gottman J.H., Jones R.R., Gardner W., Kazdin A.E., Vaught R.S. Interrupted time-series analysis and its application to behavioral data. Journal of Applied Behavior Analysis 1980, 13:543-559. 10.1901/jaba.1980.13-543.
45. Hayes S.C. Single case experimental design and empirical clinical practice. Journal of Consulting and Clinical Psychology 1981, 49:193-211. 10.1037/0022-006X.49.2.193.
46. Hayes S.C., Barnes-Holmes D., Wilson K.G. Contextual behavioral science: Creating a science more adequate to the challenge of the human condition. Journal of Contextual Behavioral Science 2012, 1:1-16.
47. Hedges L.V., Pustejovsky J.E., Shadish W.R. A standardized mean difference effect size for single case designs. Research Synthesis Methods 2012, 3:224-239. 10.1002/jrsm.1052.
48. Heyvaert M., Maes B., Van den Noortgate W., Kuppens S., Onghena P. A multilevel meta-analysis of single-case and small-n research on interventions for reducing challenging behavior in persons with intellectual disabilities. Research in Developmental Disabilities 2012, 33:766-780. 10.1016/j.ridd.2011.10.010.
49. Heyvaert M., Onghena P. Analysis of single-case data: Randomization tests for measures of effect size. Neuropsychological Rehabilitation 2013, 10.1080/09602011.2013.818564.
50. Horner R.H., Carr E.G., Halle J., McGee G., Odom S., Wolery M. The use of single-subject research to identify evidence-based practice in special education. Exceptional Children 2005, 71:165-179.
51. Houle T.T. Statistical analyses for single-case experimental designs. Single case experimental designs: strategies for studying behavior change 2009, 271-305. Allyn & Bacon, Boston, MA. 3rd ed. D.H. Barlow, M.K. Nock, M. Hersen (Eds.).
52. Ittenbach R.F., Lawhead W.F. Historical and philosophical foundations of single-case research. Design and analysis of single-case research 1996, 13-39. Lawrence Erlbaum, Mahwah, NJ. R.D. Franklin, D.B. Allison, B.S. Gorman (Eds.).
53. Jacobson N.S., Follette W.C., Revenstorf D. Psychotherapy outcome research: Methods for reporting variability and evaluating clinical significance. Behaviour Therapy 1984, 15:336-352. 10.1016/S0005-7894(84)80002-7.
54. Jones R.R., Vaught R.S., Reid J.B. Time series analysis as a substitute for single subject analysis of variance designs. Behavior change 1975, Aldine, Chicago, IL. G.R. Patterson, I.M. Mark, J.D. Matarazzo, R.A. Myuers, G.E. Schwartz, H.H. Strapp (Eds.).
55. Jones R.R., Vaught R.S., Weinrott M. Time series analysis in operant research. Journal of Applied Behavior Analysis 1977, 10:151-166. 10.1901/jaba.1977.10-151.
56. Kazdin A.E. Single-case research designs: methods for clinical and applied settings 1982, Oxford University Press, New York, NY.
57. Kazdin A.E. The meanings and measurement of clinical significance. Journal of Consulting and Clinical Psychology 1999, 67:332-339. 10.1037/0022-006X.67.3.332.
58. Kazdin A.E. Single-case research designs: methods for clinical and applied settings 2011, Oxford University Press, New York, NY. 2nd ed.
59. Kennedy M.M. Generalizing from single case studies. Evaluation Review 1979, 3:661-678. 10.1177/0193841×7900300409.
60. Koehler J., Levin J. Regulated randomization: A potentially sharper analytical tool for the multiple-baseline design. Psychological Methods 1998, 2:206-217. 10.1037/1082-989X.3.2.206.
61. Koehler J., Levin J. RegRand: Statistical software for the multiple-baseline design. Behavior Research Methods, Instruments, and Computers 2000, 32:367-371. 10.3758/BF03207807.
62. Kratochwill T.R., Brody G.H. Single subject designs: A perspective on the controversy over employing statistical inference and implications for research and training in behavior modification. Behavior Modification 1978, 2:291-307. 10.1177/014544557823001.
63. Kratochwill, T. R., Hitchcock, J., Horner, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M., & et al. (2010). Single-case designs technical documentation. Available from. http://ies.ed.gov/ncee/wwc/pdf/wwc_scd.pdf.
64. Single-case research design and analysis: New directions for psychology and education 1992, Lawrence Erlbaum, Hillsdale, NJ. T.R. Kratochwill, J.R. Levin (Eds.).
65. Kratochwill T.R., Levin J.R. Enhancing the scientific credibility of single-case intervention research: Randomization to the rescue. Psychological Methods 2010, 15:124-144. 10.1037/a0017736.
66. Kratochwill T.R., Mace F.C. Time-series designs in psychotherapy research. Issues in psychotherapy research 1984, 171-226. Plenum Press, New York, NY. M. Hersen, L. Michelson, A. Bellack (Eds.).
67. Kruskal W.H., Wallis W.A. Use of ranks in one-criterion variance analysis. Journal of the American Statistical Association 1952, 47:583-621. 10.1080/01621459.1952.10483441.
68. Levin J.R., Ferron J.M., Kratochwill T.R. Nonparametric statistical tests for single-case systematic and randomized ABAB. . .AB and alternating treatment intervention designs: New developments, new directions. Journal of School Psychology 2012, 50:599-624. 10.1016/j.jsp.2012.05.001.
69. Levin J.R., Lall V.F., Kratochwill T.R. Extensions of a versatile randomization test for assessing single-case intervention effects. Journal of School Psychology 2011, 49:55-79. 10.1016/j.jsp.2010.09.002.
70. Lorimer P.A., Simpson R.L., Myles B.S., Ganz J.B. The use of social stories as a preventative behavioral intervention in a home setting with a child with autism. Journal of Positive Behavior Interventions 2002, 4:53-60. 10.1177/109830070200400109.
71. Maggin D.M., O'Keeffe B.V., Johnson A.H. A quantitative synthesis of methodology in the meta-analysis of single-subject research for students with disabilities: 1985-2009. Exceptionality 2011, 19:109-135. 10.1080/09362835.2011.565725.
72. Maggin D.M., Swaminathan H., Rogers H.J., O'Keeffe B.V., Sugai G., Horner R.H. A generalized least squares regression approach for computing effect sizes in single-case research: Application examples. Journal of School Psychology 2011, 49:301-321. 10.1016/j.jsp.2011.03.004.
73. Maher C.G., Sherrington C., Herbert R.D., Moseley A.M., Elkins M. Reliability of the PEDro scale for rating quality of randomized controlled trials. Physical Therapy 2003, 83:713-721.
74. Mann H.B., Whitney D.R. On a test of whether one of two random variables is stochastically larger than the other. Annals of Mathematical Statistics 1947, 18:50-60. 10.1214/aoms/1177730491.
75. Manolov R., Solanas A., Bulté I., Onghena P. Data-division-specific robustness and power of randomization tests for ABAB designs. Journal of Experimental Education 2010, 78:191-214. 10.1080/00220970903292827.
76. Manolov R., Solanas A., Sierra V., Evans J.J. Choosing among techniques for quantifying single-case intervention effectiveness. Behavior Therapy 2011, 42:533-545.
77. Marascuilo L., Busk P. Combining statistics for multiple-baseline AB and replicated ABAB designs across subjects. Behavioral Assessment 1988, 10:1-28.
78. Matyas T.A., Greenwood K.M. Visual analysis for single-case time series: Effects of variability, serial dependence, and magnitude of intervention effects. Journal of Applied Behavior Analysis 1990, 23:341-351. 10.1901/jaba.1990.23-341.
79. Matyas T.A., Greenwood K.M. The effect of serial dependence on visual judgment of single-case charts: An addendum. American Journal of Occupational Therapy 1990, 10:308-320.
80. McReynolds L.V., Thompson C.K. Flexibility of single-subject experimental designs. Part I: Review of the basics of single-subject designs. Journal of Speech and Hearing Disorders 1986, 51:194-203.
81. Nourbakhshs M.R., Ottenbacher K.J. The statistical analysis of single-subject data: A comparative examination. Physical Therapy 1994, 74:768-776.
82. Onghena P. Randomization tests for extensions and variations of ABAB single-case experimental designs: A rejoinder. Behavioral Assessment 1992, 14:153-171.
83. Onghena P. Single-case designs. Encyclopedia of statistics in behavioral science 2005, Vol. 4:1850-1854. Wiley, New York, NY.
84. Onghena P., Edgington E.S. Randomization tests for restricted alternating treatments designs. Behaviour Research and Therapy 1994, 32:783-786.
85. Onghena P., Edgington E.S. Customization of pain treatments: Single-case design and analysis. Clinical Journal of Pain 2005, 21:56-68. 10.1097/00002508-200501000-00007.
86. Ottenbacher K.J. When is a picture worth a thousand p values? A comparison of visual and quantitative methods to analyze single subject data. Journal of Special Education 1990, 23:436-449. 10.1177/002246699002300407.
87. Park H., Marascuilo L., Gaylord-Ross R. Visual inspection and statistical analysis of single-case designs. Journal of Experimental Education 1990, 58:311-320.
88. Parker R.I., Hagan-Burke S. Useful effect size interpretations for single case research. Behavior Therapy 2007, 38:95-105. 10.1016/j.beth.2006.05.002.
89. Parker R.I., Vannest K.J., Brown L. The improvement rate difference for single-case research. Exceptional Children 2009, 75:135-150.
90. Parker R.I., Vannest K.J., Davis J.L. Effect size in single-case research: A review of nine nonoverlap techniques. Behavior Modification 2011, 35:303-322. 10.1177/0145445511399147.
91. Perdices M., Tate R.L. Single-subject designs as a tool for evidence-based clinical practice: Are they unrecognised and undervalued?. Neuropsychological Rehabilitation 2009, 19:904-927. 10.1080/09602010903040691.
92. Pesarin F. Multivariate permutation tests 2001, Wiley, New York, NY.
93. Pesarin F., Salmaso L. Permutation tests for complex data: theory, applications and software 2010, Wiley, Chichester, UK.
94. Rapoff M., Stark L. Editorial: Journal of Pediatric Psychology-Statement of purpose: Section on single-subject studies. Journal of Pediatric Psychology 2008, 33:16-21. 10.1093/jpepsy/jsm101.
95. Romeiser-Logan R.L., Hickman R.R., Harris S.R., Heriza C.B. Single-subject research design: Recommendations for levels of evidence and quality rating. Developmental Medicine & Child Neurology 2008, 50:99-103. 10.1111/j.1469-8749.2007.02005.x.
96. Rosenthal R. Combining results of independent studies. Psychological Bulletin 1978, 85:185-193. 10.1037//0033-2909.85.1.185.
97. Schulz R., O'Brien A., Czaja S., Ory M., Norris R., Martire L.M., et al. Dementia caregiver intervention research: In search of clinical significance. Gerontologist 2002, 42:589-602.
98. Shadish W.R., Rindskopf D.M. Methods for evidence-based practice: Quantitative synthesis of single-subject designs. New Directions for Evaluation 2007, 113:95-109. 10.1002/ev.217.
99. Shakespeare T.P., Gebski V.J., Veness M.J., Simes J. Improving interpretation of clinical studies by use of confidence levels, clinical significance curves, and risk-benefit contours. Lancet 2001, 357:1349-1353. 10.1016/S0140-6736(00)04522-0.
100. Smith J.D. Single-case experimental designs: A systematic review of published research and current standards. Psychological Methods 2012, 17:510-550. 10.1037/a0029312μ.
101. Task Force on Evidence-Based Interventions in School Psychology (2003). Procedural and coding manual for review of evidence-based interventions. Available from. http://www.indiana.edu/~ebi/documents/_workingfiles/EBImanual1.pdf.
102. Van den Noortgate W., Onghena P. Combining single-case experimental data using hierarchical linear models. School Psychology Quarterly 2003, 18:325-346. 10.1521/scpq.18.3.325.22577.
103. Van den Noortgate W., Onghena P. Hierarchical linear models for the quantitative integration of effects sizes in single-case research. Behavior Research Methods, Instruments, and Computers 2003, 35:1-10. 10.3758/BF03195492.
104. Van den Noortgate W., Onghena P. The aggregation of single-case results using hierarchical linear models. The Behavior Analyst Today 2007, 8:196-209. http://www.behavior-analyst-today.net/index.html.
105. Van den Noortgate W., Onghena P. A multilevel meta-analysis of single-subject experimental design studies. Evidence-based Communication Assessment and Intervention 2008, 2:142-151. 10.1080/17489530802505362.
106. Van Laarhoven T., Kraus E., Karpman K., Nizzi R., Valentino J. A comparison of picture and video prompts to teach daily living skills to individuals with autism. Focus on Autism and Other Developmental Disabilities 2010, 25:195-208. 10.1177/1088357610380412.
107. Vilardaga R., Hayes S.C., Levin M.E., Muto T. Creating a strategy for progress: A contextual behavioral science approach. Behavior Analyst 2009, 32:105-133.
108. Wampold B., Worsham N. Randomization tests for multiple-baseline designs. Behavioral Assessment 1986, 8:135-143.
109. White D.M., Rusch F.R., Kazdin A.E., Hartmann D.P. Applications of meta analysis in individual-subject research. Behavioral Assessment 1989, 11:281-296.
110. Wolery M., Busick M., Reichow B., Barton E.E. Comparison of overlap methods for quantitatively synthesizing single-subject data. Journal of Special Education 2010, 44:18-29. 10.1177/0022466908328009.
111. Zhan S., Ottenbacher K.J. Single subject research designs for disability research. Disability and Rehabilitation 2001, 23:1-8.