Analysis of single-case data: Randomisation tests for measures of effect size

Neuropsychological Rehabilitation

Volumn 24, Issue 3-4, 2014, Pages 507-527

  Heyvaert, Mieke ^a   Onghena, Patrick ^a  

^a UNIVERSITY OF LEUVEN   (Belgium)

Author keywords

Nonparametric statistical significance test; Randomisation test; Single case experimental design; Single subject experimental design; Statistical software

Indexed keywords

COMPUTER GRAPHICS; HUMAN; METHODOLOGY; PROCEDURES; RANDOMIZATION; STATISTICS; STATISTICS AND NUMERICAL DATA;

COMPUTER GRAPHICS; HUMANS; RANDOM ALLOCATION; RESEARCH DESIGN; STATISTICS AS TOPIC;

EID: 84903139273     PISSN: 09602011     EISSN: 14640694     Source Type: Journal    
DOI: 10.1080/09602011.2013.818564     Document Type: Article

References (84)

1. Allison, D. B., & Gorman, B. S. (1993). Calculating effect sizes for meta-analysis: The case of the single case. Behaviour Research and Therapy, 31, 621-631. doi:10.1016/0005- 7967(93)90115-B.
2. American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: Author.
3. Barlow, D. H., Nock, M. K., & Hersen, M. (2009). Single case experimental designs: Strategies for studying behavior change (3rd ed.). Boston, MA: Allyn & Bacon.
4. Borckardt, J. J., Murphy, M. D., Nash, M. R., & Shaw, D. (2004). An empirical examination of visual analysis procedures for clinical practice evaluation. Journal of Social Service Research, 30, 55-73. doi:10.1300/ J079v30n03-04.
5. Brossart, D. F., Parker, R. I., Olson, E. A., & Mahadevan, L. (2006). The relationship between visual analysis and five statistical analyses in a simple AB single-case research design. Behavior Modification, 30, 531-563. doi:10.1177/0145445503261167.
6. Bulté, I., & Onghena, P. (2008). An R package for single-case randomization tests. Behavior Research Methods, 40, 467-478. doi:10.3758/BRM.40.2.467.
7. Bulté, I., & Onghena, P. (2009). Randomization tests for multiple baseline designs: An extension of the SCRT-R package. Behavior Research Methods, 41, 477-485. doi:10.3758/BRM.41.2.477.
8. Bulté, I., & Onghena, P. (2012). When the truth hits you between the eyes. A software tool for the visual analysis of single-case experimental data. Methodology, 8, 104-114. doi:10.1027/1614-2241/a000042.
9. Bulté, I., Van den Noortgate, W., & Onghena, P. (2010, May). An R package for the nonparametric meta-analysis of small-n educational studies. Poster presented at the Annual Meeting of the American Educational Research Association (AERA), Denver, CO.
10. Busk, P. L., & Serlin, R. C. (1992). Meta-analysis for single-case research. In T. R. Kratochwill & J. R. Levin (Eds.), Single-case research design and analysis: New directions for psychology and education (pp. 187-212). Hillsdale, NJ: Lawrence Erlbaum.
11. Campbell, J. M. (2003). Efficacy of behavioural interventions for reducing problem behaviour in persons with autism: A quantitative synthesis of single-subject research. Research in Developmental Disabilities, 24, 120-138. doi:10.1016/S0891-4222(03)00014-3.
12. Campbell, J. M. (2004). Statistical comparison of four effect sizes for single-subject designs. Behavior Modification, 28, 234-246. doi:10.1177/ 0145445503259264.
13. Campbell, J. M. (2013). Commentary on PND at 25. Remedial and Special Education, 34, 20-25. doi:10.1177/0741932512454725.
14. Center, B. J., Skiba, R. J., & Casey, A. (1985-1986). A methodology for the quantitative synthesis of intra-subject design research. Journal of Special Education, 19, 387-400. doi:10.1177/002246698501900404.
15. Christ, T. J. (2007). Experimental control and threats to internal validity of concurrent and nonconcurrent multiple baseline designs. Psychology in the Schools, 44, 451-459. doi:10.1002/pits.20237.
16. Crosbie, J. (1993). Interrupted time-series analysis with brief single-subject data. Journal of Consulting and Clinical Psychology, 61, 966-974. doi:10.1037/0022-006X.61.6.966.
17. DeProspero, A., & Cohen, S. (1979). Inconsistent visual analysis of intrasubject data. Journal of Applied Behavior Analysis, 12, 285-296. doi:10.1901/jaba.1979.12-573.
18. Dugard, P., File, P., & Todman, J. (2012). Single-case and small-n experimental designs: A practical guide to randomization tests (2nd ed.). New York, NY: Routledge Academic.
19. Edgington, E. S. (1967). Statistical inference from N=1 experiments. Journal of Psychology, 65, 195-199. doi:10.1080/00223980.1967.10544864.
20. Edgington, E. S. (1972). An additive method for combining probability values from independent experiments. Journal of Psychology, 80, 351-363. doi:10.1080/00223980.1972. 9924813.
21. Edgington, E. S. (1975). Randomization tests for one-subject operant experiments. Journal of Psychology, 90, 57-68. doi:10.1080/00223980.1975. 9923926.
22. Edgington, E. S. (1980). Overcoming obstacles to single-subject experimentation. Journal of Educational Statistics, 5, 261-267. doi:10.3102/10769986005003261.
23. Edgington, E. S. (1984). Statistics and single case analysis. In M. Hersen, R. M. Eisler, & P. M. Miller (Eds.), Progress in behavior modification: Vol. 16 (pp. 83-120). New York, NY: Raven Press.
24. Edgington, E., & Onghena, P. (2007). Randomization tests (4th ed.). Boca Raton, FL: Chapman & Hall/CRC.
25. Ernst, M. D. (2004). Permutation methods: A basis for exact inference. Statistical Science, 19, 676-685. doi:10.1214/088342304000000396.
26. Ferron, J., & Jones, P. K. (2006). Tests for the visual analysis of response-guided multiple-baseline data. Journal of Experimental Education, 75, 66-81. doi:10.3200/JEXE.75.1.66-81.
27. Ferron, J., & Onghena, P. (1996). The power of randomization tests for single-case phase designs. Journal of Experimental Education, 64, 231-239.
28. Ferron, J., & Sentovich, C. (2002). Statistical power of randomization tests used with multiplebaseline designs. Journal of Experimental Education, 70, 165-178. doi:10.1080/00220970209599504.
29. Ferron, J., & Ware, W. (1994). Using randomization tests with responsive single-case designs. Behaviour Research and Therapy, 32, 787-791. doi:10.1016/0005-7967(94)90037-X.
30. Ferron, J., & Ware, W. (1995). Analyzing single-case data: The power of randomization tests. Journal of Experimental Education, 63, 167-178. doi:10.1080/00220973.1995.9943820.
31. Gage, N. A., & Lewis, T. J. (2013). Analysis of effect for single-case design research. Journal of Applied Sport Psychology, 25, 46-60. doi:10.1080/10413200.2012.660673.
32. Harbst, K. B., Ottenbacher, K. J., & Harris, S. R. (1991). Interrater reliability of therapists' judgments of graphed data. Physical Therapy, 71, 107-115.
33. Hartmann, D. P., Gottman, J. H., Jones, R. R., Gardner, W., Kazdin, A. E., & Vaught, R. S. (1980). Interrupted time-series analysis and its application to behavioral data. Journal of Applied Behavior Analysis, 13, 543-559. doi:10.1901/jaba.1980.13-543.
34. Hedges, L. V., Cooper, H., & Bushman, B. J. (1992). Testing the null hypothesis in meta-analysis: A comparison of combined probability and confidence interval procedures. Psychological Bulletin, 111, 188-194. doi:10.1037/0033- 2909.111.1.188.
35. Hedges, L. V., Pustejovsky, J. E., & Shadish, W. R. (2012). A standardized mean difference effect size for single case designs. Research Synthesis Methods, 3, 224-239. doi:10.1002/jrsm.1052.
36. Horowitz, M. J. (1982). Stress response syndromes and their treatment. In L. Goldberger & S. Breznitz (Eds.), Handbook of stress: Theoretical and clinical aspects (pp. 711-732). New York, NY: Free Press.
37. Horowitz, M., Wilner, N., & Alvarez, W. (1979). Impact of Event Scale: A measure of subjective stress. Psychosomatic Medicine, 41, 209-218.
38. Jones, R. R., Vaught, R. S., & Weinrott, M. (1977). Time series analysis in operant research. Journal of Applied Behavior Analysis, 10, 151-166. doi:10.1901/jaba.1977.10-151.
39. Joseph, S. (2000). Psychometric evaluation of Horowitz's impact of event scale: A review. Journal of Traumatic Stress, 13, 101-113. doi:10.1023/A: 1007777032063.
40. Kazdin, A. E. (1982). Single-case research designs: Methods for clinical and applied settings. New York, NY: Oxford University Press.
41. Kazdin, A. E. (1999). The meanings and measurement of clinical significance. Journal of Consulting and Clinical Psychology, 67, 332-339. doi:10.1037/0022-006X.67.3.332.
42. Kazdin, A. E. (2011). Single-case research designs: Methods for clinical and applied settings (2nd ed.). New York, NY: Oxford University Press.
43. Koehler, M. J., & Levin, J. R. (1998). Regulated randomization: A potentially sharper analytical tool for the multiple-baseline design. Psychological Methods, 3, 206-217. doi:10.1037/1082-989X.3.2.206.
44. Koehler, M. J., & Levin, J. R. (2000). RegRand: Statistical software for the multiple-baseline design. Behavior Research Methods, Instruments, & Computers, 32, 367-371. doi:10.3758/BF03207807.
45. Kratochwill, T. R., Hitchcock, J., Horner, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M., & Shadish, W. R. (2010). What Works Clearinghouse: Single-case designs technical documentation, Retrieved from http://ies.ed.gov/ ncee/wwc/pdf/wwc-scd.pdf.
46. Kratochwill, T. R., & Levin, J. R. (2010). Enhancing the scientific credibility of single-case intervention research: Randomization to the rescue. Psychological Methods, 15, 124-144. doi:10.1037/a0017736.
47. Levin, J. R., Ferron, J. M., & Kratochwill, T. R. (2012). Nonparametric statistical tests for single-case systematic and randomized ABAB. . .AB and alternating treatment intervention designs: New developments, new directions. Journal of School Psychology, 50, 599-624. doi:10.1016/j.jsp.2012. 05.001.
48. Levin, J. R., Lall, V. F., & Kratochwill, T. R. (2011). Extensions of a versatile randomization test for assessing single-case intervention effects. Journal of School Psychology, 49, 55-79. doi:10.1016/j.jsp.2010.09.002.
49. Levin, J. R., & Wampold, B. E. (1999). Generalized single-case randomization tests: Flexible analyses for a variety of situations. School Psychology Quarterly, 14, 59-93. doi:10.1037/h0088998.
50. Maggin, D. M., Swaminathan, H., Rogers, H. J., O'Keeffe, B. V., Sugai, G., & Horner, R. H. (2011). A generalized least squares regression approach for computing effect sizes in single-case research: Application examples. Journal of School Psychology, 49, 301-321. doi:10.1016/j.jsp.2011.03.004.
51. Manolov, R., Solanas, A., & Leiva, D. (2010). Comparing "visual" effect size indices for singlecase designs. Methodology, 6, 49-58. doi:10.1027/1614-2241/a000006.
52. Manolov, R., Solanas, A., Bulté, I., & Onghena, P. (2010). Data-division-specific robustness and power of randomization tests for ABAB designs. Journal of Experimental Education, 78, 191-214. doi:10.1080/ 00220970903292827.
53. Manolov, R., Solanas, A., Sierra, V., & Evans, J. J. (2011). Choosing among techniques for quantifying single-case intervention effectiveness. Behavior Therapy, 42, 533-545. doi:10.1016/j.beth.2010.12.003.
54. Matyas, T. A., & Greenwood, K. M. (1990a). Visual analysis for single-case time series: Effects of variability, serial dependence, and magnitude of intervention effects. Journal of Applied Behavior Analysis, 23, 341-351. doi:10.1901/jaba.1990.23-341.
55. Matyas, T. A., & Greenwood, K. M. (1990b). The effect of serial dependence on visual judgment of single-case charts: An addendum. American Journal of Occupational Therapy, 10, 308-320.
56. McKerracher, G., Powell, T., & Oyebode, J. (2005). A single case experimental design comparing two memory notebook formats for a man with memory problems caused by traumatic brain injury. Neuropsychological Rehabilitation, 15, 115-128. doi:10.1080/09602010443000056.
57. Nourbakhshs, M. R., & Ottenbacher, K. J. (1994). The statistical analysis of single-subject data: A comparative examination. Physical Therapy, 74, 768-776.
58. Onghena, P. (1992). Randomization tests for extensions and variations of ABAB single-case experimental designs: A rejoinder. Behavioral Assessment, 14, 153-171.
59. Onghena, P. (2005). Single-case designs. In B. Everitt & D. Howell (Eds.), Encyclopedia of statistics in behavioral science: Vol. 4 (pp. 1850-1854). New York, NY: Wiley.
60. Onghena, P., & Edgington, E. S. (1994). Randomization tests for restricted alternating treatments designs. Behaviour Research and Therapy, 32, 783-786.
61. Onghena, P., & Edgington, E. S. (2005). Customization of pain treatments: Single-case design and analysis. Clinical Journal of Pain, 21, 56-68. doi:10.1097/00002508-200501000- 00007.
62. Ottenbacher, K. J. (1990). When is a picture worth a thousand P-values? A comparison of visual and quantitative methods to analyze single subject data. Journal of Special Education, 23, 436-449. doi:10.1177/002246699002300407.
63. Park, H., Marascuilo, L., & Gaylord-Ross, R. (1990). Visual inspection and statistical analysis of single-case designs. Journal of Experimental Education, 58, 311-320.
64. Parker, R. I., & Brossart, D. F. (2003). Evaluating single-case research data: A comparison of seven statistical methods. Behavior Therapy, 34, 189-211. doi:10.1016/S0005- 7894(03)80013-8.
65. Parker, R. I., & Hagan-Burke, S. (2007). Useful effect size interpretations for single case research. Behavior Therapy, 38, 95-105. doi:10.1016/j.beth.2006.05.002.
66. Parker, R. I., Vannest, K. J., & Davis, J. L. (2011). Effect size in single-case research: A review of nine nonoverlap techniques. Behavior Modification, 35, 303-322. doi:10.1177/0145445511399147.
67. Perdices, M., & Tate, R. L. (2009). Single-subject designs as a tool for evidence-based clinical practice: Are they unrecognised and undervalued? Neuropsychological Rehabilitation, 19, 904-927. doi:10.1080/09602010903040691.
68. Pesarin, F. (2001). Multivariate permutation tests. New York, NY: Wiley.
69. Pesarin, F., & Salmaso, L. (2010). Permutation tests for complex data: Theory, applications and software. Chichester, UK: Wiley.
70. Rasquin, S. M. C., van de Sande, P., Praamstra, A. J., & Van Heugten, C. M. (2009). Cognitivebehavioural intervention for depression after stroke: Five single case studies on effects and feasibility. Neuropsychological Rehabilitation, 19, 208-222. doi:10.1080/09602010802.
71. Rosenthal, R. (1978). Combining results of independent studies. Psychological Bulletin, 85, 185-193. doi:10.1037//0033-2909.85.1.185.
72. Samuel, C., Louis-Dreyfus, A., Kaschel, R., Makiela, E., Troubat, M., Anselmi, N., . . . Azouvi, P. (2000). Rehabilitation of very severe unilateral neglect by visuo-spatio-motor cueing: Two single case studies. Neuropsychological Rehabilitation, 10, 385-399.
73. Schulz, R., O'Brien, A., Czaja, S., Ory, M., Norris, R., Martire, L. M., . . . Stevens, A. (2002). Dementia caregiver intervention research: In search of clinical significance. Gerontologist, 42, 589-602.
74. Scruggs, T. E., & Mastropieri, M. A. (2013). PND at 25: Past, present, and future trends in summarizing single-subject research. Remedial and Special Education, 34, 9-19. doi:10.1177/0741932512440730.
75. Scruggs, T. E., Mastropieri, M. A., & Casto, G. (1987). The quantitative synthesis of singlesubject research. Methodology and validation. Remedial and Special Education, 8, 24-33. doi:10.1177/074193258700800206.
76. Scruggs, T. E., Mastropieri, M. A., Cook, S. B., & Escobar, C. (1986). Early interventions for children with conduct disorders: A quantitative synthesis of single-subject research. Behavioral Disorders, 11, 260-271.
77. Shadish, W. R., Rindskopf, D. M., & Hedges, L. V. (2008). The state of the science in the metaanalysis of single-case experimental designs. Evidence-Based Communication Assessment and Intervention, 2, 188-196. doi:10.1080/17489530802581603.
78. Shakespeare, T. P., Gebski, V. J., Veness, M. J., & Simes, J. (2001). Improving interpretation of clinical studies by use of confidence levels, clinical significance curves, and risk-benefit contours. Lancet, 357, 1349-1353. doi:10.1016/S0140-6736(00)04522-0.
79. Task Force on Evidence-Based Interventions in School Psychology. (2003). Procedural and coding manual for review of evidence-based interventions, Retrieved from http://www.indiana.edu/~ebi/documents/-workingfiles/EBImanual1. pdf.
80. Todman, J. B., & Dugard, P. (2001). Single-case and small-n experimental designs: A practical guide to randomization tests. Mahwah, NJ: Erlbaum.
81. Van den Noortgate, W., & Onghena, P. (2003a). Combining single-case experimental data using hierarchical linear models. School Psychology Quarterly, 18, 325-346. doi:10.1521/scpq.18.3.325.22577.
82. Van den Noortgate, W., & Onghena, P. (2003b). Hierarchical linear models for the quantitative integration of effects sizes in single-case research. Behavior Research Methods, Instruments, & Computers, 35, 1-10. doi:10.3758/BF03195492.
83. Van den Noortgate, W., & Onghena, P. (2008). A multilevel meta-analysis of single-subject experimental design studies. Evidence-based Communication Assessment and Intervention, 2, 142-151. doi:10.1080/ 17489530802505362.
84. Wolery, M., Busick, M., Reichow, B., & Barton, E. E. (2010). Comparison of overlap methods for quantitatively synthesizing single-subject data. Journal of Special Education, 44, 18-29. doi:10.1177/0022466908328009.