Interrater Agreement Between Visual Analysts of Single-Case Data: A Meta-Analysis

Behavior Modification

Volumn 39, Issue 4, 2015, Pages 510-541

  Ninci, Jennifer ^a   Vannest, Kimberly J ^a   Willson, Victor ^a   Zhang, Nan ^a  

^a TEXAS A AND M UNIVERSITY   (United States)

Author keywords

agreement; reliability; single case research; visual analysis

Indexed keywords

EPIDEMIOLOGY; HUMAN; META ANALYSIS; OBSERVER VARIATION; PATTERN RECOGNITION; REPRODUCIBILITY; STATISTICAL ANALYSIS;

DATA INTERPRETATION, STATISTICAL; EFFECT MODIFIER, EPIDEMIOLOGIC; HUMANS; OBSERVER VARIATION; PATTERN RECOGNITION, VISUAL; REPRODUCIBILITY OF RESULTS;

EID: 84930508970     PISSN: 01454455     EISSN: 15524167     Source Type: Journal    
DOI: 10.1177/0145445515581327     Document Type: Article

References (61)

1. Baer D. M.Perhaps it would be better not to know everything.Journal of Applied Behavior Analysis. 1977;10:167-172
2. *Bailey D. B.Effects of lines of progress and semilogarithmic charts on ratings of charted data.Journal of Applied Behavior Analysis. 1984;17:359-365
3. *Bobrovitz C. D.,Ottenbacher K. J.Comparison of visual inspection and statistical analysis of single-subject data in rehabilitation research.American Journal of Physical Medicine & Rehabilitation. 1998;77:94-102
4. Bonner M.,Barnett D. W.Intervention-based school psychology services: Training for child-level accountability; preparing for program-level accountability.Journal of School Psychology. 2004;42:23-43
5. Brossart D. F.,Parker R. I.,Olson E. A.,Mahadevan L.The relationship between visual analysis and five statistical analyses in a simple AB single-case research design.Behavior Modification. 2006;30:531-563
6. Burns M. K.Meta-analysis of single-case design research: Introduction to the special issue.Journal of Behavioral Education. 2012;21:175-184
7. Carter M.Reconsidering overlap-based measures for quantitative synthesis of single-subject data: What they tell us and what they don’t.Behavior Modification. 2013;37:378-390
8. *Danov S. E.,Symons F. J.A survey evaluation of the reliability of visual inspection and functional analysis graphs.Behavior Modification. 2008;32:828-839
9. DeProspero A.,Cohen S.Inconsistent visual analyses of intrasubject data.Journal of Applied Behavior Analysis. 1979;12:573-579
10. Fisch G. S.Evaluating data from behavioral analysis: Visual inspection or statistical models?.Behavioural Processes. 2001;54:137-154
11. Franklin R. D.,Gorman B. S.,Beasley T. M.,Allison D. B.Design and analysis of single-case research. Franklin R. D.Allison D. B.Gorman B. S., ed. Mahwah, NJ: Lawrence Erlbaum; 1996:119-158.
12. Furlong M. J.,Wampold B. E.Intervention effects and relative variation as dimensions in experts’ use of visual inference.Journal of Applied Behavior Analysis. 1982;15:415-421
13. Gage N. A.,Lewis T. J.Analysis of effect for single-case design research.Journal of Applied Sport Psychology. 2013;25:46-60
14. Gast D. L.Single subject research methodology in behavioral sciences. New York, NY: Routledge; 2010:
15. *Gibson G.,Ottenbacher K.Characteristics influencing the visual analysis of single-subject data: An empirical analysis.The Journal of Applied Behavioral Science. 1988;24:298-314
16. Grigg N. C.,Snell M. E.,Loyd B.Visual analysis of student evaluation data: A qualitative analysis of teacher decision making.Journal of the Association for Persons With Severe Handicaps. 1989;14:23-32
17. Hagopian L. P.,Fisher W. W.,Thompson R. H.,Owen-DeSchryver J.,Iwata B. A.,Wacker D. P.Toward the development of structured criteria for interpretation of functional analysis data.Journal of Applied Behavior Analysis. 1997;30:313-326
18. *Harbst K. B.,Ottenbacher K. J.,Harris S. R.Interrater reliability of therapists’ judgments of graphed data.Physical Therapy. 1991;71:107-115
19. Hartmann D. P.,Barrios B. A.,Wood D. D.Comprehensive handbook of psychological assessment, behavioral assessment. Haynes S. N.Hieby E. M., ed. New York, NY: John Wiley; 2004:108-127.
20. Higgins J. P. T.,Thompson S. G.Quantifying heterogeneity in a meta-analysis.Statistics in Medicine. 2002;21:1539-1558
21. Higgins J. P. T.,Thompson S. G.,Deeks J. J.,Altman D. G.Measuring inconsistency in meta-analyses.British Medical Journal. 2003;327:557-560
22. *Hojem M. A.,Ottenbacher K. J.Empirical investigation of visual-inspection versus trend-line analysis of single-subject data.Physical Therapy. 1988;68:983-988
23. Horner R. H.,Swaminathan H.,Sugai G.,Smolkowski K.Considerations for the systematic analysis and use of single-case research.Education & Treatment of Children. 2012;35:269-290
24. House A. E.,House B. J.,Campbell M. B.Measures of interobserver agreement: Calculation formulas and distribution effects.Journal of Behavioral Assessment. 1981;3:37-57
25. *James I. A.,Smith P. S.,Milne D.Teaching visual analysis of time series data.Behavioural and Cognitive Psychotherapy. 1996;24:247-262
26. *Johnson M. B.,Ottenbacher K. J.Trend line influence on visual analysis of single-subject data in rehabilitation research.Disability and Rehabilitation. 1991;13:55-59
27. *Jones R. R.,Weinrott M. R.,Vaught R. S.Effects of serial dependency on the agreement between visual and statistical inference.Journal of Applied Behavior Analysis. 1978;11:277-283
28. *Kahng S. W.,Chung K. M.,Gutshall K.,Pitts S. C.,Kao J.,Girolami K.Consistent visual analyses of intrasubject data.Journal of Applied Behavior Analysis. 2010;43:35-45
29. Kazdin A. E.Artifact, bias, and complexity of assessment: The ABCs of reliability.Journal of Applied Behavior Analysis. 1977;10:141-150
30. Kennedy C. H.Single-case designs for educational research. Boston, MA: Allyn & Bacon; 2005:
31. Knapp T. J.Behavior analysts’ visual appraisal of behavior change in graphic display.Behavioral Assessment. 1983;5:155-164
32. KratochwillT. R.HitchcockJ. H.HornerR. H.LevinJ. R.OdomS. L.RindskopfD. M.ShadishW. R. (2010). Single-case designs technical documentation. Retrieved from http://ies.ed.gov/ncee/wwc/pdf/wwc_scd.pdf
33. Kratochwill T. R.,Hitchcock J. H.,Horner R. H.,Levin J. R.,Odom S. L.,Rindskopf D. M.,Shadish W. R.Single-case intervention research design standards.Remedial and Special Education. 2013;34:26-38
34. Kratochwill T. R.,Levin J. R.Meta- and statistical analysis of single-case intervention research data: Quantitative gifts and a wish list.Journal of School Psychology. 2014;52:231-235
35. *Lieberman R. G.,Yoder P. J.,Reichow B.,Wolery M.Visual analysis of multiple baseline across participants graphs when change is delayed.School Psychology Quarterly. 2010;25:28-44
36. Lipsey M.,Wilson D.Practical meta-analysis. Thousand Oaks, CA: SAGE; 2001:
37. Maggin D. M.,Briesch A. M.,Chafouleas S. M.An application of the What Works Clearinghouse standards for evaluating single-subject research: Synthesis of the self-management literature base.Remedial and Special Education. 2013;34:44-58
38. Manolov R.,Sierra V.,Solanas A.,Botella J.Assessing functional relations in single-case designs: Quantitative proposals in the context of the evidence-based movement.Behavior Modification. 2014;38:878-913
39. Matyas T. A.,Greenwood K. M.The effect of serial dependence on visual judgment of single-case charts: An addendum.Occupational Therapy Journal of Research. 1990a;10:208-220
40. Matyas T. A.,Greenwood K. M.Visual analysis of single-case time series: Effects of variability, serial dependence, and magnitude of intervention effects.Journal of Applied Behavior Analysis. 1990b;23:341-351
41. Mercer S. H.,Sterling H. E.The impact of baseline trend control on visual analysis of single-case data.Journal of School Psychology. 2012;50:403-419
42. Ninci J.,Neely L. C.,Hong E. R.,Boles M. B.,Gilliland W. D.,Ganz J. B.,..Vannest K. J.Meta-analysis of single-case research on teaching functional living skills to individuals with ASD.Review Journal of Autism and Developmental Disorders. 2015;:
43. *Normand M. P.,Bailey J. S.The effects of celeration lines on visual data analysis.Behavior Modification. 2006;30:295-314
44. *Ottenbacher K. J.Reliability and accuracy of visually analyzing graphed data from single-subject designs.American Journal of Occupational Therapy. 1986;40:464-469
45. *Ottenbacher K. J.Visual inspection of single-subject data: An empirical analysis.Mental Retardation. 1990a;28:283-290
46. *Ottenbacher K. J.When is a picture worth a thousand p values? A comparison of visual and quantitative methods to analyze single subject data.Journal of Special Education. 1990b;23:436-449
47. Ottenbacher K. J.Interrater agreement of visual analysis in single-subject decisions: Quantitative review and analysis.American Journal on Mental Retardation. 1993;98:135-142
48. *Ottenbacher K. J.,Cusick A.An empirical investigation of interrater agreement for single-subject data using graphs with and without trend lines.Journal of the Association for Persons With Severe Handicaps. 1991;16:48-55
49. *Park H. S.,Marascuilo L.,Gaylord-Ross R.Visual inspection and statistical analysis in single-case designs.The Journal of Experimental Education. 1990;58:311-320
50. Parker R. I.,Vannest K. J.Bottom-up analysis of single-case research designs.Journal of Behavioral Education. 2012;21:254-265
51. Parsonson B. S.,Baer D. M.Single-case research design and analysis: New directions for psychology and education. Kratochwill T. R.Levin J. R., ed. Hillsdale, NJ: Lawrence Erlbaum; 1992:15-40.
52. Perone M.Statistical inference in behavior analysis: Experimental control is better.The Behavior Analyst. 1999;22:109-116
53. *Roane H. S.,Fisher W. W.,Kelley M. E.,Mevers J. L.,Bouxsein K. J.Using modified visual-inspection criteria to interpret functional analysis outcomes.Journal of Applied Behavior Analysis. 2013;46:130-146
54. Rojahn J.,Schulze H. H.The linear regression line as a judgmental aid in visual analysis of serially dependent A-B time-series data.Journal of Psychopathology and Behavioral Assessment. 1985;7:191-205
55. Sidman M.Tactics of scientific research: Evaluating experimental data in psychology. New York, NY: Basic Books; 1960:
56. *Skiba R.,Deno S.,Marston D.,Casey A.Influence of trend estimation and subject familiarity on practitioners’ judgments of intervention effectiveness.The Journal of Special Education. 1989;22:433-446
57. Smith J. D.Single-case experimental designs: A systematic review of published research and current standards.Psychological Methods. 2012;17:510-550
58. Spirrison C. L.,Mauney L. T.Acceptability bias: The effects of treatment acceptability on visual analysis of graphed data.Journal of Psychopathology and Behavioral Assessment. 1994;16:85-94
59. Vannest K. J.,Davis H. S.Advances in Learning and Behavioral Disabilities: Vol. 26. Evidence-based practices. Cook B. G.Tankersley M.Landrum T. J., ed. West Yorkshire. England: Emerald Group; 2013:93-119.
60. Wampold B. E.,Furlong M. J.The heuristics of visual inference.Behavioral Assessment. 1981;3:79-82
61. Ximenes V. M.,Manolov R.,Solanas A.,Quera V.Factors affecting visual analysis in single-case designs.Spanish Journal of Psychology. 2009;12:823-832