Reliability generalization for childhood autism rating scale

Journal of Autism and Developmental Disorders

Volumn 43, Issue 12, 2013, Pages 2855-2865

  Breidbord, Jonathan ^a   Croudace, Tim J ^b,c  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

^b UNIVERSITY OF YORK   (United Kingdom)

^c UNIVERSITY OF CAMBRIDGE   (United Kingdom)

Author keywords

Internal consistency; Meta analysis; Psychometrics; Questionnaires; Reliability generalisation; Reproducibility of results; Score stability

Indexed keywords

ARTICLE; ASPERGER SYNDROME; AUTISM; AUTOMUTILATION; CHILD; CHILDHOOD AUTISM RATING SCALE; CONTROLLED STUDY; DIAGNOSTIC VALUE; DISEASE SEVERITY; EMOTION; EMPIRICAL RESEARCH; ENVIRONMENTAL CHANGE; EXPLORATORY RESEARCH; FACTORIAL ANALYSIS; FEMALE; HUMAN; INTELLIGENCE QUOTIENT; INTERNAL CONSISTENCY; INTERRATER RELIABILITY; MAJOR CLINICAL STUDY; MALE; NAMED INVENTORIES, QUESTIONNAIRES AND RATING SCALES; NONVERBAL COMMUNICATION; PRIORITY JOURNAL; RELIABILITY; SCHOOL CHILD; SCORING SYSTEM; SYMPTOMATOLOGY; TEST RETEST RELIABILITY; VERBAL COMMUNICATION;

ADOLESCENT; AUTISTIC DISORDER; CHILD; CHILD BEHAVIOR; CHILD, PRESCHOOL; FEMALE; HUMANS; MALE; REPRODUCIBILITY OF RESULTS; SEVERITY OF ILLNESS INDEX;

EID: 84888069165     PISSN: 01623257     EISSN: 15733432     Source Type: Journal    
DOI: 10.1007/s10803-013-1832-9     Document Type: Article

References (81)

1. Arbelle, S.; Sigman, M. D.; & Kasari, C. (1994). Compliance with parental prohibition in autistic children. Journal of Autism and Developmental Disorders, 24, 693-702.
2. Arizona Software. (2008). GraphClick graph-digitization software: Release 3.0 [computer software]. Neuchâtel: Arizona Software.
3. Bagby, R. M.; Ryder, A. G.; Schuller, D. B.; & Marshall, M. B. (2004). The Hamilton Depression Rating Scale: Has the gold standard become a lead weight? American Journal of Psychiatry, 161, 2163-2177.
4. Baghdadli, A.; Assouline, B.; Sonié, S.; Pernon, E.; Darrou, C.; Michelon, C.; et al. (2012). Developmental trajectories of adaptive behaviors from early childhood to adolescence in a cohort of 152 children with autism spectrum disorders. Journal of Autism and Developmental Disorders, 42, 1314-1325.
5. Baker, W. L.; White, C. M.; Cappelleri, J. C.; Kluger, J.; & Coleman, C. I. (2009). Understanding heterogeneity in meta-analysis: The role of meta-regression. International Journal of Clinical Practice, 63, 1426-1434.
6. Bebko, J. M.; Konstantareas, M. M.; & Springer, J. (1987). Parent and professional evaluations of family stress associated with characteristics of autism. Journal of Autism and Developmental Disorders, 17, 565-576.
7. Bernabei, P.; Cerquiglini, A.; Cortesi, F.; & D'Ardia, C. (2007). Regression versus no regression in the autistic disorder: Developmental trajectories. Journal of Autism and Developmental Disorders, 37, 580-588.
8. Bornmann, L.; Mutz, R.; & Daniel, H. D. (2010). A reliability-generalization study of journal peer reviews: A multilevel meta-analysis of inter-rater reliability and its determinants. PLoS One, 5, 1-10.
9. Botella, J.; Suero, M.; & Gambara, H. (2010). Psychometric inferences from a meta-analysis of reliability and internal consistency coefficients. Psychological Methods, 4, 386-397.
10. Bristol, M. M. (1987). Mothers of children with autism or communication disorders: Successful adaptation and the Double ABCX model. Journal of Autism and Developmental Disorders, 17, 469-486.
11. Campbell, J. S. (2003). Meta-analysis and reliability generalization of the Psychopathy Checklist applied to youth. Unpublished doctoral dissertation from University of Northern Colorado.
12. Chen, G. M.; Yoder, J. J.; Ganzei, B. L.; Goodwin, M. S.; & Belmonte, M. K. (2012). Harnessing repetitive behaviours to engage attention and learning in a novel therapy for autism: An exploratory analysis. Frontiers in Psychology, doi: 10.3389/fpsyg.2012.00012.
13. Chez, M. G.; & Buchanan, C. P. (2000). Comment: Reply: Reply to B. Rimland's "Comments on 'Secretin and autism: A two-part clinical investigation'". Journal of Autism and Developmental Disorders, 30, 97-98.
14. Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284-290.
15. Connelly, B. S. (2008). The reliability, convergence, and predictive validity of personality ratings: An other perspective. Unpublished doctoral dissertation from University of Minnesota.
16. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.
17. Duhachek, A.; & Iacobucci, D. (2004). Alpha's standard error (ASE): An accurate and precise confidence interval estimate. Journal of Applied Psychology, 5, 792-808.
18. Fein, D.; Skoff, B.; & Mirsky, A. F. (1981). Clinical correlates of brainstem dysfunction in autistic children. Journal of Autism and Developmental Disorders, 11, 303-315.
19. Freeman, N. L.; Perry, A.; & Factor, D. C. (1991). Child behaviours as stressors: Replicating and extending the use of the CARS as a measure of stress: A research note. Journal of Child Psychology and Psychiatry, 32, 1025-1030.
20. Hakstian, A. R.; & Whalen, T. E. (1976). A k-sample significance test for independent alpha coefficients. Psychometrika, 41, 219-231.
21. Hedges, L. V.; Tipton, E.; & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1, 39-65.
22. Hobson, R. P.; Lee, A.; & Brown, R. (1999). Autism and congenital blindness. Journal of Autism and Developmental Disorders, 29, 45-56.
23. Howell, R. T.; & Shields, A. L. (2008). The file drawer problem in reliability generalization: A strategy to compute fail-safe N with reliability coefficients. Educational and Psychological Measurement, 68, 120-128.
24. Hranilovic, D.; Bujas-Petkovic, Z.; Vragovic, R.; Vuk, T.; Hock, K.; & Jernej, B. (2007). Hyperserotonemia in adults with autistic disorder. Journal of Autism and Developmental Disorders, 37, 1934-1940.
25. Kasari, C.; & Sigman, M. (1997). Linking parental perceptions to interactions in young children with autism. Journal of Autism and Developmental Disorders, 27, 39-57.
26. Konstantareas, M. M.; & Homatidis, S. (1989). Assessing child symptom severity and stress in parents of autistic children. Journal of Child Psychology and Psychiatry, 30, 459-470.
27. Konstantareas, M. M.; & Homatidis, S. (1999). Chromosomal abnormalities in a series of children with autistic disorder. Journal of Autism and Developmental Disorders, 29, 275-285.
28. Konstantareas, M. M.; Homatidis, S.; & Plowright, C. M. S. (1992). Assessing resources and stress in parents of severely dysfunctional children through the Clarke modification of Holroyd's Questionnaire on Resources and Stress. Journal of Autism and Developmental Disorders, 22, 217-234.
29. Konstantareas, M. M.; & Stewart, K. (2006). Affect regulation and temperament in children with autism spectrum disorder. Journal of Autism and Developmental Disorders, 36, 143-154.
30. Kurita, H.; Miyake, Y.; & Katsuno, K. (1989). Reliability and validity of the Childhood Autism Rating Scale-Tokyo version (CARS-TV). Journal of Autism and Developmental Disorders, 19, 389-396.
31. Lance, C. E.; Butts, M. M.; & Michels, L. C. (2006). The sources of four commonly reported cutoff criteria: What did they really say? Organizational Research Methods, 9, 202-220.
32. Laushey, K. M.; & Heflin, L. J. (2000). Enhancing social skills of kindergarten children with autism through the training of multiple peers as tutors. Journal of Autism and Developmental Disorders, 30, 183-193.
33. LeBreton, J. M.; & Senter, J. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11, 815-852.
34. Li, A.; & Bagger, J. (2007). The balanced inventory of desirable responding (BIDR): A reliability generalization study. Educational and Psychological Measurement, 67, 525-544.
35. Lord, C.; Schopler, E.; & Revicki, D. (1982). Sex differences in autism. Journal of Autism and Developmental Disorders, 12, 317-330.
36. Matese, M.; Matson, J. L.; & Sevin, J. (1994). Comparison of psychotic and autistic children using behavioral observation. Journal of Autism and Developmental Disorders, 24, 83-94.
37. Matson, J. L.; Smiroldo, B. B.; & Hastings, T. L. (1998). Validity of the autism/pervasive developmental disorder subscale of the Diagnostic Assessment for the Severely Handicapped-II. Journal of Autism and Developmental Disorders, 28, 77-81.
38. Mokkink, L. B.; Terwee, C. B.; Stratford, P. W.; Alonso, J.; Patrick, D. L.; Riphagen, I.; et al. (2009). Evaluation of the methodological quality of systematic reviews of health status measurement instruments. Quality of Life Research, 18, 313-333.
39. Nassen, L. M. (2008). An internal consistency reliability generalization of the Child Behavior Checklist. Unpublished doctoral dissertation from University of Northern Colorado.
40. Njardvik, U.; Matson, J. L.; & Cherry, K. E. (1999). A comparison of social skills in adults with autistic disorder, pervasive developmental disorder not otherwise specified, and mental retardation. Journal of Autism and Developmental Disorders, 29, 287-295.
41. Nordin, V.; Gillberg, C.; & Nyden, A. (1998). The Swedish version of the Childhood Autism Rating Scale in a clinical setting. Journal of Autism and Developmental Disorders, 28, 69-75.
42. Pereira, T. V.; Patsopoulos, N. A.; Salanti, G.; & Ioannidis, J. P. A. (2010). Critical interpretation of Cochran's Q test depends on power and prior assumptions about heterogeneity. Research Synthesis Methods, 1, 149-161.
43. Perry, A.; Condillac, R. A.; Freeman, N. L.; Dunn-Geier, J.; & Belair, J. (2005). Multi-site study of the Childhood Autism Rating Scale (CARS) in five clinical groups of young children. Journal of Autism and Developmental Disorders, 35, 625-634.
44. Ponterotto, J. G.; & Charter, R. A. (2009). Statistical extensions of Ponterotto and Ruckdeschel's (2007) reliability matrix for estimating the adequacy of internal consistency coefficients. Perceptual and Motor Skills, 108, 878-886.
45. Reichler, R. J.; & Schopler, E. (1971). Observations on the nature of human relatedness. Journal of Autism and Childhood Schizophrenia, 1, 283-296.
46. Rodriguez, M. C.; & Jung, T. (2003). The replicability of reliability generalization. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL. Accessed May 27, 2011 from http://www.edmeasurement.net/aera/papers/rg%20replication%20aera.pdf.
47. Rodriguez, M. C.; & Maeda, Y. (2006). Meta-analysis of coefficient alpha. Psychological Methods, 11, 306-322.
48. Rousson, V.; Gasser, T.; & Seifert, B. (2002). Assessing intrarater, interrater and test-retest reliability of continuous measurements. Statistics in Medicine, 21, 3431-3446.
49. Ryngala, D. J.; Shields, A. L.; & Caruso, J. C. (2005). Reliability generalization of the revised Children's Manifest Anxiety Scale. Educational and Psychological Measurement, 65, 259-271.
50. Sæmundsen, E.; Magnússon, P.; Smári, J.; & Sigurdardóttir, S. (2003). Autism Diagnostic Interview-Revised and the Childhood Autism Rating Scale: Convergence and discrepancy in diagnosing autism. Journal of Autism and Developmental Disorders, 33, 319-328.
51. Samtani, A.; Sterling-Lewis, K.; Scholten, R. J. P. M.; Woolfenden, S.; Hooft, L.; & Williams, K. (2011). Diagnostic tests for autism spectrum disorder (ASD) in preschool children. Cochrane Database for Systematic Reviews Collaboration,. doi: 10.1002/14651858.cd009044.
52. Sawilowsky, S. S. (2000). Psychometrics versus datametrics: Comment on Vacha-Haase's "reliability generalization" method and some Epm [sic] editorial policies. Educational and Psychological Measurement, 60, 157-173.
53. Schlosser, R. W.; Wendt, O.; & Sigafoos, J. (2007). Not all systematic reviews are created equal: Considerations for appraisal. Evidence-Based Communication Assessment and Intervention, 1, 138-150.
54. Schopler, E.; Andrews, C. E.; & Strupp, K. (1979a). Do autistic children come from upper-middle-class parents? Journal of Autism and Developmental Disorders, 9, 139-152.
55. Schopler, E.; & Reichler, R. J. (1971). Parents as cotherapists in the treatment of psychotic children. Journal of Autism and Childhood Schizophrenia, 1, 87-102.
56. Schopler, E.; & Reichler, R. J. (1972). How well do parents understand their own psychotic child? Journal of Autism and Childhood Schizophrenia, 2, 387-400.
57. Schopler, E.; Reichler, R. J.; DeVellis, R. F.; & Daly, K. (1980). Toward objective classification of childhood autism: Childhood Autism Rating Scale (CARS). Journal of Autism and Developmental Disorders, 10, 91-103.
58. Schopler, E.; Reichler, R. J.; & Renner, B. R. (1988). The Childhood Autism Rating Scale (CARS). Los Angeles, CA: Western Psychological Services.
59. Schopler, E.; Rutter, M.; & Chess, S. (1979b). Editorial: Change of journal scope and title. Journal of Autism and Developmental Disorders, 9, 1-10.
60. Sevin, J. A.; Matson, J. L.; Coe, D. A.; Fee, V. E.; & Sevin, B. M. (1991). A comparison and evaluation of three commonly used autism scales. Journal of Autism and Developmental Disorders, 21, 417-432.
61. Sloan, J. L. (1978). Differential development of autistic symptoms in a pair of fraternal twins. Journal of Autism and Childhood Schizophrenia, 8, 191-202.
62. Sloan, J. L.; & Marcus, L. (1981). Some findings on the use of the adaptive behavior scale with autistic children. Journal of Autism and Developmental Disorders, 11, 191-199.
63. Stanley, G. C.; & Konstantareas, M. M. (2007). Symbolic play in children with autism spectrum disorder. Journal of Autism and Developmental Disorders, 37, 1215-1223.
64. StataCorp. (2012). Stata statistical software: Release 12.1 [computer software]. College Station, TX: StataCorp LP.
65. Sun, S. (2011). Meta-analysis of Cohen's kappa. Health Services Outcomes Research Methodology, 11, 145-163.
66. Teal, M. B.; & Wiebe, M. J. (1986). A validity analysis of selected instruments used to assess autism. Journal of Autism and Developmental Disorders, 16, 485-494.
67. Thompson, B.; & Vacha-Haase, T. (2000). Psychometrics is datametrics: The test is not reliable. Educational and Psychological Measurement, 60, 174-195.
68. Trajković, G.; Starčević, V.; Latas, M.; Leštarević, M.; Ille, T.; Bukumirić, Z.; et al. (2011). Reliability of the Hamilton Rating Scale for Depression: A meta-analysis over a period of 49 years. Psychiatry Research, 189, 1-9.
69. Travis, L.; Sigman, M.; & Ruskin, E. (2001). Links between social understanding and social behavior in verbally able children with autism. Journal of Autism and Developmental Disorders, 31, 119-130.
70. Vacha-Haase, T.; Kogan, L. R.; & Thompson, B. (2000). Sample compositions and variabilities in published studies versus those in test manuals: Validity of score reliability inductions. Educational and Psychological Measurement, 60, 509-522.
71. Vacha-Haase, T.; & Thompson, B. (2011). Score reliability: A retrospective look back at 12 years of reliability generalization studies. Measurement and Evaluation in Counseling and Development, 44, 159-168.
72. Van Bourgondien, M. E.; Marcus, L. M.; & Schopler, E. (1992). Comparison of DSM-III-R and Childhood Autism Rating Scale diagnoses of autism. Journal of Autism and Developmental Disorders, 22, 493-506.
73. Vassar, M.; & Hale, W. (2007). Reliability reporting practices in youth life satisfaction research. Social Indicators Research, 83, 487-496.
74. Vassar, M.; Ridge, J. W.; & Hill, A. D. (2008). Inducing score reliability from previous reports: An examination of life satisfaction studies. Social Indicators Research, 87, 27-45.
75. Viding, E.; & Blakemore, S. J. (2007). Endophenotype approach to developmental psychopathology: Implications for autism research. Behavioral Genetics, 37, 51-60.
76. Wiggins, L. D.; Robins, D. L.; Adamson, L. B.; Bakeman, R.; & Henrich, C. C. (2012). Support for a dimensional view of autism spectrum disorders in toddlers. Journal of Autism and Developmental Disorders, 42, 191-200.
77. Wilkinson, L.; & American Psychological Association (APA) Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604.
78. Wilson, R.; Pascalis, O.; & Blades, M. (2007). Familiar face recognition in children with autism: The differential use of inner and outer face parts. Journal of Autism and Developmental Disorders, 37, 314-320.
79. Woolfenden, S.; Sarkozy, V.; Ridley, G.; & Williams, K. (2012). A systematic review of the diagnostic stability of autism spectrum disorder. Research in Autism Spectrum Disorders, 6, 345-354.
80. Yamashita, Y.; Fujimoto, C.; Nakajima, E.; Isagai, T.; & Matsuishi, T. (2003). Possible association between congenital cytomegalovirus infection and autistic disorder. Journal of Autism and Developmental Disorders, 33, 455-459.
81. Zheng, Y. P.; Zhao, J. P.; Phillips, M.; Liu, J. B.; Cai, M. F.; Sun, S. Q.; et al. (1988). Validity and reliability of the Chinese Hamilton Depression Rating Scale. British Journal of Psychiatry, 152, 660-664.