Absolute and relative reliability of percentage of syllables stuttered and severity rating scales

Journal of Speech, Language, and Hearing Research

Volumn 57, Issue 4, 2014, Pages 1284-1295

  Karimi, Hamid ^a,b   O'Brian, Sue ^a   Onslow, Mark ^a   Jones, Mark ^c  

^a UNIVERSITY OF SYDNEY   (Australia)

^b ISFAHAN UNIVERSITY OF MEDICAL SCIENCES   (Iran)

^c UNIVERSITY OF QUEENSLAND   (Australia)

Author keywords

Absolute reliability; Percentage of syllables stuttered; Relative reliability; Severity rating scale; Stuttering

Indexed keywords

ADULT; AGED; EVALUATION STUDY; FEMALE; HUMAN; MALE; MIDDLE AGED; PROCEDURES; REPRODUCIBILITY; SEVERITY OF ILLNESS INDEX; SPEECH ANALYSIS; STATISTICS AND NUMERICAL DATA; STUTTERING; YOUNG ADULT;

ADULT; AGED; FEMALE; HUMANS; MALE; MIDDLE AGED; REPRODUCIBILITY OF RESULTS; SEVERITY OF ILLNESS INDEX; SPEECH PRODUCTION MEASUREMENT; STUTTERING; YOUNG ADULT;

EID: 84905702440     PISSN: 10924388     EISSN: None     Source Type: Journal    
DOI: 10.1044/2014_JSLHR-S-13-0106     Document Type: Article

References (99)

1. Allen, C., Nikolopoulos, T. P., Dyar, D., & O'Donoghue, G. M. (2001). Reliability of a rating scale for measuring speech intelligibility after pediatric cochlear implantation. Otology & Neurotology, 22, 631-633.
2. Armson, J., & Kiefte, M. (2008). The effect of SpeechEasy on stuttering frequency, speech rate, and speech naturalness. Journal of Fluency Disorders, 33, 120-134.
3. Aron, M. L. (1967). The relationships between measurements of stuttering behavior. Journal of the South African Logopedic Society, 14, 15-34.
4. Atkinson, G. (2003). What is this thing called measurement error? In T. Reilly & M. Marfell-Jones (Eds.), Kinathropometry VIII: Proceeding of the 8th International Conference of Kinathropometry (I S A K) (pp. 3-13). New York, NY: Routledge.
5. Atkinson, G., & Nevill, A. M. (1998). Statistical methods for assessing measurement error (reliability) in variables relevant to sport medicine. Sports Medicine, 26, 217-238.
6. Batterham, A. M., & George, K. P. (2003). Reliability in evidence-based clinical practice: A primer for allied health professionals. Physical Therapy in Sport, 4, 122-128.
7. Beal, D. S., Quraan, M. A., Cheyne, D. O., Taylor, M. J., Gracco, V. L., & De Nil, L. F. (2011). Speech-induced suppression of evoked auditory fields in children who stutter. NeuroImage, 54, 2994-3003.
8. Beilby, J. M., Byrnes, M. L., & Yaruss, J. S. (2012). Acceptance and commitment therapy for adults who stutter: Psychosocial adjustment and speech fluency. Journal of Fluency Disorders, 37, 289-299.
9. Bland, J. M., & Altman, D. G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. Lancet, 1, 307-310.
10. Bland, J. M., & Altman, D. G. (1995). Comparing methods of measurement: Why plotting difference against standard method is misleading. Lancet, 346, 1085-1087.
11. Bland, J. M., & Altman, D. G. (1999). Measuring agreement in method comparison studies. Statistical Methods in Medical Research, 8, 135-160.
12. Boberg, E., & Kully, D. (1994). Long-term results of an intensive treatment program for adults and adolescents who stutter. Journal of Speech and Hearing Research, 37, 1050-1059.
13. Bothe, A. K. (2008). Identification of children's stuttered and non-stuttered speech by highly experienced judges: Binary judgments and comparisons with disfluency-types definitions. Journal of Speech, Language, and Hearing Research, 51, 867-878.
14. Brundage, S. B., Bothe, A. K., Lengeling, A. N., & Evans, J. J. (2006). Comparing judgments of stuttering made by students, clinicians, and highly experienced judges. Journal of Fluency Disorders, 31, 271-283.
15. Bruton, A., Conway, J. H., & Holgate, S. T. (2000). Reliability: What is it and how is it measured? Physiotherapy, 86, 94-99.
16. Carey, B., O'Brian, S., Onslow, M., Block, S., Packman, A., & Jones, M. (2010). Randomized controlled non-inferiority trial of a telehealth treatment for chronic stuttering: The Camperdown Program. International Journal of Language and Communication Disorders, 45, 108-120.
17. Chang, S. E., Horwitz, B., Ostuni, J., Reynolds, R., & Ludlow, C. L. (2011). Evidence of left inferior frontal-premotor structural and functional connectivity deficits in adults who stutter. Cerebral Cortex, 21, 2507-2518.
18. Choo, A. L., Chang, S. E., Zengin-Bolatkale, H., Ambrose, N. G., & Loucks, T. M. (2012). Corpus callosum morphology in children who stutter. Journal of Communication Disorders, 45, 279-289.
19. Cordes, A. K. (1994). The reliability of observational data: I. Theories and methods for speech-language pathology. Journal of Speech and Hearing Research, 37, 264-279.
20. Cordes, A. K., & Ingham, R. J. (1995). Judgments of stuttered and nonstuttered intervals by recognized authorities in stuttering research. Journal of Speech and Hearing Research, 38, 33-41.
21. Cordes, A. K., & Ingham, R. J. (1999). Effects of time-interval judgment training on real-time measurement of stuttering. Journal of Speech, Language, and Hearing Research, 42, 862-879.
22. Costello, J. M. (1975). Time-out procedures for the modification of stuttering: Three case studies. Journal of Speech and Hearing Disorders, 40, 216-231.
23. Cream, A., O'Brian, S., Jones, M., Block, S., Harrison, E., Lincoln, M., Onslow, M. (2010). Randomized controlled trial of video self-modeling following speech restructuring treatment for stuttering. Journal of Speech, Language, and Hearing Research, 53, 887-897.
24. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Orlando, FL: Holt, Rinehart & Winston.
25. Cullinan, W. L., Prather, E. M., & Williams, D. E. (1963). Comparison of procedures for scaling severity of stuttering. Journal of Speech and Hearing Research, 6, 187-194.
26. Curran, M. F., & Hood, S. B. (1977). Listener ratings of severity for specific disfluency types in children. Journal of Fluency Disorders, 2, 87-98.
27. de Vet, H. C., Terwee, C. B., Knol, D. L., & Bouter, L. M. (2006). When to use agreement versus reliability measures. Journal of Clinical Epidemiology, 59, 1033-1039.
28. Domholdt, E. (2005). Rehabilitation research: Principles and applications (3rd ed.). Philadelphia, PA: Elsevier Saunders.
29. Donoghue, D., & Stokes, E. K. (2009). How much change is true change? The minimum detectable change of the Berg Balance Scale in elderly people. Journal of Rehabilitation Medicine, 41, 343-346.
30. Eve, C. L., Onslow, M., Andrews, C., & Adams, R. (1995). Clinical measurement of early stuttering severity: The reliability of a 10-point scale. Australian Journal of Human Communication Disorders, 23, 26-39.
31. Femrell, L., Åvall, M., & Lindström, E. (2012). Two-year follow-up of the Lidcombe Program in ten Swedish-speaking children. Folia Phoniatrica et Logopaedica, 64, 248-253.
32. Foundas, A. L., Bollich, A. M., Feldman, J., Corey, D. M., Hurley, M., Lemen, L. C., & Heilman, K. M. (2004). Aberrant auditory processing and atypical planum temporale in developmental stuttering. Neurology, 63, 1640-1646.
33. Foundas, A. L., Corey, D. M., Angeles, V., Bollich, A. M., Crabtree-Hartman, E., & Heilman, K. M. (2003). Atypical cerebral laterality in adults with persistent developmental stuttering. Neurology, 61, 1378-1385.
34. Gallop, R. F., & Runyan, C. M. (2012). Long-term effectiveness of the SpeechEasy fluency enhancement device. Journal of Fluency Disorders, 37, 334-343.
35. Guyatt, G., Walter, S., & Norman, G. (1987). Measuring change over time: Assessing usefulness of evaluative instruments. Journal of Chronic Diseases, 40, 171-178.
36. Harasym, J., & Langevin, M. (2012). Stuttering treatment for a school-age child with Down syndrome: A descriptive case report. Journal of Fluency Disorders, 37, 253-262.
37. Harris, V., Onslow, M., Packman, A., Harrison, E., & Menzies, R. (2002). An experimental investigation of the impact of the Lidcombe Program on early stuttering. Journal of Fluency Disorders, 27, 203-214.
38. Harrison, E., Onslow, M., Andrews, C., Packman, A., & Webber, M. (1998). Control of stuttering with prolonged speech: Development of a one-day instatement program. In A. Cordes & R. J. Ingham (Eds.), Treatment efficacy in stuttering: A search for empirical bases (pp. 191-212). San Diego, CA: Singular.
39. Harrison, E., Onslow, M., & Menzies, R. (2004). Dismantling the Lidcombe Program of early stuttering intervention: Verbal contingencies for stuttering and clinical measurement. International Journal of Language & Communication Disorders, 39, 257-267.
40. Hewat, S., Onslow, M., Packman, A., & O'Brian, S. (2006). A Phase II clinical trial of self-imposed time-out treatment for stuttering in adults and adolescents. Disability and Rehabilitation, 28, 33-42.
41. Hopkins, W. G. (2000). Measures of reliability in sports medicine and science. Sports Medicine, 30, 1-15.
42. Ingham, R. J., & Cordes, A. (1992). Interclinic differences in stuttering-event counts. Journal of Fluency Disorders, 17, 171-176.
43. Ingham, R. J., Kilgo, M., Ingham, J. C., Moglia, R., Belknap, H., & Sanchez, T. (2001). Evaluation of a stuttering treatment based on reduction of short phonation intervals. Journal of Speech, Language, and Hearing Research, 44, 1229-1244.
44. Johnson, W. (1955). The time, the place, and the problem. In W. Johnson & R. R. Leutenegger (Eds.), Stuttering in children and adults (pp. 3-24). Minneapolis: University of Minnesota Press.
45. Jones, M., Dobson, A., & O'Brian, S. (2011). A graphical method for assessing agreement with the mean between multiple observers using continuous measures. International Journal of Epidemiology, 40, 1308-1313.
46. Jones, M., Onslow, M., Packman, A., O'Brian, S., Hearne, A., Williams, S., Schwarz, I. (2008). Extended follow-up of a randomized controlled trial of the Lidcombe Program of Early Stuttering Intervention. International Journal of Language & Communication Disorders, 43, 649-661.
47. Jones, M., Onslow, M., Packman, A., Williams, S., Ormond, T., Schwarz, I., & Gebski, V. (2005). Randomised controlled trial of the Lidcombe programme of early stuttering intervention. British Medical Journal, 331, 659-661.
48. Karimi, H., O'Brian, S., Onslow, M., Jones, M., Menzies, R., & Packman, A. (2013). Unscheduled telephone calls to measure percent syllables stuttered during clinical trials. Journal of Speech, Language, and Hearing Research, 56, 1455-1461.
49. Kreiman, J., & Gerratt, B. R. (2011). Comparing two methods for reducing variability in voice quality measurements. Journal of Speech, Language, and Hearing Research, 54, 803-812.
50. Kreiman, J., Gerratt, B. R., & Ito, M. (2007). When and why listeners disagree in voice quality assessment tasks. The Journal of the Acoustical Society of America, 122, 2354-2364.
51. Kreiman, J., Gerratt, B. R., Kempster, G. B., Erman, A., & Berke, G. S. (1993). Perceptual evaluation of voice quality: Review, tutorial, and a framework for future research. Journal of Speech and Hearing Research, 36, 21-40.
52. Kully, D., & Boberg, E. (1988). An investigation of interclinic agreement in the identification of fluent and stuttered syllables. Journal of Fluency Disorders, 13, 309-318.
53. Langevin, M., Huinck, W. J., Kully, D., Peters, H. F. M., Lomheim, H., & Tellers, M. (2006). A cross-cultural, long-term outcome evaluation of the ISTAR Comprehensive Stuttering Program across Dutch and Canadian adults who stutter. Journal of Fluency Disorders, 31, 229-256.
54. Langevin, M., Kully, D., Teshima, S., Hagler, P., & Narasimha Prasad, N. G. (2010). Five-year longitudinal treatment outcomes of the ISTAR Comprehensive Stuttering Program. Journal of Fluency Disorders, 35, 123-140.
55. Lewis, D., & Sherman, D. (1951). Measuring the severity of stuttering. Journal of Speech Disorders, 16, 320-326.
56. Lincoln, M., & Onslow, M. (1997). Long-term outcome of early intervention for stuttering. American Journal of Speech-Language Pathology, 6, 51-58.
57. MacDonald, J. D., & Martin, R. R. (1973). Stuttering and disfluency as two reliable and unambiguous response classes. Journal of Speech and Hearing Research, 16, 691-699.
58. Martin, R. R., Haroldson, S. K., & Triden, K. A. (1984). Stuttering and speech naturalness. Journal of Speech and Hearing Disorders, 49, 53-58.
59. Martin, R., Haroldson, S., & Woessner, G. (1988). Perceptual scaling of stuttering severity. Journal of Fluency Disorders, 13, 27-47.
60. Mulcahy, K., Hennessey, N., Beilby, J., & Byrnes, M. (2008). Social anxiety and the severity and topography of stuttering in adolescents. Journal of Fluency Disorders, 33, 306-319.
61. Naylor, R. V. (1953). A comparative study of methods of estimating the severity of stuttering. Journal of Speech and Hearing Disorders, 18, 30-37.
62. Neef, N. E., Jung, K., Rothkegel, H., Pollok, B., von Gudenberg, A. W., Paulus, W., & Sommer, M. (2011). Right-shift for non-speech motor processing in adults who stutter. Cortex, 47, 945-954.
63. Nelson, M. (1997). The validation of a dietary assessment. In B. M. Margetts & M. Nelson (Eds.), Design concepts in nutri tional epidemiology (pp. 242-272). Oxford, England: Oxford Medical Publications.
64. Nevill, A. N., & Atkinson, G. (1997). Assessing agreement between measurements recorded on a ratio scale in sports medicine and sports science. British Journal of Sports Medicine, 31, 314-318.
65. O'Brian, S., Jones, M., Pilowsky, R., Onslow, M., Packman, A., & Menzies, R. (2010). A new method to sample stuttering in preschool children. International Journal of Speech-Language Pathology, 12, 173-177.
66. O'Brian, S., Packman, A., & Onslow, M. (2004). Self-rating of stuttering severity as a clinical tool. American Journal of Speech-Language Pathology, 13, 219-226.
67. O'Brian, S., Packman, A., & Onslow, M. (2008). Telehealth delivery of the Camperdown Program for adults who stutter: A Phase I trial. Journal of Speech, Language, and Hearing Research, 51, 184-195.
68. O'Brian, S., Packman, A., Onslow, M., Cream, A., O'Brian, N., & Bastock, K. (2003). Is listener comfort a viable construct in stuttering research? Journal of Speech, Language, and Hearing Research, 46, 503-509.
69. O'Brian, S., Packman, A., Onslow, M., & O'Brian, N. (2003). Generalizability theory II: Application to perceptual scaling of speech naturalness in adults who stutter. Journal of Speech, Language, and Hearing Research, 46, 718-723.
70. O'Brian, S., Packman, A., Onslow, M., & O'Brian, N. (2004). Measurement of stuttering in adults: Comparison of stuttering-rate and severity-scaling methods. Journal of Speech, Language, and Hearing Research, 47, 1081-1087.
71. O'Donoghue, P. (2012). Statistics for sport and exercise studies. London, England: Routledge.
72. Olds, T. (2002). Five errors about error. Journal of Science and Medicine in Sport, 5, 336-340.
73. Onslow, M., Adams, R., & Ingham, R. (1992). Reliability of speech naturalness ratings of stuttered speech during treatment. Journal of Speech and Hearing Research, 35, 994-1001.
74. Onslow, M., Andrews, C., & Costa, L. (1990). Parental severity scaling of early stuttered speech: Four case studies. Australian Journal of Human Communication Disorders, 18, 47-61.
75. Onslow, M., Andrews, C., & Lincoln, M. (1994). A control/experimental trial of an operant treatment for early stuttering. Journal of Speech and Hearing Research, 37, 1244-1259.
76. Onslow, M., Costa, L., Andrews, C., Harrison, E., & Packman, A. (1996). Speech outcomes of a prolonged-speech treatment for stuttering. Journal of Speech and Hearing Research, 39, 734-749.
77. Onslow, M., Packman, A., Stocker, S., van Doorn, J., & Siegel, G. M. (1997). Control of children's stuttering with response-contingent time-out: Behavioral, perceptual and acoustic data. Journal of Speech, Language, and Hearing Research, 40, 121-133.
78. Packman, A., Onslow, M., & van Doorn, J. (1994). Prolonged speech and modification of stuttering: Perceptual, acoustic, and electro-glottographic data. Journal of Speech and Hearing Research, 37, 724-737.
79. Rousey, C. L. (1958). Stuttering severity during prolonged spontaneous speech. Journal of Speech and Hearing Research, 1, 40-47.
80. Ryan, B. P., & Van Kirk Ryan, B. (1983). Programmed stuttering therapy for children: Comparison of four establishment programs. Journal of Fluency Disorders, 8, 291-322.
81. Ryan, B.P., & Van Kirk Ryan, B. (1995). Programmed stuttering treatment for children: Comparison of two establishment programs through transfer, maintenance, and follow-up. Journal of Speech and Hearing Research, 38, 61-75.
82. Saliva, J., Ysseldyke, J. E., & Bolt, S. (2010). Assessment in special and inclusive education (11th ed.). New York, NY: Houghton-Mifflin.
83. Schiavetti, N., Martin, R. R., Haroldson, S. K., & Metz, D. E. (1994). Psychophysical analysis of audiovisual judgments of speech naturalness of nonstutterers and stutterers. Journal of Speech and Hearing Research, 37, 46-52.
84. Sherman, D., & McDermott, R. (1958). Individual ratings of severity of moments of stuttering. Journal of Speech and Hearing Research, 1, 61-67.
85. Shrivastav, R., Sapienza, C. M., & Nandur, V. (2005). Application of psychometric theory to the measurement of voice quality using rating scales. Journal of Speech, Language, and Hearing Research, 48, 323-335.
86. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428.
87. Sim, J., & Wright, C. C. (2005). The kappa statistic in reliability studies: Use, interpretation, and sample size requirements. Physical Therapy, 85, 257-268.
88. Stuart, A., Kalinowski, J., Saltuklaroglu, T., & Guntupalli, V. K. (2006). Investigations of the impact of altered auditory feedback in-the-ear devices on the speech of people who stutter: One-year follow-up. Disability and Rehabilitation, 28, 757-765.
89. Tasko, S. M., McClean, M. D., & Runyan, C. M. (2007). Speech motor correlates of treatment-related changes in stuttering severity and speech naturalness. Journal of Communication Disorders, 40, 42-65.
90. Teshima, S., Langevin, M., Hagler, P., & Kully, D. (2010). Post-treatment speech naturalness of comprehensive stuttering program clients and differences in ratings among listener groups. Journal of Fluency Disorders, 35, 44-58.
91. Tinsley, H. E., & Weiss, D. J. (1975). Interrater reliability and agreement of subjective judgments. Journal of Counseling Psychology, 22, 358-374.
92. Trajkowski, N., Andrews, C., Onslow, M., O'Brian, S., Packman, A., & Menzies, R. (2011). A Phase II trial of the Westmead Program: Syllable-timed speech treatment for preschool children who stutter. International Journal of Speech-Language Pathology, 13, 500-509.
93. Unger, J. P., Glück, C. W., & Cholewa, J. (2012). The immediate effects of AAF devices on the characteristics of stuttering: A clinical analysis. Journal of Fluency Disorders, 37, 122-134.
94. Van Borsel, J., & Eeckhout, H. (2008). The speech naturalness of people who stutter speaking under delayed auditory feedback as perceived by different groups of listeners. Journal of Fluency Disorder, 33, 241-251.
95. Weir, J. P. (2005). Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. Journal of Strength and Conditioning Research, 19, 231-240.
96. Williams, D. E., & Kent, L. R. (1958). Listener evaluations of speech interruptions. Journal of Speech and Hearing Research, 1, 124-131.
97. Young, M. A. (1969). Observer agreement: Cumulative effects of rating many samples. Journal of Speech and Hearing Research, 12, 135-143.
98. Young, M. A. (1969). Observer agreement: Cumulative effects of repeated ratings of the same samples and of knowledge of group results. Journal of Speech and Hearing Research, 12, 144-155.
99. Young, M. A. (1970). Anchoring and sequence effects for the category scaling of stuttering severity. Journal of Speech and Hearing Research, 13, 360-368.