A breakdown of reliability coefficients by test type and reliability method, and the clinical implications of low reliability

Journal of General Psychology

Volumn 130, Issue 3, 2003, Pages 290-304

  Charter, Richard A ^a  

^a VA LONG BEACH HEALTHCARE SYSTEM   (United States)

Author keywords

Clinical decision errors; Reliability; True scores; True score confidence intervals

Indexed keywords

EID: 0142184365     PISSN: 00221309     EISSN: 19400888     Source Type: Journal    
DOI: 10.1080/00221300309601160     Document Type: Article

References (63)

1. Aiken, L. R. (1991). Psychological testing and assessment (7th ed.). Boston: Allyn & Bacon.
2. Aiken, L. S., West, S. G., Sechrest, L., Reno, R. R., Roediger, H. L., Scarr, S., Kazdin, A. E., & Sherman, S. J. (1990). The training in statistics, methodology, and measurement in psychology. American Psychologist, 45, 721-734.
3. Alsawalmeh, Y. M., & Feldt, L. S. (1992). Test of the hypothesis that the intraclass co-efficient is the same for two measurement procedures. Applied Psychological Measurement, 16, 195-205.
4. Alsawalmeh, Y. M., & Feldt, L. S. (1994a). A modification of Feldt's test of the equality of two dependent alpha coefficients. Psychometrika, 59, 49-57.
5. Alsawalmeh, Y. M., & Feldt, L. S. (1994b). Testing the equality of two related intraclass reliability coefficients. Applied Psychological Measurement, 18, 183-190.
6. Alsawalmeh, Y. M., & Feldt, L. S. (1999). Testing the equality of independent alpha coefficients adjusted for test length. Educational and Psychological Measurement, 59, 373-383.
7. Anastasi, A., & Urbina, S. (1997). Psychological testing (7th ed.). Upper Saddle River, NJ: Prentice Hall.
8. Andrulis, R. S. (1977). Adult assessment: A source book of tests and measures of human behavior. Springfield, IL: Charles C Thomas.
9. Angoff, W. H. (1953). Test reliability and effective test length. Psychometrika, 18, 1-14.
10. Charter, R. A. (1996). Revisiting the standard errors of measurement, estimate, and prediction and their application to test scores. Perceptual and Motor Skills, 82, 1139-1144.
11. Charter, R. A. (1999). Sample size requirements for precise estimates of reliability, gen-eralizability, and validity coefficients. Journal of Clinical and Experimental Neuropsychology, 21, 559-566.
12. Charter, R. A. (2001a). –Damn the recision, full speed ahead with the clinical interpretation.— Journal of Clinical and Experimental Neuropsychology, 23, 692-694.
13. Charter, R. A. (2001b). It is time to bury the Spearman-Brown –prophecy— formula for some common applications. Educational and Psychological Measurement, 61, 690-696.
14. Charter, R. A. (2001c). Testing the equality of two or more split-half reliability coefficients. Psychological Reports, 88, 844-846.
15. Charter, R. A. (2003a). Studies are using samples too small to produce sufficiently precise reliability coefficients. The Journal of General Psychology, 130, 117-129.
16. Charter, R. A. (2003b). Understanding and evaluating interjudge reliability: The one-and two-judge case. Manuscript submitted for publication.
17. Charter, R. A., Adkins, T., Alekoumbides, A., & Seacat, G. (1987). Reliability of the WAIS, WMS, and Reitan Battery: Raw scores and standardized scores corrected for age and education. International Journal of Clinical Neuropsychology, 9, 28-32.
18. Charter, R. A., & Feldt, L. S. (1996). Testing the equality of two alpha coefficients. Perceptual and Motor Skills, 82, 763-768.
19. Charter, R. A., & Feldt, L. S. (2000). The relationship between two methods of evaluating an examinee's difference scores. Journal of Psychoeducational Assessment, 18, 125-142.
20. Charter, R. A., & Feldt, L. S. (2001a). Confidence intervals for true scores: Is there a correct approach? Journal of Psychoeducational Assessment, 19, 350-364.
21. Charter, R. A., & Feldt, L. S. (2001b). Meaning of reliability in terms of correct and incorrect clinical decisions: The art of decision making is still alive. Journal of Clinical and Experimental Neuropsychology, 23, 530-537.
22. Charter, R. A., & Feldt, L. S. (2002). The importance of reliability as it relates to true score confidence intervals. Measurement and Evaluation in Counseling and Development, 35, 104-112.
23. Charter, R. A., Walden, D. K., & Hoffman, C. (2000). Interscorer reliabilities for Memory and Location scores of the Tactual Performance Test. The Clinical Neuropsychologist, 12, 245-247.
24. Charter, R. A., & Webster, J. S. (1997). The psychometric structure of the Seashore Rhythm Test. The Clinical Neuropsychologist, 11, 167-173.
25. Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284-290.
26. Corcoran, K., & Fisher, J. (2000). Measures for clinical practice: A source book (3rd ed.). New York: Free Press.
27. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometri-ka, 16, 297-334.
28. Educational Testing Service. (1995). The ETS test collection catalog: Vol. 2. Vocational tests and measurement devices (2nd ed.). Phoenix, AZ: Oryx.
29. Feldt, L. S. (1967). A note on the use of confidence bands to evaluate the reliability of a difference between two scores. American Educational Research Journal, 4, 139-145.
30. Feldt, L. S. (1975). Estimation of the reliability of a test divided into two parts of unequal length. Psychometrika, 40, 557-561.
31. Feldt, L. S., & Brennan, R. L. (1989). Reliability. In R. H. Linn (Ed.), Educational Measurement (3rd ed.). American Counsel of Education. New York: Macmillan.
32. Feldt, L. S., & Charter, R. A. (2003). Estimating the reliability of a test split into two parts of equal or unequal length. Psychological Methods, 8, 102-109.
33. Feldt, L. S., Woodruff, D. J., & Salih, F. A. (1987). Statistical inference for coefficient alpha. Applied Psychological Measurement, 11, 93-103.
34. Folbrecht, J. R., Charter, R. A., Walden, D. K., & Dobbs, S. M. (1999). Psychometric properties of the Boston Qualitative Scoring System for the Rey-Osterrieth Complex Figure. The Clinical Neuropsychologist, 13, 442-449.
35. Gilmer, J. S., & Feldt, L. S. (1983). Reliability estimation for a test with parts of unknown lengths. Psychometrika, 48, 99-111
36. Goldman, B. A., Mitchel, D. F., & Egelson, P. E. (1997). Directory of unpublished experimental measures (Vol. 7). Washington, DC: American Psychological Association.
37. Gregory, R. J. (1999). Foundations of intellectual assessment: The WAIS-III and other tests in clinical practice. Boston: Allyn & Bacon.
38. Guilford, J. P. (1950). Fundamental statistics in psychology and education (2nd ed.). New York: McGraw-Hill.
39. Guilford, J. P., & Fruchter, B. (1978). Fundamental statistics in psychology and education (6th ed.). New York: McGraw-Hill.
40. Hersen, M., & Bellack, A. S. (Eds.) (1988). Dictionary of behavioral assessment techniques. New York: Pergamon.
41. Hogan, T. P., Benjamin, A., & Brezinski, K. (2000). Reliability methods: A note on the frequency of use of various types. Educational and Psychological Measurement, 60, 523-531.
42. Hopkins, K. D., Stanley, J. C., & Hopkins, B. R. (1990). Educational and psychological measurement and evaluation (7th ed.). Englewood Cliff, NJ: Prentice Hall.
43. Huck, S. W., & Cormier, W. G. (2001). Reading statistics and research (2nd ed.). New York: HarperCollins.
44. Johnson, O. G. (1976). Tests and measurements in child development: Handbook 11 (Vols. 1-2). San Francisco: Jassey-Bass.
45. Kelly, T. L. (1927). Interpretation of educational measurements. New York: Tarrytown-on-Hudson.
46. Keyser, D. J., & Sweetland, R. C. (Eds.) (1984-1988). Test critiques (Vols. 1-7). Kansas City, MO: Test Corporation of America.
47. Keyser, D. J., & Sweetland, R. C. (Eds.) (1991-1992). Test critiques (Vols. 8-9). Austin, TX: Pro-Ed.
48. Kline, P. (2000). Handbook of psychological testing. New York: Routledge.
49. Lopez, M. N., Charter, R. A., & Newman, J. R. (2000). Psychometric properties of the Halstead Category Test. The Clinical Neuropsychologist, 14, 157-161.
50. Lopez, M. N., Charter, R. A., & Schelling, B. (2000). Interrater reliability of the Career Ability Placement Survey. Journal of Applied Rehabilitation Counseling, 31, 33-35.
51. Maddox, T. (1997). Tests: A comprehensive reference for assessments in psychology, education, and business (4th ed.). Austin, TX: Pro-Ed.
52. Meier, S. T., & Davis, S. R. (1990). Trends in reporting psychometric properties of scores used in counseling psychology research. Journal of Counseling Psychology, 37, 113-115.
53. Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York: McGraw-Hill.
54. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.
55. Pedhazur, E. J., & Schmelkin, L. P. (1991). Measurement, design, and analysis: An integrated approach. Hillsdale, NJ: Erlbaum.
56. Rapport, L. J., Charter, R. A., Dutra, R. L., Farchione, T. J., & Kingsley, J. (1997). Psychometric properties of the Rey-Osterrieth Complex Figure: Lezak-Osterrieth versus Denman Scoring Systems. The Clinical Neuropsychologist, 2, 46-53.
57. Rosenthal, R., & Rosnow, R. L. (1991). Essentials of behavioral research: Methods and data analysis (2nd ed.). New York: McGraw-Hill.
58. Salvia, J., & Ysseldyke, J. (1988). Assessment in special and remedial education (4th ed.). Boston: Houghton Mifflin.
59. Shaw, D. J. (1966). The reliability and validity of the Halstead Category Test. Journal of Clinical Psychology, 22, 176-180.
60. Sternberg, R. J. (Ed.) (1994). Encyclopedia of human intelligence (Vol. 2). New York: Macmillan.
61. Wechsler, D. (1981). Manual for the Wechsler Adult Intelligence Scale-Revised. New York: Psychological Corporation.
62. Weiner, E. A., & Stewart, B. J. (1984). Assessing individuals: Psychological and educational tests and measurement. Boston: Little, Brown.
63. Woodruff, D. J., & Feldt, L. S. (1986). Tests for equality of several alpha coefficients when their sample estimates are dependent. Psychometrika, 51, 393-413.