Whose iq is it?-assessor bias variance in high-stakes psychological assessment

Psychological Assessment

Volumn 26, Issue 1, 2014, Pages 207-214

  McDermott, Paul A ^a   Watkins, Marley W ^b   Rhoad, Anna M ^a  

^a UNIVERSITY OF PENNSYLVANIA   (United States)

^b BAYLOR UNIVERSITY   (United States)

Author keywords

Assessment; Assessor variance; Measurement bias; Wisc iv

Indexed keywords

ADOLESCENT; ARTICLE; ASSESSOR BIAS VARIANCE; ATTENTION DEFICIT DISORDER; CHILD; CONTROLLED STUDY; EMOTIONAL DISORDER; FEMALE; FULL SCALE IQ; HUMAN; INTELLIGENCE QUOTIENT; LEARNING DISORDER; MAJOR CLINICAL STUDY; MALE; PSYCHOLOGIC ASSESSMENT; PSYCHOLOGIST; RATING SCALE; RELIABILITY; SCHOOL; SCORING SYSTEM; SPECIAL EDUCATION; SPEECH DISORDER; STATISTICAL PARAMETERS; VERBAL COMPREHENSION INDEX; WECHSLER INTELLIGENCE SCALE; WECHSLER INTELLIGENCE SCALE FOURTH EDITION; AFFECTIVE SYMPTOMS; ATTENTION DEFICIT DISORDER WITH HYPERACTIVITY; INTELLECTUAL DISABILITY; INTELLIGENCE; LEARNING DISORDERS; OBSERVER VARIATION; REPRODUCIBILITY; STATISTICS AND NUMERICAL DATA;

ADOLESCENT; AFFECTIVE SYMPTOMS; ATTENTION DEFICIT DISORDER WITH HYPERACTIVITY; CHILD; FEMALE; HUMANS; INTELLECTUAL DISABILITY; INTELLIGENCE; LEARNING DISORDERS; MALE; OBSERVER VARIATION; REPRODUCIBILITY OF RESULTS; WECHSLER SCALES;

EID: 84896706295     PISSN: 10403590     EISSN: 1939134X     Source Type: Journal    
DOI: 10.1037/a0034832     Document Type: Article

References (79)

1. Ackerman, P. L., & Heggestad, E. D. (1997). Intelligence, personality, and interests: Evidence for overlapping traits. Psychological Bulletin, 121, 219-245.
2. Ahearn, E. M. (2009). State eligibility requirements for specific learning disabilities. Communication Disorders Quarterly, 30, 120-128.
3. Allard, G., Butler, J., Faust, D., & Shea, M. T. (1995). Errors in hand scoring objective personality tests: The case of the Personality Diagnostic Questionnaire-Revised (PDQ-R). Professional Psychology: Research and Practice, 26, 304-308.
4. Allard, G., & Faust, D. (2000). Errors in scoring objective personality tests. Assessment, 7, 119-129.
5. Allen, M. J., & Yen, W. M. (2001). Introduction to measurement theory. Prospect Heights, IL: Waveland Press.
6. Babad, E. Y., Mann, M., & Mar-Hayim, M. (1975). Bias in scoring the WISC subtests. Journal of Consulting and Clinical Psychology, 43, 268.
7. Beasley, M. G., Lobasher, M., Henley, S., & Smith, I. (1988). Errors in computation of WISC and WISC-R intelligence quotients from raw scores. Journal of Child Psychology and Psychiatry, 29, 101-104.
8. Belk, M. S., LoBello, S. G., Ray, G. E., & Zachar, P. (2002). WISC-III administration, clerical, and scoring errors made by student examiners. Journal of Psychoeducational Assessment, 20, 290-300.
9. Bergeron, R., Floyd, R. G., & Shands, E. I. (2008). States' eligibility guidelines for mental retardation: An update and consideration of part scores and unreliability of IQs. Education and Training in Developmental Disabilities, 43, 123-131.
10. Bradley, F. O., Hanna, G. S., & Lucas, B. A. (1980). The reliability of scoring the WISC-R. Journal of Consulting and Clinical Psychology, 48, 530-531.
11. Burnham, K. P., & Anderson, D. R. (2004). Multimodel inference: Understanding AIC and BIC in model selection. Sociological Methods and Research, 33, 261-304.
12. Charter, R. A., Walden, D. K., & Padilla, S. P. (2000). Too many simple clerical scoring errors: The Rey Figure as an example. Journal of Clinical Psychology, 56, 571-574.
13. Duckworth, A. L., Quinn, P. D., Lynam, D. R., Loeber, R., & Stouthamer-Loeber, M. (2011). Role of test motivation in intelligence testing. Proceedings of the National Academy of Sciences, 108, 7716-7720.
14. Edwards, O. W., & Rottman, A. (2011). Empirical analysis of the relationship between student examiners' learning with deliberate test practice and examinees' intelligence test performance. Journal of Instructional Psychology, 38, 157-163.
15. Erdodi, L. A., Richard, D. C. S., & Hopwood, C. (2009). Importance of relying on the manual: Scoring error variance in the WISC-IV vocabulary subtest. Journal of Psychoeducational Assessment, 27, 374-385.
16. Etherton, J. L., & Axelxrod, B. N. (2013). Do administration instructions alter optimal neuropsychological test performance? Data from healthy volunteers. Applied Neuropsychology: Adult, 20, 15-19.
17. Flanagan, D. P., & Kaufman, A. S. (2009). Essentials of WISC-IV assessment (2nd ed.). Hoboken, NJ: Wiley.
18. Franklin, M. R., Jr., Stillman, P. L., Burpeau, M. Y., & Sabers, D. L. (1982). Examiner error in intelligence testing: Are you a source? Psychology in the Schools, 19, 563-569.
19. Fuchs, D., & Fuchs, L. S. (1986). Test procedure bias: A meta-analysis of examiner familiarity effects. Review of Educational Research, 56, 243- 262.
20. Fuchs, D., & Fuchs, L. S. (1989). Effects of examiner familiarity on Black, Caucasian, and Hispanic children: A meta-analysis. Exceptional Children, 55, 303-308.
21. Fuchs, D., Fuchs, L. S., Garwick, D. R., & Featherstone, N. (1983). Test performance of language-handicapped children with familiar and unfamiliar examiners. Journal of Psychology: Interdisciplinary and Applied, 114, 37-46.
22. Fuchs, D., Fuchs, L. S., & Power, M. H. (1987). Examiner familiarity on LD and MR students' language performance. Remedial and Special Education, 8, 47-52.
23. Fuchs, D., Fuchs, L. S., Power, M. H., & Dailey, A. M. (1985). Bias in the assessment of handicapped children. American Educational Research Journal, 22, 185-198.
24. Garb, H. N. (2010). The social psychology of clinical judgment. In J. E. Maddux & J. P. Tangney (Eds.), Social psychological foundations of clinical practice (pp. 297-311). New York, NY: Guilford Press.
25. Glass, L. A., Ryan, J. J., & Charter, R. A. (2010). Discrepancy score reliabilities in the WAIS-IV standardization sample. Journal of Psychoeducational Assessment, 28, 201-208.
26. Glass, L. A., Ryan, J. J., & Charter, R. A., & Bartels, J. M. (2009). Discrepancy score reliabilities in the WISC-IV standardization sample. Journal of Psychoeducational Assessment, 27, 138-144.
27. Glutting, J. J., Oakland, T., & Konold, T. R. (1994). Criterion-related bias with the Guide to Assessment of Test-Session Behavior for the WISC-III and WIAT: Possible ethnicity, gender, and SES effects. Journal of School Psychology, 32, 355-369.
28. Goddard, R., Simons, R., Patton, W., & Sullivan, K. (2004). Psychologist hand-scoring error rates on the Rothwell-Miller Interest Blank: A comparison of three job allocation systems. Australian Journal of Psychology, 56, 25-32.
29. Gresham, F. M. (2009). Interpretation of intelligence test scores in Atkins cases: Conceptual and psychometric issues. Applied Neuropsychology, 16, 91-97.
30. Groth-Marnat, G. (2009). Handbook of psychological assessment (5th ed.). Hoboken, NJ: Wiley.
31. Hanna, G. S., Bradley, F. O., & Holen, M. C. (1981). Estimating major sources of measurement error in individual intelligence scales: Taking our heads out of the sand. Journal of School Psychology, 19, 370-376.
32. Hofmann, D. A., & Gavim, M. B. (1998). Centering decisions in hierarchical linear models: Implications for research in organizations. Journal of Management, 24, 623-641.
33. Hoyt, W. T., & Kerns, M.-D. (1999). Magnitude and moderators of bias in observer ratings: A meta-analysis. Psychological Methods, 4, 403-424.
34. Kamphaus, R., Worrell, F. C., & Harrison, P. (2005). Principles for evaluation and eligibility determination for specific learning disabilities: A report of the ad hoc committee of division 16. The School Psychologist, 59, 157-159.
35. Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50, 1-73.
36. Kozey, M., & Siegel, L. S. (2008). Definitions of learning disabilities in Canadian provinces and territories. Canadian Psychology, 49, 162-171.
37. Kozora, E., Kongs, S., Hampton, M., & Zhang, L. (2008). Effects of examiner error on neuropsychological test results in a multi-site study. Clinical Neuropsychologist, 22, 977-988.
38. Kuentzel, J. G., Hetterscheidt, L. A., & Barnett, D. (2011). Testing intelligently includes double-checking Wechsler IQ scores. Journal of Psychoeducational Assessment, 29, 39 - 46.
39. Lee, D., Reynolds, C. R., & Willson, V. L. (2003). Standardized test administration: Why bother? Journal of Forensic Neuropsychology, 3, 55-81.
40. Legris, Y. (2004). Development of a new teaching method to reduce scoring errors on the WISC-III and WAIS-III. Dissertation Abstracts International: Section A. Humanities and Social Sciences, 65(1-A), 68.
41. Littell, R. C., Milliken, G. A., Stroup, W. W., Wolfinger, R. D., & Schabenberger, O. (2006). SAS systems for mixed models (2nd ed.). Cary, NC: SAS Institute.
42. LoBello, S. G., & Holley, G. (1999). WPPSI-R administration, clerical, and scoring errors by student examiners. Journal of Psychoeducational Assessment, 17, 15-23.
43. Loe, S. A., Kadulbek, R. M., & Marks, W. J. (2007). Administration and scoring errors on the WISC-IV among graduate student examiners. Journal of Psychoeducational Assessment, 25, 237-247.
44. Mascolo, J. T. (2009). Appendix 1: Linking WISC-IV assessment results to educational strategies and instructional supports. In D. Flanagan & A. Kaufman (Eds.), Essentials of WISC-IV assessment (2nd ed., pp. 1-17). Hoboken, NJ: Wiley.
45. Matthey, S., Lee, C., Črnčec, R., & Trapolini, T. (2013). Errors in scoring the Edinburgh Postnatal Depression scale. Archives of Women's Mental Health, 16, 117-122.
46. McClain, M.-C., & Pfeiffer, S. (2012). Identification of gifted students in the United States today: A look at state definitions, policies, and practices. Journal of Applied School Psychology, 28, 59-88.
47. McDermott, P. A., Fantuzzo, J. F., Waterman, C., Angelo, L. A., Warley, H. W., Gadsden, V. L., & Zhang, X. (2009). Measuring preschool cognitive growth while it's still happening: The Learning Express. Journal of School Psychology, 47, 337-366.
48. Moon, G. W., Blakey, W. A., Gorsuch, R. L., & Fantuzzo, J. W. (1991). Frequent WAIS-R administration errors: An ignored source of inaccurate measurement. Professional Psychology: Research and Practice, 22, 256-258.
49. Mrazik, M., Janzen, T. M., Dombrowski, S. C., Barford, S. W., & Krawchuk, L. L. (2012). Administration and scoring errors of graduate students learning the WISC-IV: Issues and controversies. Canadian Journal of School Psychology, 27, 279 -290.
50. Oakland, T., Lee, S. W., & Axelrad, K. M. (1975). Examiner differences on actual WISC protocols. Journal of School Psychology, 13, 227-233.
51. Patterson, M., Slate, J. R., Jones, C. H., & Steger, H. S. (1995). The effects of practice administrations in learning to administer and score the WAIS-R: A partial replication. Educational and Psychological Measurement, 55, 32-37.
52. Phay, A. J. (1990). Shipley Institute of Living Scale: Part 1-Moderator variables. Medical Psychotherapy, 3, 1-15.
53. Plumb, G. R. (1955). Scoring difficulty of Wechsler comprehension responses. Journal of Educational Psychology, 46, 179-183.
54. Prifitera, A., Saklofske, D. H., & Weiss, L. G. (2008). WISC-IV clinical assessment and intervention (2nd ed.). San Diego, CA: Academic Press.
55. Ramos, E., Alfonso, V. C., & Schermerhorn, S. M. (2009). Graduate students' administration and scoring errors on the Woodcock-Johnson III tests of cognitive abilities. Psychology in the Schools, 46, 650-657.
56. Raudenbush, S. W., & Sadoff, S. (2008). Statistical inference when classroom quality is measured with error. Journal of Research on Educational Effectiveness, 1, 138-154.
57. Reynolds, C. R., & Milam, D. A. (2012). Challenging intellectual testing results. In D. Faust (Ed.), Coping with psychiatric and psychological testimony (6th ed., pp. 311-334). New York, NY: Oxford University Press.
58. Ruscio, J. (2007). The clinician as subject. In S. O. Lilienfeld & W. T. O'Donohue (Eds.), The great ideas of clinical science: 17 principles that every mental health professional should understand (pp. 29-47). New York, NY: Routledge.
59. Ryan, J. J., & Schnakenberg-Ott, S. D. (2003). Scoring reliability on the Wechsler Adult Intelligence Scale-Third Edition (WAIS-III). Assessment, 10, 151-159.
60. SAS Institute. (2011). SAS version 9.3 for Windows [Computer program]. Cary, NC: Author.
61. Sattler, J. M. (2008). Assessment of children: Cognitive foundations (5th ed.). San Diego, CA: Author.
62. Sattler, J. M., Winget, B. M., & Roth, R. J. (1969). Scoring difficulty of WAIS and WISC comprehension, similarities, and vocabulary responses. Journal of Clinical Psychology, 25, 175-177.
63. Schafer, K., De Santi, S., & Schneider, L. S. (2011). Errors in ADAS-cog administration and scoring may undermine clinical trials results. Current Alzheimer Research, 8, 373-376.
64. Simons, R., Goddard, R., & Patton, W. (2002). Hand-scoring error rates in psychological testing. Assessment, 9, 292-300.
65. Slate, J. R., & Chick, D. (1989). WISC-R examiner errors: Cause for concern. Psychology in the Schools, 26, 78-84.
66. Slate, J. R., & Jones, C. H. (1990). Identifying students' errors in administering the WAIS-R. Psychology in the Schools, 27, 83-87.
67. Slate, J. R., Jones, C. H., Murray, R. A., & Coulter, C. (1993). Evidence that practitioners err in administering and scoring the WAIS-R. Measurement and Evaluation in Counseling and Development, 25, 156-161.
68. Snijders, T., & Baker, R. (1999). Multilevel analysis. Thousand Oaks, CA: Sage.
69. Spruill, J., Oakland, T., & Harrison, P. (2005). Assessment of mental retardation. In A. Prifitera, D. H. Saklofske, & L. G. Weiss (Eds.), WISC-IV clinical use and interpretation: Scientist-practitioner perspectives (pp. 299 -331). San Diego, CA: Elsevier.
70. Sullivan, K. (2000). Examiners' errors on the Wechsler Memory Scale-Revised. Psychological Reports, 87, 234-240.
71. Szarko, J. E., Brown, A. J., & Watkins, M. W. (2013). Examiner familiarity effects for children with autism spectrum disorders. Journal of Applied School Psychology, 29, 37-51.
72. Terman, L. M. (1918). Errors in scoring Binet tests. Psychological Clinic, 12, 33-39.
73. Thorndike, R. M., & Thorndike-Christ, T. (2010). Measurement and evaluation in psychology and education (8th ed.). Boston, MA: Pearson.
74. Viswanathan, M. (2005). Measurement error and research design. Thousand Oaks, CA: Sage.
75. Wasserman, J. D., & Bracken, B. A. (2013). Fundamental psychometric considerations in assessment. In J. R. Graham & J. A. Naglieri (Eds.), Handbook of psychology: Assessment psychology (Vol. 10, 2nd ed., pp. 50-80). Hoboken, NJ: Wiley.
76. Waterman, C., McDermott, P. A., Fantuzzo, J. W., & Gadsden, V. L. (2012). The matter of assessor variance in early childhood education- Or whose score is it anyway? Early Childhood Research Quarterly, 27, 46-54.
77. Wechsler, D. (2003). Interpretive considerations. In WISC-IV technical and interpretive manual (pp. 99-108). San Antonio, TX: Psychological Corporation.
78. Weiss, L. G., Saklofske, D. H., Prifitera, A., & Holdnack, J. A. (2006). WISC-IV advanced clinical interpretation. San Diego, CA: Academic Press.
79. Wolfe-Christensen, C., & Callahan, J. L. (2008). Current state of standardization adherence: A reflection of competency in psychological assessment. Training and Education in Professional Psychology, 2, 111-116.