A comparison of content-balancing procedures for estimating multiple clinical domains in computerized adaptive testing: Relative precision, validity, and detection of persons with misfitting responses

Applied Psychological Measurement

Volumn 34, Issue 6, 2010, Pages 410-423

  Riley, Barth B ^a   Dennis, Michael L ^a   Conrad, Kendon J ^b  

^a CHESTNUT HEALTH SYSTEMS   (United States)

^b UNIVERSITY OF ILLINOIS AT CHICAGO   (United States)

Author keywords

computerized adaptive testing; diagnostic classification; Global Appraisal of Individual Needs (GAIN); Rasch measurement; substance use disorder

Indexed keywords

EID: 77955742755     PISSN: 01466216     EISSN: 15523497     Source Type: Journal    
DOI: 10.1177/0146621609349802     Document Type: Article

References (48)

1. American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (DSM-IV-TR) (4th ed., text rev.). Washington, DC: Author.
2. Bond, T.G., & Fox, C.M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences (2nd ed.). Mahwah, NJ: Lawrence Erlbaum.
3. Briggs, D.C., & Wilson, M. (2003). An introduction to multidimensional measurement using Rasch models. Journal of Applied Measurement, 4, 87-100.
4. Conrad, K.J., Bezruczko, N., Park, H., Chan, Y.F., Riley, B.B., Diamond, G.,et al.(2008). Atypical suicide risk using Rasch person fit statistics to improve screening and understanding. Manuscript in preparation.
5. Conrad, K.J., Conrad, K.M., Dennis, M.L., & Riley, B.B. (2008). GAIN working papers: Rasch analysis of the Internal Mental Distress Scale. Retrieved fromhttp://www.chestnut.org/li/gain/psychometricreports/gainworkingpapersraschanalysisoftheinternalmentaldistressscale.pdf.
6. Dennis, M.L., Chan, Y.-F., & Funk, R. (2006). Development and validation of the GAIN Short Screener (GSS) for internalizing, externalizing, and substance use disorders and crime/violence problems among adolescents and adults. American Journal on Addictions, 15, 80-91.
7. Dennis, M.L., Titus, J.C., White, M.K., Unsicker, J., & Hodgkins, D. (2003). Global appraisal of individual needs: Administration guide for the GAIN and related measures. Bloomington, IL: Chestnut Health Systems.
8. Derogatis, L.R., Lipman, R.S., & Covi, L. (1973). SCL-90: An outpatient psychiatric rating scale- Preliminary report. Psychopharmacology Bulletin, 9, 13-28.
9. Dimitrov, D.M., & Smith, R.M. (2006). Adjusted Rasch person fit statistics. Journal of Applied Measurement, 7, 170-183.
10. Fayers, P.M. (2007). Applying item response theory and computer adaptive testing: The challenges for health outcomes assessment. Quality of Life Research, 16(Suppl. 1), 187-194.
11. Gardner, W., Kelleher, K.J., & Pajer, K.A. (2002). Multidimensional adaptive testing for mental health problems in primary care. Medical Care, 40, 812-823.
12. Gardner, W., Shear, K., Kelleher, K.J., Pajer, K.A., Mammen, O., Buysse, D.,et al.(2004). Computerized adaptive measurement of depression: A simulation study. BMC Psychiatry, 4, 13.
13. Gibbons, R.D., Weiss, D.J., Kupfer, D.J., Frank, E., Fagiolini, A., Grochocinski, V.J.,et al.(2008). Using computerized adaptive testing to reduce the burden of mental health assessment. Psychiatric Services, 59, 361-368.
14. Haley, S.M., Fragala-Pinkham, M., & Ni, P. (2006). Sensitivity of a computer adaptive assessment for measuring functional mobility changes in children enrolled in a community fitness programme. Clinical Rehabilitation, 20, 616-622.
15. Handel, R.W., Ben-Porath, Y.S., & Watt, M. (1999). Computerized adaptive assessment with the MMPI-2 in a clinical setting. Psychological Assessment, 11, 369-380.
16. Kane, M.T., & Brennan, R.L. (1980). Agreement coefficients as indices of dependability for domain referenced tests. Applied Psychological Measurement, 4, 105-126.
17. Karabatsos, G. (2003). Comparing the aberrant response detection performance of thirty-six person fit statistics. Applied Measurement in Education, 16, 277-298.
18. King, D.W., King, L.A., Fairbank, J.A., Schlenger, W.E., & Surface, C.R. (1993). Enhancing the precision of the Mississippi Scale for Combat-Related Posttraumatic Stress Disorder: An application of item response theory. Psychological Assessment, 5, 457-471.
19. Li, M.F., & Olejnik, S. (1997). The power of Rasch person-fit statistics in detecting unusual response patterns. Applied Psychological Measurement, 21, 215-231.
20. Linacre, J.M. (2006). Winsteps Rasch measurement computer program (Version X.X) [Software]. Available fromhttp://www.winsteps.com/.
21. Lipman, R.S., Covi, L., & Shapiro, A.K. (1979). The Hopkins Symptom Checklist (HSCL): Factors derived from the HSCL-90. Journal of Affective Disorders, 1, 9-24.
22. Luecht, R.M. (1996). Multidimensional computerized adaptive testing in a certification or licensure context. Applied Psychological Measurement, 20, 389-404.
23. Luecht, R.M., Champlain, A.D., & Nungester, R.J. (1998). Maintaining content validity in computerized adaptive testing. Advances in Health Sciences Education, 3, 29-41.
24. Meijer, R.R., Molenaar, I.W., & Sijtsma, K. (1994). Item, test and group characteristics and their influence on nonparametric appropriateness measurement. Applied Psychological Measurement, 18, 111-120.
25. Meijer, R.R., & Sijtsma, K. (2001). Methodology review: Evaluating person fit. Applied Psychological Measurement, 25, 107-135.
26. Molenaar, I.W., & Hoijink, H. (1990). The many null distributions of person fit indices. Psychometrika, 55, 75-106.
27. Molenaar, I.W., & Hoijink, H. (1996). Person fit and the Rasch model, with an application of knowledge of logical quantors. Applied Measurement in Education, 9, 27-45.
28. Nering, M.L. (1995). The distribution of person fit using true and estimated person parameters. Applied Psychological Measurement, 19, 121-129.
29. Petersen, M.A., Groenvold, M., Aaronson, N., Fayers, P., Sprangers, M., & Bjorner, J.B. (2006). Multidimensional computerized adaptive testing of the EORTC QLQ-C30: Basic developments and evaluations. Quality of Life Research, 15, 315-329.
30. Reise, S.P., & Due, A.M. (1991). The influence of test characteristics on the detection of aberrant response patterns. Applied Psychological Measurement, 15, 217-226.
31. Riley, B.B., Conrad, K.J., Bezruczko, N., & Dennis, M.L. (2007). Relative precision, efficiency and construct validity of different starting and stopping rules for a computerized adaptive test: The GAIN Substance Problem Scale. Journal of Applied Measurement, 8, 48-64.
32. Sacks, S., Melnick, G., Coen, C., Banks, S., Friedmann, P.D., Grella, C.,et al.(2007). CJDATS Co-occur ring Disorders Screening Instrument (CODSI) for mental disorders (MD): A validation study. Criminal Justice & Behavior, 34, 1198-1215.
33. Segall, D.O. (1996). Multidimensional adaptive testing. Psychometrika, 61, 331-354.
34. Segall, D.O. (2001). General ability measurement: An application of multidimensional item response theory. Psychometrika, 66, 79-97.
35. Smith, R.M., Schumacker, R.E., & Bush, M.J. (1998). Using item mean squares to evaluate fit to the Rasch model. Journal of Outcome Measurement, 2, 66-78.
36. Tseng, F.-L. (2001). Multidimensional adaptive testing using the weighted likelihood estimation. Dissertation Abstracts International Section A: Humanities & Social Sciences, 61(12-A), 4746.
37. van der Linden, W.J. (1999). Multidimensional adaptive testing with a minimum error-variance criterion. Journal of Educational and Behavioral Statistics, 24, 398-412.
38. van Krimpen-Stoop, E.M., & Meijer, R.R. (1999). The null distribution of person-fit statistics for conventional and adaptive tests. Applied Psychological Measurement, 23, 327-345.
39. van Krimpen-Stoop, E.M., & Meijer, R.R. (2000). Detecting person-misfit in adaptive testing using statistical process control techniques. In W. J. van der Linden & C. A. W. Glas (Eds.), Computer adaptive testing: Theory and practice (pp. 1-25). Boston: Kluwer Academic.
40. van Krimpen-Stoop, E.M., & Meijer, R.R. (2001). CUSUM-based person-fit statistics for adaptive testing. Journal of Educational and Behavioral Statistics, 26, 199-218.
41. Veldkamp, B.P. (2002). Multidimensional constrained test assembly. Applied Psychological Measurement, 26, 133-146.
42. Veldkamp, B.P., & van der Linden, W.J. (2002). Multidimensional adaptive testing with constraints on test content. Psychometrika, 67, 575-588.
43. Wainer, H. (2000). Computerized adaptive testing: A primer (2nd ed.). Mahwah, NJ: Lawrence Erlbaum.
44. Wang, W.-C., & Chen, P.-H. (2004). Implementation and measurement efficiency of multidimensional computerized adaptive testing. Applied Psychological Measurement, 28, 295-316.
45. Wang, W.-C., Chen, P.-H., & Cheng, Y.-Y. (2004). Improving measurement precision of test batteries using multidimensional item response models. Psychological Methods, 9, 116-136.
46. Wright, B.D. (1995). Which standard error? Rasch Measurement Transactions, 9(2), 436.
47. Wright, B.D., & Douglas, G.A. (1996). Estimating measures with known item difficulties. Rasch Measurement Transactions, 10(2), 499.
48. Wright, B.D., & Stone, M.H. (1979). Best test design. Chicago: University of Chicago, MESA Press.