Validity Issues in Computer-Based Testing

Educational Measurement: Issues and Practice

Volumn 20, Issue 3, 2001, Pages 16-25

  Huff, Kristen L ^a   Sireci, Stephen G ^a  

^a UNIVERSITY OF MASSACHUSETTS   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84955517491     PISSN: 07311745     EISSN: 17453992     Source Type: Journal    
DOI: 10.1111/j.1745-3992.2001.tb00066.x     Document Type: Article

References (68)

1. American Council on Education. (1995) Guidelines for computer-adaptive test development and use in education. Washington, DC: Author.
2. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999) Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
3. Bejar, I. (1991) A methodology for scoring open-ended architectural design problems. Journal of Applied Psychology, 76, 522–532.
4. Bennett, R. E. (1999) Using new technology to improve assessment. Educational Measurement: Issues and Practice, 18 (3), 5–12.
5. Bennett, R. E., Morley, M., Quardt, D., Rock, D. A., Singley, M. K., Katz, I. R., & Nhouyvanisvong, A. (1999) Psychometric and cognitive functioning of an underdetermined computer-based response type for quantitative reasoning. Journal of Educational Measurement, 36, 233–252.
6. Bennett, R. E., & Rock, D. A. (1995) Generalizability, validity, and examinee perceptions of a computer-delivered formulating-hypotheses test. Journal of Educational Measurement, 32, 19–36.
7. Bennett, R. E., & Sebrechts, M. M. (1997) A computer-based task for measuring the representational component of quantitative proficiency. Journal of Educational Measurement, 34, 64–77.
8. Bock, R. D., Muraki, E., & Pfeiffenberger, W. (1988) Item pool maintenance in the presence of item parameter drift. Journal of Educational Measurement, 25, 275–285.
9. Booth, J. F. (1991) The key to valid computer-based testing: The use interface. European Review of Applied Psychology, 41, 281–293.
10. Booth, J. F. (1998a.) The user interface in computer-based selection and assessment: Applied and theoretical problematics of an evolving technology. International Journal of Selection & Assessment, 6, 61–81.
11. Booth, J. F. (1998b.) Uses of PC technology in selection and assessment. International Journal of Selection & Assessment, 6, 57–60.
12. Brittingham, P., Morley, M. E., Singley, M. K., Zelman, M. G., Jha, K. N., Fife, J. H., Rarich, R. L., Katz, I. R., & Bennett, R. E. (2000) The mathematics test creation assistant: A tool for automated item generation. http://www.ncme.org/about/awards/bennett.html.
13. Clauser, B. E., Ross, L. P., Clyman, S. G., Rose, K. M., Margolis, M. J., Nungester, R. J., Piemme, T. E., Chang, L., ElBayoumi, G., Malakoff, G. L., & Pincetl, P. S. (1997) Development of a scoring algorithm to replace expert rating for scoring a complex performance-based assessment. Applied measurement in Education, 10, 345–358.
14. Clauser, B. E., Subhiyah, R. G., Nungester, R. J., Ripkey, D. R., Clyman, S. G., & McKinley, D. (1995) Scoring a performance-based assessment by modeling the judgments of experts. Journal of Educational Measurement, 32, 397–415.
15. Cronbach, L. J. (1971) Test validation. In R. L. Thorndike, (Ed.), Educational measurement (2nd ed., pp. 443–507). Washington, DC: American Council on Education.
16. Darling-Hammond, L., & Lieberman, A. (1992) The shortcomings of standardized tests. The Chronicle of Higher Education, January 29, B1-B2.
17. Drasgow, F., & Olson-Buchanan, J. B. (Eds.). (1999) Innovations in Computerized Assessment. Mahwah, NJ: Erlbaum.
18. Drasgow, F., Olson-Buchanan, J. B., & Moberg, P. J. (1999) Development of an interactive video assessment: Trials and tribulations. In F. Drasgow and J. B. Olson-Buchanan, (Eds.), Innovations in computerized assessment (pp. 177–196). Mahwah, NJ: Lawrence-Erlbaum Associates.
19. Embretson, S. E. (1999) Generating items during testing: Psychometric issues and models. Psychometrika, 64, 407–433.
20. Fischer, G.H. (1974) Einfuhrung in die theorie psychologischer tests. [Introduction to mental test theory. In German]. Berne: Huber.
21. Fischer, G. H., & Formann, A. K. (1982) Some applications of logistic latent trait models with linear constraints on the parameters. Applied Psychological Measurement, 6, 397–416.
22. Fischer, G. H., & Pendl, P. (1980) Individualized testing on the basis of the dichotomous Rasch model. In L. J.Th. van der Kamp, W. F. Langerak, & D. N. M. de Gruijter, (Eds.), Psychometrics for educational debates. New York: Wiley.
23. Fitzgerald, C. T. (1998, August), Test adaptation in a large-scale certification program: Challenges, processes, and lessons learned. Paper presented at the annual meeting of the American Psychological Association, San Francisco, CA.
24. Gershon, R. C., & Bergstrom, B. (April, 1991). Individual differences in computer adaptive testing: Anxiety, computer literacy, and satisfaction. Paper presented at the National Council on Measurement in Education, San Francisco.
25. Greaud, V., & Green, B. F. (1986) Equivalence of conventional and computer presentation of speed tests. Applied Psychological Measurement, 10, 23–34.
26. Green, B. F. (1988) Construct validity of computer-based tests. In H. Wainer & H. Braun, (Eds.), Test validity (77–86). Hillsdale, NJ: Erlbaum.
27. Hambleton, R. K. (1997, October) Promising GMAT item formats for the 21st century. Invited presentation at the international workshop on the GMAT, Paris, France.
28. Hambleton, R. K., & Sireci, S. G. (1997) Future directions for norm-referenced and criterion-referenced educational assessments. International Journal of Educational Research, 27(5), 379–393.
29. Jodoin, M. G. An empirical examination of IRT information for innovative item formats in a computer-based certification testing program (Laboratory of Psychometric and Evaluative Methods Research Report No. 417). Amherst, MA: School of Education, University of Massachusetts.
30. Kane, M. T. (1992) An argument-based approach to validity. Psychological Bulletin, 112, 527–535.
31. Linn, R. L. (1997) Evaluating the validity of assessments: The consequences of use. Educational Measurement: Issues and Practice, 16 (2), 14–16.
32. Linn, R. L. (1998) Partitioning responsibility for the evaluation of the consequences of assessment programs. Educational Measurement: Issues and Practice, 17 (2), 28–30.
33. Luecht, R. M., & Clauser, B. E. (1998, September) Test models for complex computer-based testing. Paper presented at the ETS Computer-based Testing Colloquium, Philadelphia, PA.
34. Martinez, M. E., & Bennett, R. E. (1992) A review of automatically scorable constructed-response item types for large-scale assessment. Applied Measurement in Education, 5, 151–169.
35. McBride, J. R., & Martin, J. T. (1983) Reliability and validity of adaptive ability tests in a military setting. In D. J. Weiss, (Ed.), New horizons in testing: Latent trait test theory and computerized adaptive testing (pp. 223–236). New York: Academic Press.
36. Messick, S. (1989) Validity. In R. Linn, (Ed.), Educational measurement (3rd ed., pp. 13–103). Washington, DC: American Council on Education.
37. Messick, S. (1992) The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23, 12–23.
38. Mills, C. (2000, February) Unlocking the promise of CBT. Keynote address presented at the annual meeting of the Association of Test Publishers, Carmel Valley, CA.
39. Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (1999, January) On the roles of task model variables in assessment design. Los Angeles, CA: Center for the Study of Evaluation, CRESST, UCLA. Available online at http://www.cse.ucla.edu/CRESST/Reports/TECH500.pdf.
40. Parshall, C. G. (1999, April) Audio CBTs: Measuring more through the use of speech and non-speech sound. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal.
41. Parshall, C. G., Davey, T., & Pashley, P. (2000) Innovative item types for computerized testing. In W. J. van der Linden, C. Glas, (Eds.), Computer-adaptive testing: Theory and practice. Boston, MA: Kluwer Academic.
42. Resnick, L. B., & Resnick, D. P. (1989, October) Tests as standards of achievement in schools. Paper presented at the Invitational Conference of the Educational Testing Service, New York, NY.
43. Robin, F. (2000) Development and evaluation of test assembly procedures for computerized adaptive testing (Laboratory of Psychometric and Evaluative Methods Research Report No. 391). Amherst, MA: School of Education, University of Massachusetts.
44. Sacks, P. (2000) Standardized minds: The high price of America's testing culture and what we can do to change it. Cambridge, MA: Perseus Books.
45. Sands, W. A., Waters, B. K., & McBride, J. R. (Eds.). (1997) Computerized adaptive testing: from inquiry to operation. Washington, DC: American Psychological Association.
46. Schnipke, D. L., & Scrams, D. J. (September, 1998). Exploring issues of examinee behavior: Insights gained from response-time analysis. Paper presented at the ETS Computer-based Testing Colloquium, Philadelphia, PA.
47. Shepard, L. A. (1993) Evaluating test validity. Review of Research in Education, 19, 405–450.
48. Sireci, S. G. (1998) The construct of content validity. Social Indicators Research, 45, 83–117.
49. Steffen, M., & Way, W. D. (1999, April) Test-taking strategies in computerized adaptive testing. Paper presented at the meeting of the National Council on Measurement in Education, Montreal.
50. Steinberg, L., Thissen, D., & Wainer, H. (2000) Validity. In Wainer, H., (Ed.), Computerized-adaptive testing: A primer (2nd ed., pp. 185–229). Mahwah, NJ: Erlbaum.
51. Stocking, M. L., & Swanson, L. (1993) A method for severely constrained item selection in adaptive testing. Applied Psychological Measurement, 17, 277–292.
52. Swanson, L., & Stocking, M. L. (1993) A model and heuristic for solving very large selection problems. Applied Psychological Measurement, 17, 151–166.
53. Sympson, J. B., Weiss, D. J., & Ree, M. J. (1983) A validity comparison of adaptive testing in a military technical training environment (AFHLR-TR-81-40). Brooks AFB, TX: Manpower and Personnel Division, Air Force Human Resources Laboratory.
54. Sympson, J. B., Weiss, D. J., & Ree, M. J. (1984, April) Predictive validity of computerized adaptive testing in a military training environment. Paper presented at the meeting of the American Educational Research Association, New Orleans.
55. Taylor, C., Kirsch, I., Eignor, D., & Jamieson, J. (1999) Examining the relationship between computer familiarity and performance on computer-based language tasks. Language Learning, 49, 219–274.
56. van der Linden, W. J. (1998a, September). On complexity in computer-based testing Paper presented at the ETS Computer-based Testing Colloquium, Philadelphia, PA.
57. van der Linden, W. J. (1998b). Optimal assembly of psychological and educational tests. Applied Psychological Measurement, 22, 195–211.
58. van der Linden, W. J., Scrams, D. J., & Schnipke, D.L. (1999) Using response-time constraints to control for differential speededness in computerized adaptive testing. Applied Psychological Measurement, 23, 195–210.
59. Veldkamp, B. P. (1999) Multiple objective test assembly problems. Journal of Educational Measurement, 36, 253–266.
60. Vispoel, W. P., Wang, T., & Bleiler, T. (1997) Computerized adaptive and fixed-item testing of music listening skill: A comparison of efficiency, precision, and concurrent validity. Journal of Educational Measurement, 34, 43–63.
61. Wainer, H. (1993) Some practical considerations when converting a linearly administered test to an adaptive format. Educational Measurement: Issues and Practice, 12, 15–20.
62. Wainer, H. (Ed.). (2000) Computerized adaptive testing: A primer (2nd ed.). Mahwah, NJ: Lawrence-Erlbaum Associates.
63. Wainer, H., & Kiely, G. L. (1987) Item clusters and computerized adaptive testing: A case for testlets. Journal of Educational Measurement, 24, 185–201.
64. Wainer, H., & Lewis, C. (1990) Towards a psychometrics for testlets. Journal of Educational Measurement, 27, 1–14.
65. Williamson, D. M., Bejar, I. I., & Hone, A. S. (1999) ‘Mental Model’ comparison of automated and human scoring. Journal of Educational Measurement, 36, 158–184.
66. Wise, S. L. (1996, April) A critical analysis of the arguments for and against item review in computerized adaptive testing. Paper presented at the meeting of the National Council on Measurement in Education, New York.
67. Zara, A. (1997, March) Administering and scoring the computerized adaptive test. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago, IL.
68. Zenisky, A. L., & Sireci, S. G. (in press). Technological innovations in large-scale assessment. Applied Measurement in Education.