Test design with cognition in mind

Educational Measurement: Issues and Practice

Volumn 25, Issue 4, 2006, Pages 21-35

  Gorin, Joanna S ^a  

^a ARIZONA STATE UNIVERSITY   (United States)

Author keywords

Assessment design; Congnitively based assessment; Item writing; Test construction; Validity

Indexed keywords

EID: 33846424986     PISSN: 07311745     EISSN: 17453992     Source Type: Journal    
DOI: 10.1111/j.1745-3992.2006.00076.x     Document Type: Article

References (72)

1. Behrens, J. T., Mislevy, R. J., Bauer, M., Williamson, D. M., & Levy, R. (2004). Introduction to evidence-centered design and lessons learned from its application in a global e-learning program. International Journal of Testing, 4, 295-302.
2. Bejar, I. I. (1993). A generative approach to psychological and educational measurement. In N. Fredriksen, R. J. Mislevy, & I. I. Bejar (Eds.), Test theory for a new generation of tests (pp. 323-359). Hillsdale, NJ: Erlbaum.
3. Bejar, I. I., Lawless, R. R., Morley, M. E., Wagner, M. E., Bennett, R. E., & Revuelta, J. (2002). A feasibility study of in-the-fly item generation in adaptive testing (GRE Board Professional Rep. No. 98-12P). Princeton, NJ: Educational Testing Service.
4. Bennett, R. E. (1999). Using new technology to improve assessment. Educational Measurement: Issues and Practice, 18, 5-12.
5. Bennett, R. E., & Bejar, I. I. (1998). Validity and automated scoring: It's not only the scoring. Educational Measurement: Issues and Practice, 17(4), 9-16.
6. Bennett, R. E., Jenkins, F., Persky, H., & Weiss, A. (2003). Assessing complex-problem solving performances (ETS Report No. RM-03-03). Princeton, NJ: Educational Testing Service.
7. Briggs, D. C., Alonzo, A. C., Schwab, C., & Wilson, M. (2006). Diagnostic assessment with ordered multiple-choice items. Educational Assessment, 11(1), 33-63.
8. Buck, G. & Tatsuoka, K. (1998). Application of the rule-space procedure to language testing: Examining attributes of a free response listening test. Language Testing, 15(2), 119-157.
9. Buck, G., Tatsuoka, K., & Kostin, I. (1997). The subskills of reading: Rule-space analysis of a multiple choice test of second language reading comprehension. Language Learning, 47(3), 423-466.
10. Chen, Y.-H., Gorin, J. S., Thompson, M., & Tatsuoka, K. K. (2006). Cognitively diagnostic examination of Taiwanese mathematics achievement on TIMSS-1999. Unpublished doctoral dissertation. Tempe, AZ: Arizona State University.
11. Cooke, N. J. (1994). Varieties of knowledge elicitation techniques. International Journal of Human Computer Studies, 41, 801-849.
12. Desimone, L., & LeFloch, K. (2004). Are we asking the right questions? Using cognitive interviews to improve surveys in education research. Educational Evaluation and Policy Analysis, 26(1), 1-22.
13. DeVellis, R. F. (1991). Scale development: Theory and applications. Thousand Oaks, CA: Sage Publications.
14. Diehl, K. A. (2004). Algorithmic item generation and problem solving strategies in matrix completion problems. Dissertation Abstracts International: Section B: The Sciences and Engineering, 64, 4075.
15. Educational Testing Service (Ed.) (1998). GRE: Practicing to the general test: Big book. Princeton, N.J.: Educational Testing Service.
16. Embretson, S. E. (1994). Application of cognitive design systems to test development. In C. R. Reynolds (Ed.), Cognitive assessment: A multidisciplinary perspective (pp. 107-135). New York: Plenum Press.
17. Embretson, S. E. (1998). A cognitive design system approach to generating valid tests: Application to abstract reasoning. Psychological Methods, 3, 300-396.
18. Embretson, S. E. (1999). Generating items during testing: Psychometric issues and models. Psychometrika, 64, 407-433.
19. Embretson, S. E., & Gorin, J. S. (2001). Improving construct validity with cognitive psychology principles. Journal of Educational Measurement, 38(4), 343-368.
20. Embretson, S. E., & Wetzel, C. D. (1987). Component latent trait models for paragraph comprehension. Applied Psychological Measurement, 11, 175-193.
21. Enright, M. K., Morely, M. & Sheehan, K. M. (2002). Items by design: The impact of systematic variation on item statistical characteristics. Applied Measurement in Education, 15(1), 49-74.
22. Enright, M. K., & Sheehan, K. M. (2002). Modeling the difficulty of quantitative reasoning items: Implications for item generation. In S. H. Irvine & P. C. Kyllonen (Eds.), Item generation for test development (pp. 129-157). Mahwah, NJ: Lawrence Erlbaum Associates.
23. Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data (Rev. ed.). Cambridge, MA: MIT Press.
24. Ferrara, S., & DeMauro, G. E. (2006). Standardized assessment of individual achievement in K-12. In R. L. Brennan (Ed.), Educational measurement (4th ed.). Westport, CT: American Council on Education/Praeger.
25. Ferrara, S., Duncan, T. G., Freed, R., Vélez-Paschke, A., McGivern, J., Mushlin, S., Mattessich, A., Rogers, A., & Westphalen, K. (2004). Examining test score validity by examining item construct validity: Preliminary analysis of evidence of the alignment of targeted and observed content, skills, and cognitive processes in a middle school science assessment. Paper presented at the 2004 Annual Meeting of the American Educational Research Association.
26. Fischer, G. H. (1973). Linear logistic test model as an instrument in educational research. Acta Psychologica 37, 359-374.
27. Frederiksen, N. (1986). Construct validity and construct similarity: Methods for use in test development and test validation. Multivariate Behavioral Research, 21(1), 3-28.
28. Frederiksen, N. (1990). Introduction. In N. Frederiksen, R. Glaser, A. Lesgold, & M. G. Shafto (Eds.), Diagnostic monitoring of skill and knowledge acquisition (pp. ix-xvii). Hillsdale, NJ: Lawrence Erlbaum.
29. Gierl, M. J., Tang, X., & Wang, C. (2005). Identifying content and cognitive dimensions on the SAT (College Board Research Report No. 2005-11). New York: College Board Press.
30. Gorin, J. S. (2005). Manipulation of processing difficulty on reading comprehension test questions: The feasibility of verbal item generation. Journal of Educational Measurement, 42, 351-373.
31. Gorin, J. S. (2006). Using alternative data sources to inform item difficulty modeling, Paper presented at the 2006 Annual Meeting of the National Council on Educational Measurement.
32. Gorin, J. S., & Embretson, S. E. (2006). Item difficulty modeling of paragraph comprehension items. Applied Psychological Measurement, 30(5), 394-411.
33. Graf, E. A., Peterson, S., Steffen, M., & Lawless, R. (2005). Psychometric and cognitive analysis as a basis for the design and revision of quantitative item models. (ETS Research Report No. RR-05-25). Princeton, NJ: Educational Testing Service.
34. Haladyna, T. M. (2004). Developing and validating multiple-choice test items. Mahwah, NJ: Lawrence Erlbaum Associates.
35. Hartz, S. M. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. Dissertation Abstracts International: Section B: The Sciences and Engineering, 63(2-B), 864.
36. Irvine, S. H. (2002). The foundations for item generation for mass testing. In S. H. Irvine & P. C. Kyllonen (Eds.), Item generation for test development (pp. 3-24). Mahwah, NJ: Erlbaum.
37. Ivie, J. L., Kupzyk, K. A., & Embretston, S. E. (2004). Final report of Cognitive Components Study - Predicting strategies for solving multiple-choice quantitative reasoning items: An eyetracker study. Princeton, NJ: Educational Testing Service and Lawrence, KS: University Press of Kansas.
38. Katz, I. R., Bennett, E., & Berger, A. E. (2000). Effects of response format on difficulty of SAT-Mathematics items: It's not the strategy. Journal of Educational Measurement, 37, 39-57.
39. Katz, S., & Lautenschlager, G. J. (1994). Answering reading comprehension items without passages in the SAT-I, the ACT, and the GRE. Educational Assessment, 2, 295-308.
40. Katz, S., & Lautenschlager, G. J. (2001). The contribution of passage and no-passage factors to item performance on the SAT reading task. Educational Assessment, 7, 165-176.
41. Kintsch, W., & vanDijk, A. (1978). Toward a model of text comprehension and production. Psychological Review, 85, 363-394.
42. Lane, S. (2004). Validity of high-stakes assessment: Are students engaged in complex thinking? Educational Measurement: Issues and Practice, 23(3), 6-14.
43. Leighton, J. P. (2004). Avoiding misconception, misuse, and missed opportunities: The collection of verbal reports in educational achievement testing. Educational Measurement: Issues and Practice, 23(4), 6-15.
44. Messick, S. (1975). The standard problem: Meaning and values in measurement and evaluation. American Psychologist, 30, 955-966.
45. Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741-749.
46. Mislevy, R. J. (1994). Evidence and inference in educational assessment. Psychometrika, 59, 439-468.
47. Mislevy, R. J. (1995). Probability-based inference in Cognitive Diagnosis. In P. D. Nichols, S. F. Chipman, & R. L. Brennan (Eds.) Cognitively Diagnostic Assessment. Hillsdale, NJ: Lawrence Erlbaum Associates.
48. Mislevy, R. J., Steinberg, L. S., Breyer, F. J., Almond, R. G., & Johnson, L. (2002). Making sense of data from complex assessments. Applied Measurement in Education, 25, 363-389.
49. National Research Council (2001). Knowing what students know: The science and design of educational assessment. Washington, DC: National Academy Press.
50. Netemeyer, R. G., Bearden, W. O., & Sharma, S. (2003). Scaling procedures: Issues and applications. Thousand Oaks, CA: Sage Publications.
51. Pek, P. K., & Poh, K. L. (2004). A Bayesian tutoring system for Newtonian mechanics: Can it adapt to different learners? Journal of Educational Computing Research, 31 (3), 281-307.
52. Powers, D. E., & Wilson, S. T. (1993). Passage dependence of the New SAT reading comprehension questions (College Board Report No. 93-3). New York: College Board.
53. Quasha, W. H., & Likert, R. (1937). The revised Minnesota paper form board test. Journal of Educational Psychology, 28, 197-204.
54. Raven, J. C., (1941). Standardisation of progressive matrices. British Journal of Medical Psychology, 19, 137-150.
55. Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124, 372-422.
56. Rayner, K., Warren, T., Juhasz, B. J., & Liversedge, S. P. (2004). The effect of plausibility on eye movements in reading. Journal of Experimental Psychology: Learning, Memory, & Cognition, 30, 1290-1301.
57. Sheehan, K. M., & Ginther, A. (2001). What do passage-based multiple-choice verbal reasoning items really measure? An analysis of the cognitive skills underlying performance on the current TOEFL reading section. Paper presented at the 2000 Annual Meeting of the National Council of Measurement in Education.
58. Sheehan, K., Kostin, I., & Persky, H. (2006). Predicting item difficulty as a function of inferential processing requirements: An examination of the reading skills underlying performance on the NAEP grade 8 reading assessment. Paper presented at the 2006 Annual Meeting of the National Council of Measurement in Education.
59. Shute, V. J., Graf, E. A., & Hansen, E. G. (2006). Designing adaptive, diagnostic math assessments for sighted and visually disabled students (ETS Research Report No. RR-06-01). Princeton, NJ: Educational Testing Service.
60. Sireci, S. G., & Zenisky, A. L. (2006). Innovative item formats in computer-based testing: In pursuit of improved construct representation. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of test development. Mahwah, NJ: Lawrence Erlbaum Associates.
61. Snow, R. E., & Lohman, D. F. (1989). Implication of cognitive psychology for education measurement. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 263-331). New York: Macmillan.
62. Tatsuoka, K. K. (1985). A probabilistic model for diagnosing misconceptions by the pattern classification approach. Journal of Educational Statistics, 10(1), 55-73.
63. Tatsuoka, K. K. (1995). Architecture of knowledge structures and cognitive diagnosis: A statistical pattern recognition and classification approach. In P. D. Nichols, S. F. Chipman, & R. L. Brennan (Eds.) Cognitively Diagnostic Assessment. Hillsdale, NJ: Lawrence Erlbaum Associates.
64. Tatsuoka, K. K., Corter, J. E., & Guerrero, A. (2004). Coding manual for identifying involvement of content, skill, and process subskills for the TIMSS-R 8th grade and 12th grade general mathematics test items. Technical Report. New York: Department of Human Development, Teachers College, Columbia University.
65. Tatsuoka, K. K., Corter, J. E., & Tatsuoka, C. (2004). Patterns of diagnosed mathematical content and process skills in TIMSS-R across a sample of 20 countries. American Educational Research Journal, 41 (4), 901-926.
66. Tatsuoka, K. K., Guerrero, A., Corter, J. E., Yamada, T., & Tatsuoka, C. (2003). International comparisons of mathematical thinking skills in the TIMSS-R. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago, Illinois.
67. Underwood, G., Jebbett, L., & Roberts, K. (2004). Inspecting pictures for information to verify a sentence: Eye movements in general encoding and in focused search. Quarterly Journal of Experimental Psychology A: Human Experimental Psychology, 57A, 165-182.
68. von Davier, M. (2005). A general diagnostic model applied to language testing data (ETS Research Report No. RR-05-16). Princeton, NJ: Educational Testing Service.
69. Whitely, S. E. (1980). Multicomponent latent trait models for ability tests. Psychometrika, 45, 479-494.
70. Williamson, D. M., Bauer, M., Steinberg, L. S., Mislevy, R. J., Behrens, J. T., & DeMark, S. F. (2004). Design rationale for a complex performance assessment. International Journal of Testing, 4(4), 303-332.
71. Wilson, M. (2005). Constructing measures: An item response modeling approach. Mahwah, NJ: Lawrence Erlbaum Associates.
72. Xu, X., & von Davier, M. (2006). Cognitive diagnosis for NAEP proficiency data (ETS Research Report No. RR-06-08). Princeton, NJ: Educational Testing Service.