Defining and evaluating models of cognition used in educational measurement to make inferences about examinees' thinking processes

Educational Measurement: Issues and Practice

Volumn 26, Issue 2, 2007, Pages 3-16

  Leighton, Jacqueline P ^a,b   Gierl, Mark J ^a,b  

^a UNIVERSITY OF ALBERTA   (Canada)

^b UNIVERSITY OF ALBERTA   (Canada)

Author keywords

Cognitive models; Construct validity; Diagnostic testing; Educational measurement

Indexed keywords

EID: 34248160030     PISSN: 07311745     EISSN: 17453992     Source Type: Journal    
DOI: 10.1111/j.1745-3992.2007.00090.x     Document Type: Article

References (75)

1. Anderson, J. R., Qin, Y., Sohn, M. H., Stenger, V. A., & Carter, C. S. (2003). An information-processing model of the BOLD response in symbol manipulation tasks. Psychonomic Bulletin & Review, 10, 241 261.
2. Atkinson, R. C. (2004). Achievement versus aptitude in college admissions. In R. Zwick (Ed Rethinking the SAT: The future of standardized testing in university admissions. New York : Routledge.
3. Baddeley, A. D., & Logie, R. H. (1999). Working memory: The multiple-component model. In A. Miyake & P. Shah (Eds Models of working memory: Mechanisms of active maintenance and executive control (pp. 28 61). Cambridge, UK : Cambridge University Press.
4. Barnett, S. M., & Koslowski, B. (2002). Adaptive expertise: Effects of type of experience and the level of theoretical understanding it generates. Thinking and Reasoning, 8, 237 267.
5. Behrens, J. T., Mislevy, R. J., Bauer, M., Williamson, D. M., & Levy, R. (2004). Introduction to evidence centered design and lessons learned from its application in a global E-learning program. International Journal of Testing, 4, 295 301.
6. Bejar, I. (2002). Generative testing: From conception to implementation. In S. H. Irvine and P. C. Kyllonen (Eds Item generation for test development (pp. 199 217). Mahwah, NJ : Lawrence Erlbaum Associates.
7. Biggs, J., & Collis, K. (1982). Evaluating the quality of learning: The SOLO taxonomy. New York : Academic Press.
8. Bloom, B. S Ed 1956). Taxonomy of educational objectives, Handbook 1: Cognitive domain. Addison Boston : Wesley.
9. Briggs, D. C., Alonzo, A. C., Schwab, C., & Wilson, M. (2006). Diagnostic assessment with ordered multiple-choice item. Educational Assessment, 11, 33 63.
10. Brown, A. L., Campione, J. C., Webber, L. S., & McGilly, K. (1992). Interactive learning environments: A new look at assessment and instruction. In B. R. Gifford and M. C. O'Connor (Eds Changing assessments (pp. 121 212). Boston : Kluwer Academic.
11. Buck, G., Tatsuoka, K., & Kostin, I. (1997). The subskills of reading: Rule-space analysis of a multiple-choice test of second language reading comprehension. Language Learning, 47, 423 466.
12. Carpenter, P. A., Just, M. A., & Shell, P. (1990). What one intelligence test measures: A theoretical account of processing in the Raven's Progressive Matrices Test. Psychological Review, 97, 404 431.
13. Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 6, 271 315.
14. College Entrance Examination Board (2006). SAT Subject Tests™ Preparation Booklet.
15. Cronbach, L. J. (1971). Test validation. In R. L. Thorndike (Ed Educational measurement (2nd ed., pp. 443 507). Washington, DC : American Council on Education.
16. Embretson, S. E. (1998). A cognitive design system approach to generating valid tests: Application to abstract reasoning. Psychological Methods, 3, 300 396.
17. Embretson, S. E. (1999). Cognitive psychology applied to testing. In F. T. Durso (Ed Handbook of applied cognition (pp. 629 660). Chichester, England : John Wiley.
18. Embretson, S. E. (2002). Generating abstract reasoning items with cognitive theory. In S. H. Irvine and P. C. Kyllonen (Eds Item generation for test development (pp. 219 250). Mahwah, NJ : Lawrence Erlbaum Associates.
19. Embretson, S. E. (2005). Measuring human intelligence with artificial intelligence. In R. J. Sternberg and J. E. Pretz (Eds Cognition and Intelligence (pp. 251 267). Cambridge, UK : Cambridge University Press.
20. Embretson, S., & Gorin, J. (2001). Improving construct validity with cognitive psychology principles. Journal of Educational Measurement, 38, 343 368.
21. Embretson, S. E., & Wetzel, D. (1987). Component latent trait models for paragraph comprehension tests. Applied Psychological Measurement, 11, 175 193.
22. Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Cambridge, MA : MIT Press.
23. Garlick, D. (2002). Understanding the nature of the general factor of intelligence: The role of individual differences in neural plasticity as an explanatory mechanism. Psychological Review, 109, 116 136.
24. Gierl, M. J. (1997). Comparing the cognitive representations of test developers and students on a mathematics achievement test using Bloom's taxonomy. Journal of Educational Research, 91, 26 32.
25. Gierl, M. J., Tan, A., & Wang, C. (April, 2005). Identifying content and cognitive dimensions on the SAT (Research Report No. 2005-11). New York : The College Board.
26. Gorin, J. S. (2005) Manipulation of processing difficulty on reading comprehension test questions: The feasibility of verbal item generation. Journal of Educational Measurement, 42, 351 373.
27. Haladyna, T. M., & Downing, S. M. (2004). Construct-irrelevant variance in high-stakes testing. Educational Measurement: Issues and Practice, Spring, 17 27.
28. Hamilton, L. S., Nussbaum, E. M., & Snow, R. E. (1997). Interview procedures for validating science assessments. Applied Measurement in Education, 10, 181 200.
29. Healy, A. F. (Ed 2005). Experimental cognitive psychology and its applications. Washington, DC : American Psychological Association.
30. Idol, L. (2007). Models of curriculum-based assessment: A blueprint for learning (4th ed Austin, TX : Pro-Ed International Publisher.
31. Idol, L., Nevin, A., & Paolucci-Whitcomb, P. (1999). Models of curriculum-based assessment: A blueprint for learning (3rd ed Austin, TX : Pro-Ed International Publisher.
32. Irvine, S. H., & Kyllonen, P. C. (2002). Item generation for test development. Mahwah, NJ : Lawrence Erlbaum.
33. Kalchman, M., Moss, J., & Case, R. (2001). Psychological models for the development of mathematical understanding: Rational numbers and functions. In S. M. Carver and D. Klahr (Eds Cognition and instruction: Twenty-five years of progress (pp. 1 38). Mahwah, NJ : Lawrence Erlbaum Associates.
34. Kane, M. T. (1992) An argument-based approach to validity. Psychological Bulletin, 112, 527 535.
35. Kuhn, D. (2001). How do people know? Psychological Science, 12, 1 8.
36. Lane, S. (2004). 2004 NCME Presidential Address. Validity of high-stakes assessment: Are students engaged in complex thinking? Educational Measurement: Issues and Practice, 23, 6 14.
37. Leighton, J. P. (2004). Avoiding misconceptions, misuse, and missed opportunities: The collection of verbal reports in educational achievement testing. Educational Measurement: Issues and Practice, Winter, 1 10.
38. Leighton, J. P., & Gierl, M. J. (Eds in press-a). Cognitive diagnostic assessment for education: Theory and applications. Cambridge, UK : Cambridge University Press.
39. Leighton, J. P., & Gierl, M. J. (in press-b). Verbal reports as data for cognitive diagnostic assessment. In J. P. Leighton and M. J. Gierl (Eds Cognitive diagnostic assessment for education: Theory and applications. Cambridge, UK : Cambridge University Press.
40. Leighton, J. P., Gierl, M. J., & Hunka, S. (2004). The attribute hierarchy model: An approach for integrating cognitive theory with assessment practice. Journal of Educational Measurement, 41, 205 236.
41. Leighton, J. P., & Gokiert, R. (2005). The cognitive effects of test item features: Identifying construct irrelevant variance and informing item generation. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal, Canada.
42. Lohman, D. F. (2000). Complex information processing and intelligence. In R. J. Sternberg (Ed Handbook of intelligence (pp. 285 340). Cambridge, UK : Cambridge University Press.
43. Louis, K. S., Febey, K., & Schroeder, R. (2005). State-mandated accountability in high schools: Teachers' interpretations of a new era. Educational Evaluation and Policy Analysis, 27, 177 204.
44. Luecht, R. M. (2005). Extracting multidimensional information from multiple-choice question distractors for diagnostic scoring. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal, Canada.
45. Messick, S. (1989). Validity. In R. L. Linn (Ed Educational measurement (3rd ed., pp. 13 103). New York : American Council on Education/Macmillan.
46. Mislevy, R. J. (1996). Test theory reconceived. Journal of Educational Measurement, 33, 379 416.
47. Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1, 3 67.
48. National Research Council (2001). Knowing what students know: The science and design of educational assessment. Washington, DC : National Academy Press.
49. Newell, A., & Simon, H. A. (1972). Human problem solving. Englewood Cliffs, NJ : Prentice Hall.
50. Nichols, P. D., Chipman, S. F., & Brennan, R. L. (Eds 1995). Cognitively diagnostic assessment. Hillsdale, NJ : Erlbaum.
51. Norris, S. P., Leighton, J. P., & Phillips, L. M. (2004). What is at stake in knowing the content and capabilities of children's minds? A case for basing high stakes tests on cognitive models. Theory and Research in Education, 2, 283 308.
52. Organization for Economic Cooperation and Development (OECD) (2004). Learning for tomorrow's world: First results from PISA 2003. Paris: OECD.
53. Pearson, P. D., & Garavaglia, D. R. (2003). NAEP Validity studies: Improving the information value of performance items in large scale assessments, NCES 2003-08. Washington, DC : U.S. Department of Education, National Center for Education Statistics.
54. Pellegrino, J. W., Baxter, G. P., & Glaser, R. (1999). Addressing the "Two Disciplines" problem: Linking theories of cognition and learning with assessment and instructional practice. Review of Research in Education, 24, 307 353.
55. Poggio, A., Clayton, D. B., Glasnapp, D., Poggio, J., Haack, P., & Thomas, J. (2005). Revisiting the item format question: Can the multiple choice format meet the demand for monitoring higher-order skills? Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal, Canada.
56. Raven, J. C., Court, J. H., & Raven, J. (1992). Manual for Raven's progressive matrices and vocabulary scale. San Antonio, TX : Psychological Corporation.
57. Resnick, L. B., & Resnick, D. P. (1992). Assessing the thinking curriculum: New tools for educational reform. In B. R. Gifford and M. C. O'Connor (Eds Changing assessments (pp. 37 76). Boston, MA : Kluwer Academic.
58. Rogers, W. T., & Harley, D. (1999). An empirical comparison of three- and four-choice items and tests: Susceptibility to testwiseness and internal consistency reliability. Educational and Psychological Measurement, 59, 234 247.
59. Rogers, W. T., & Yang, P. (1996). Test-wiseness: Its nature and application. European Journal of Psychological Assessment, 12, 247 259.
60. Shinn, M. R Ed 1998). Advanced applications of curriculum-based measurement. New York : Guilford Press.
61. Siegler, R. S. (2005). Models of categorization: What are the limits? In L. Gershkoff-Stowe and D. Rakison (Eds Building object categories in developmental time (pp. 433 439). Mahwah, NJ : Lawrence Erlbaum Associates.
62. Snow, R. (1993). Construct validity and constructed-response tests. In R. Bennett and W. Ward (Eds Construction versus choice in cognitive measurement (pp. 45 60). Hillsdale, NJ : Lawrence Erlbaum Associates.
63. Snow, R. E., & Lohman, D. F. (1989). Implications of cognitive psychology for educational measurement. In R. L. Linn (Ed Educational measurement (3rd ed., pp. 263 331). New York : American Council on Education Macmillan.
64. Stanovich, K. E., Sá, W. C., & West, R. F. (2004). Individual differences in reasoning. In J. P. Leighton and R. J. Sternberg (Eds Nature of reasoning (pp. 375 409). New York : Cambridge University Press.
65. Sternberg, R. J. (1984). What cognitive psychology can and cannot do for test development. In B. S. Plake and J. Mitchell (Eds Social and technical issues in testing: Implications for test construction and usage (pp. 39 60). Hillsdale, NJ : Lawrence Erlbaum.
66. Sternberg, R. J. (1999). Intelligence as developing expertise. Contemporary Educational Psychology, 24, 359 375.
67. Tatsuoka, K., Corter, J. E., & Tatsuoka, C. (2004). Patterns of diagnosed mathematical content and process skills in TIMSS-R across a sample of 20 countries. American Educational Research Journal, 41, 901 926.
68. Taylor, K. L., & Dionne, J.-P. (2000). Accessing problem-solving strategy knowledge: The complementary use of concurrent verbal protocols and retrospective debriefing. Journal of Educational Psychology, 92, 413 425.
69. Thissen, D., & Edwards, M. C. (2005). Diagnostic scores augmented using multidimensional item response theory. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal, Canada.
70. United States Department of Education (2004). Testing for results: Helping families, schools, and communities understand and improve student achievement. In NCLB (Stronger accountability) (chap. 1). Retrieved February 15, 2006, from http://www.ed.gov/nclb/accountability/ayp/testingforresults.html.
71. Wang, X. B., Deng, H., Williams, K., & Laitusis, V. (2005). Assessing the interpretability of four diagnostic reporting methods for the new SAT. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal, Canada.
72. Webb, N. L. (2006). Identifying content for student achievement tests. In S. M. Downing and T. M. Haladyna (Eds Handbook of test development (pp. 155 180). Mahwah, NJ : Erlbaum.
73. Wellman, H. M., & Lagattuta, K. H. (2004). Theory of mind for learning and teaching: The nature and role of explanation. Cognitive Development, 19, 479 497.
74. Wilson, M. (2005). Constructing measures: An item response modeling approach. Mahwah, NJ : Erlbaum.
75. Wilson, M., & Sloane, K. (2000). From principles to practice: An embedded assessment system. Applied Measurement in Education, 13, 181 208.