On the roles of external knowledge representations in assessment design

Journal of Technology, Learning, and Assessment

Volumn 8, Issue 2, 2010, Pages 1-58

  Mislevy, Robert J ^a   Rutstein, Daisy Wise ^h   Winters, Fielding I ^a   Behrens, John T ^b   Demark, Sarah F ^d   Frezzo, Dennis C ^e   Stanley, Ken ^d   Bennett, Randy E ^c   Levy, Roy ^f   Robinson, Daniel H ^g   Shute, Valerie J ⁱ  

^a UNIVERSITY OF MARYLAND   (United States)

^b UNIVERSITY OF NOTRE DAME   (United States)

^c EDUCATIONAL TESTING SERVICE   (United States)

^d CISCO SYSTEMS   (United States)

^e Academy Learning Systems Development Group   (United States)

^f ARIZONA STATE UNIVERSITY   (United States)

^g UNIVERSITY OF TEXAS   (United States)

^h UNIVERSITY OF MARYLAND   (United States)

ⁱ FLORIDA STATE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 77949981887     PISSN: None     EISSN: 15402525     Source Type: Journal    
DOI: None     Document Type: Article

References (88)

1. Ainsworth, S.E. (1999). A functional taxonomy of multiple representations. Computers and Education, 33, 131-152.
2. Almond, R.G., Steinberg, L.S., & Mislevy, R.J. (2002). Enhancing the design and delivery of assessment systems: A four-process architecture. Journal of Technology, Learning, and Assessment, 1(5). http://www.bc.edu/research/intasc/jtla/journal/v1n5.shtml.
3. American Association for the Advancement of Science. (2001). Atlas of science literacy. Washington, D.C.
4. Bachman, L.F. (2003). Building and supporting a case for test use. Language Assessment Quarterly, 2, 1-34.
5. Behrens, J.T., Mislevy, R.J., Bauer, M., Williamson, D.M., & Levy, R. (2004). Introduction to evidence centered design and lessons learned from its application in a global e-learning program. International Journal of Testing, 4, 295-301.
6. Bejar, I.I. (2002). Generative testing: From conception to implementation. In S. H. Irvine & P. C. Kyllonen (Eds.), Item generation for test development (pp. 199-217). Hillsdale, NJ: Lawrence Erlbaum Associates.
7. Bejar, I.I., & Braun, H.I. (1999). Architectural simulations: From research to implementation. Final Report to the National Council of Architectural Registration Boards. (ETS RM-99-02). Princeton, NJ: Educational Testing Service.
8. Bejar, I.I., Lawless, R.R., Morley, M.E., Wagner, M.E., Bennett, R.E., & Revuelta, J. (2003). A feasibility study of on-the-fly item generation in adaptive testing. Journal of Technology, Learning, and Assessment, 2(3). Available from http://www.jtla.org.
9. Belsley, D.A., Kuh, E., & Welch, R.E. (1980). Regression diagnostics: Identifying influential data and source of collinearity, John Wiley, New York.
10. Bennett, R.E., & Bejar, I.I. (1998). Validity and automated scoring: It's not only the scoring. Educational Measurement: Issues and Practice, 17(4), 9-17.
11. Bennett, R.E., Morley, M., & Quardt, D. (2000). Three response types for broadening the conception of mathematical problem solving in computerized tests. Applied Psychological Measurement, 24, 294-309.
12. Bennett, R.E., Morley, M., Quardt, D., Singley, M.K., Katz, I.R., & Nhouyvanisvong, A. (1999). Generating examples: A new response type for measuring quantitative reasoning. Journal of Educational Measurement, 36, 233-252.
13. Bentler, P.M. (2006). EQS 6 structural equation modeling software. Encino, CA: Multivariate Software, Inc.
14. Bloom, B.S. (Ed.) (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook I, cognitive domain. New York: Longman.
15. Bormuth, J.R. (1970). On the theory of achievement test items. Chicago: University of Chicago Press.
16. Breese, J.S., Goldman, R.P., & Wellman, M.P. (1994). Introduction to the special section on knowledge-based construction of probabilistic and decision models. IEEE Transactions on Systems, Man, and Cybernetics, 24, 1577-1579.
17. Butterfield, E.C., Nielsen, D., Tangen, K.L., & Richardson, M.B. (1985). Theoretically based psychometric measures of inductive reasoning. In S. E. Embretson (Ed.), Test design: Developments in psychology and psychometrics (pp. 77-148). New York: Academic Press.
18. Cameron, C.A., Beemsterboer, P.L., Johnson, L.A., Mislevy, R.J., Steinberg, L.S., & Breyer, F.J. (2000). A cognitive task analysis for dental hygiene. Journal of Dental Education, 64, 333-351.
19. Card, S.K., Moran, T.P., & Newell, A. (1983). The psychology of humancomputer interaction. Hillsdale, NJ: Lawrence Erlbaum Associates.
20. Clauser, B.E., Ross, L.P., Clyman, S.G., Rose, K.M., Margolis, M.J., Nungester, R.J., Piemme, T.E., Chang, L., El-Bayoumi, G., Malakoff, G.L., & Pincetl, P.S. (1997). Development of a scoring algorithm to replace expert rating for scoring a complex performance-based assessment. Applied Measurement in Education, 10, 345-358.
21. Collins, A., & Ferguson, W. (1993). Epistemic forms and epistemic games: Structures and strategies to guide inquiry. Educational Psychologist, 28, 25-42.
22. Conati, C., Gertner, A., & VanLehn, K. (2002). Using Bayesian networks to manage uncertainly in student modeling. User Modeling & User-Adapted Interaction, 12, 371-417.
23. Davidson, F. and Lynch, B.K. (2001). Testcraft: A teacher's guide to writing and using language test specifications. New Haven: Yale University Press.
24. Deane, P. (2006). Strategies for evidence identification through linguistic assessment of textual responses. In D. M. Williamson, R. J. Mislevy, and I. I. Bejar (Eds.), Automated Scoring of Complex Tasks in Computer Based Testing (pp. 313-371). Mahwah, NJ: Lawrence Erlbaum Associates.
25. DeMark, S.F., & Behrens, J.T. (2004). Using statistical natural language processing for understanding complex responses to free-response tasks. International Journal of Testing, 4, 371-390.
26. DeMark, S.F., West, P.A., & Behrens, J.T. (2005). Explorations in domain analysis and task model specification sensitive to underlying knowledge representations. Presented at the annual meeting of the American Education Research Association, April 15, 2005, San Francisco, CA.
27. Embretson, S. (1983). Construct validity: Construct representation versus nomothetic span. Psychological Bulletin, 93, 179-197.
28. Embretson, S.E. (1998). A cognitive design system approach to generating valid tests: Application to abstract reasoning. Psychological Methods, 3, 380-396.
29. Ericsson, K.A. (1996). The Road to Excellence: The Acquisition of Expert Performance in the Arts and Sciences, Sports, and Games. Mahwah, NJ: Lawrence Erlbaum Associates.
30. Frezzo, D.C. (2009). Using Activity Theory to Understand the Role of a Simulation-Based Learning Environment in a Computer Networking Course. Unpublished doctoral dissertation, University of Hawai'i, Manoa.
31. Frezzo, D.C., Behrens, J.T., & Mislevy, R.J. (2009). Design patterns for learning and assessment: Facilitating the introduction of a complex simulation-based learning environment into a community of instructors. The Journal of Science Education and Technology. http://www.springerlink.com/content/566p6g4307405346/fulltext.pdf.
32. Frezzo, D.C., & Stanley, K. (2005). Knowledge representations driving the design of computerized performance assessments in a complex simulated environment. Presented at the symposium Knowledge Representation in Assessment, at the Annual Meeting of the American Educational Research Association, April 15, 2005, Montreal, Canada.
33. Gibson, J.J. (1966). The senses considered as perceptual systems. Boston: Houghton Mifflin.
34. Gierl, M.J., Zhou, J., & Alves, C. (2008). Developing a taxonomy of item model types to promote assessment engineering. Journal of Technology, Learning, and Assessment, 7(2). Available from http://www.jtla.org.
35. Gitomer, D.H., & Steinberg, L.S. (1999). Representational issues in assessment design. In I. E. Sigel (Ed.), Development of mental representation (pp. 351-370). Hillsdale, NJ: Lawrence Erlbaum Associates.
36. Gotwals, A., & Songer, N. (2006). Cognitive Predictions: BioKIDS Implementation of the PADI Assessment System (PADI Technical Report 10). Menlo Park, CA: SRI International.
37. Haladyna, T.M. & Shindoll, R.R. (1989). Shells: A method for writing effective multiple-choice test items. Evaluation and the Health Professions, 12, 97-104.
38. Hamel, L., Mislevy, R.J., & Winters, F. (2008). Design rationale for an assessment task authoring system: a wizard for creating "mystery inquiry" assessment tasks (PADI Technical Report 19). Menlo Park, CA: SRI International.
39. Hamel, L., & Schank, P. (2005). Participatory, example-based data modeling in PADI (PADI Technical Report 4). Menlo Park, CA: SRI International.
40. Hamel, L., & Schank, P. (2006). A Wizard for PADI assessment design (PADI Technical Report 11). Menlo Park, CA: SRI International.
41. Hively, W., Patterson, H.L., & Page, S.H. (1968). A "universe-defined"system of arithmetic achievement tests. Journal of Educational Measurement, 5, 275-290.
42. Katz, I.R. (1994). Coping with the complexity of design: Avoiding conflicts and prioritizing constraints. In A. Ram, N. Nersessian, & M. Recker (Eds.), Proceedings of the Sixteenth Annual Meeting of the Cognitive Science Society (pp. 485-489). Mahwah, NJ: Lawrence Erlbaum Associates.
43. Katz, I.R. (1995). FRADS: A system for facilitating rapid prototyping by end users. In Y. Anzai & K. Ogawa (Eds.), Proceedings of the Sixth Annual International Conference on Human-Computer Interaction. Amsterdam: Elsevier Science Publishers.
44. Katz, I.R., Lipps, A.W., & Trafton, J.G. (2002). Factors affecting difficulty in the generating examples item type. ETS Research Report RR-02-07. Princeton, NJ: Educational Testing Service.
45. Kindfield, A.C.H. (1999). Generating and using diagrams to learn and reason about biological processes. Journal of the Structure and Learning and Intelligent Systems, 14, 81-124.
46. Lambiotte, J.G., Dansereau, D.F., Cross, D.R., & Reynolds, S.B. (1989). Multirelational semantic maps. Educational Psychology Review, 1, 331-367.
47. Larkin, J.H., & Simon, H.A. (1987). Why a diagram is (sometimes) worth ten thousand words. Cognitive Science, 11, 65-99.
48. Lehrer, R., & Schauble, L. (2002). Symbolic communication in mathematics and science: Co-constituting inscription and thought. In E. D. Amsel & J. Byrnes (Eds.), Language, literacy, and cognitive development: The development and consequences of symbolic communication. (pp. 167-192). Mahwah, NJ: Lawrence Erlbaum Associates.
49. Lewandowsky, S., & Behrens, J.T. (1999). Statistical graphs and maps. In F. T. Durso, R. S. Nickerson, R. W. Schvaneveldt, S. T. Dumais, D.S. Lindsay, & M. T. H. Chi (Eds.). Handbook of Applied Cognition (pp. 513-549). Chichester, UK: Wiley.
50. Luecht, R.M. (2002). From design to delivery: Engineering the mass production of complex performance assessments. Paper presented at the Annual Meeting of the National Council on Measurement in Education, New Orleans, LA.
51. Luecht, R.M. (April, 2007). Assessment engineering in language testing: From data models and templates to psychometrics. Invited paper presented at the annual meeting of the National Council on Measurement in Education, Chicago, IL.
52. Margolis, M.J., & Clauser, B.E. (2006). A regression-based procedure for automated scoring of a complex medical performance assessment. In D. M. Williamson, R. J. Mislevy, and I. I. Bejar (Eds.), Automated Scoring of Complex Tasks in Computer Based Testing (pp. 132-167). Mahwah, NJ: Lawrence Erlbaum Associates.
53. Markman, A.B. (1999). Knowledge representation. Mahwah, NJ: Lawrence Erlbaum Associates.
54. Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23(2), 13-23.
55. Mislevy, R.J. (2003). Substance and structure in assessment arguments. Law, Probability, and Risk, 2, 237-258.
56. th ed.) (pp. 257-305). Westport, CT: American Council on Education/Praeger Publishers.
57. Mislevy, R.J., & Haertel, G. (2006). Implications for evidence-centered design for educational assessment. Educational Measurement: Issues and Practice, 25, 6-20.
58. Mislevy, R.J., Hamel, L., Fried, R.G., Gaffney, T., Haertel, G., Hafter, A., Murphy, R., Quellmalz, E., Rosenquist, A., Schank, P., Draney, K., Kennedy, C., Long, K., Wilson, M., Chudowsky, N., Morrison, A., Pena, P., Songer, N., & Wenk, A. (2003). Design patterns for assessing science inquiry (PADI Technical Report 1). Menlo Park, CA: SRI International.
59. Mislevy, R.J., & Liu, M. (2009). Design patterns in the project "Leveraging evidence-centered design within scenario-based statewide science assessment."Presented at the Annual Meeting of the American Educational Research Association, San Diego, CA, April 13, 2009.
60. Mislevy, R.J., & Riconscente, M.M. (2006). Evidence-centered assessment design: Layers, structures, and terminology. In S. Downing & T. Haladyna (Eds.), Handbook of Test Development (pp. 61-90). Mahwah, NJ: Lawrence Erlbaum Associates.
61. Mislevy, R.J., Steinberg, L.S., & Almond, R.G. (2002). On the roles of task model variables in assessment design. In S. Irvine & P. Kyllonen (Eds.), Item generation for test development (pp. 97-128). Mahwah, NJ: Lawrence Erlbaum Associates.
62. Mislevy, R.J., Steinberg, L.S., & Almond, R.G. (2003). On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1, 3-67.
63. Mosenthal, P. & Kirsch, I. (1989). Understanding documents: Intersecting lists. Journal of Reading, 33, 210-213.
64. Palmer, S.E. (1978). Fundamental aspects of cognitive representation. In E. Rosch & B. B. Lloyd (Eds.), Cognition and categorization (pp. 259-303). Hillsdale, NJ: Lawrence Erlbaum Associates.
65. Pinker, S. (1990). A theory of graph comprehension. In R. Freedle (Ed.), Artificial intelligence and the future of testing (pp. 73-126). Hillsdale, NJ: Lawrence Erlbaum Associates.
66. Riconscente, M.M., Mislevy, R.J., & Hamel, L. (2005). An introduction to PADI task templates. (PADI Technical Report 3). Menlo Park, CA: SRI International.
67. Robinson, D.H. (1998). Graphic organizers as aids to text learning. Reading Research and Instruction, 37, 85-105.
68. Roid, G., & Finn, P. (1977). Algorithms for developing test questions from sentences in instructional materials. Interim Report, January-September. San Diego, CA: Navy Personnel Research and Development Center.
69. Rupp, A.A. (2002). Feature selection for choosing and assembling measurement models: a building-block-based organization. International Journal of Testing, 2, 311-360.
70. Scalise, K., & Gifford, B. (2006). Computer-based assessment in E-Learning: A framework for constructing "Intermediate Constraint"questions and tasks for technology platforms. Journal of Technology, Learning, and Assessment, 4(6) [online journal]. http://escholarship.bc.edu/jtla/vol4/6.
71. Schraagen, J.M., Chipman, S.F., & Shalin, V. J. (2000). Cognitive task analysis. Mahwah, NJ: Lawrence Erlbaum Associates.
72. Sfard, A. (1998). On two metaphors for learning and the dangers of choosing just one. Educational Researcher 27, 4-13.
73. Shute, V.J., Torreano, L.A., & Willis, R.E. (2000). DNA: Toward an automated knowledge elicitation and organization tool. In S. Lajoie (Ed.), Computers as cognitive tools: No more walls, II. Mahwah, NJ: Lawrence Erlbaum Associates.
74. Simon, H.A. (1978). Information-processing theory of human problem solving. In W. K. Estes (Ed.), Handbook of learning and cognitive processes (Vol. 5), Human information processing (pp. 271-295). Hillsdale, N.J.: Lawrence Erlbaum Associates.
75. Simon, H.A., & Kotovsky, K. (1963). Human acquisition of concepts for sequential patterns. Psychological Review, 70, 534-546.
76. Singley, M.K., & Bennett, R.E. (2002). Item generation and beyond: Applications of schema theory to mathematics assessment. In S. Irvine & P. Kyllonen (Eds.), Item generation for test development (pp.361-384). Hillsdale, NJ: Lawrence Erlbaum Associates.
77. Stewart, J., & Hafner, R. (1994). Research on problem solving: Genetics. In D. Gabel (Ed.), Handbook of research on science teaching and learning (pp 284-300). New York: Macmillan.
78. Thurstone, L.L., & Thurstone, T.G. (1941). Factorial studies of intelligence. Psychometric Monographs, No. 2.
79. Thurstone, L.L., & Thurstone, T.G. (1962). Primary mental abilities (Rev. ed.). Chicago: Science Research Associates.
80. Toulmin, S.E. (1958). The uses of argument. Cambridge: Cambridge University Press.
81. Tufte, E. (1990). Envisioning information. Cheshire, CT: Graphics Press.
82. Tukey, J.W. (1990). Data-based graphics: Visual display in the decades to come. Statistical Science, 5, 327-339.
83. Whitehead, A.N. (1911). An Introduction to mathematics. New York: Holt.
84. Wiggins, G.P. (1998). Educative assessment: Designing assessments to inform and improve student performance. San Francisco: Jossey-Bass.
85. Williamson, D.M., Bauer, M., Steinberg, L.S., Mislevy, R.J., & Behrens, J.T. (2004). Design rationale for a complex performance assessment. International Journal of Testing, 4, 303-332.
86. Williamson, D.M., Mislevy, R.J., & Bejar, I.I. (Eds.). (2006). Automated scoring of complex tasks in computer based testing. Mahwah, NJ: Lawrence Erlbaum Associates.
87. Wilson, M.R. (2005). Constructing measures: An item response modeling approach. Mahwah, NJ: Lawrence Erlbaum Associates.
88. Wise, D. (2005). Design patterns for assessing troubleshooting in computer networks. Presented at the annual meeting of the American Education Research Association, April 15, 2005, San Francisco, CA.