Design and Validation of Technology-Based Performance Assessments

Handbook of Research on Educational Communications and Technology, Third Edition

Volumn , Issue , 2008, Pages 595-604

  Baker, Eva L ^a   Chung, Gregory K W K ^a   Delacruz, Girlie C ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Assessment: The systematic observation of achievement using any of a number of formats, including paper, technology, or live judgment.; Performance: Ability of student or trainee to demonstrate accomplishment in a very specific domain such as a formal examination, across domains and subject matters, or in realistic contexts.; Validation (validity): The process and outcome of collecting and interpreting results of assessments or measurement so that inferences from findings are warranted by evidence.

Indexed keywords

EID: 81855200609     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.4324/9780203880869-49     Document Type: Chapter

References (88)

1. American Educational Research Association (AERA), American Psychological Association (APA), and National Council for Measurement in Education (NCME). (1999). Standards for Educational and Psychological Testing. Washington, D.C.: American Educational Research Association.*
2. Attali, Y. and Burstein, J. (2006). Automated essay scoring with e-rater ® V.2. J. Technol. Learn. Assess., 4(3) (available from http://www.jtla.org).
3. Baker, E. L. (1974). Beyond objectives: domain-referenced tests for evaluation and instructional improvement. Educ. Technol., 14(6), 10-16.
4. Baker, E. L. (1997). Model-based performance assessment. Theory Into Pract., 36(4), 247-254.*
5. Baker, E. L. (2002). Design of automated authoring systems for tests. In Technology and Assessment: Thinking Ahead-Proceedings from a Workshop, edited by Board on Testing and Assessment, National Research Council, pp. 79-89. Washington, D.C.: National Academy Press.
6. Baker, E. L. (2005). Technology and effective assessment systems. In Uses and Misuses of Data for Educational Accountability and Improvement, NSSE Yearbook, Vol. 104, Part 2, edited by J. L. Herman and E. H. Haertel, pp. 358-378. Chicago, IL: National Society for the Study of Education.
7. Baker, E. L. and Herman, J. L. (1983). Task structure design: beyond linkage. J. Educ. Meas., 20, 149-164.
8. Baker, E. L. and Mayer, R. E. (1999). Computer-based assessment of problem solving. Comput. Hum. Behav., 15, 269-282.*
9. Baker, E. L. and O’Neil, Jr., H. F. (1987). Assessing instructional outcomes. In Instructional Technology, edited by R. M. Gagné, pp. 343-377. Hillsdale, NJ: Lawrence Erlbaum Associates.
10. Baker, E. L. and O’Neil, Jr., H. F. (1996). Performance assessment and equity. In Implementing Performance Assessment: Promises, Problems, and Challenges, edited by M. B. Kane and R. Mitchell, pp. 183-199. Mahwah, NJ: Lawrence Erlbaum Associates.
11. Baker, E. L. and Schacter, J. (1996). Expert benchmarks for student academic performance: the case for gifted children. Gifted Child Q., 40(2), 61-65.
12. Baker, E. L., Freeman, M., and Clayton, S. (1991). Cognitive assessment of history for large-scale testing. In Testing and Cognition, edited by M. C. Wittrock and E. L. Baker, pp. 131-153. Englewood Cliffs, NJ: Prentice Hall.*
13. Baker, E. L., O’Neil, Jr., H. F., and Linn, R. L. (1993). Policy and validity prospects for performance-based assessment. Am. Psychol., 48, 1210-1218.*
14. Baker, E. L., Linn, R. L., Abedi, J., and Niemi, D. (1995). Dimensionality and generalizability of domain-independent performance assessments. J. Educ. Res., 89, 197-205.
15. Baker, E. L., Aschbacher, P. R., Niemi, D., and Sato, E. (2005). CRESST Performance Assessment Models: Assessing Content Area Explanation, CSE Tech. Rep. No. 652. Los Angeles, CA: University of California/National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
16. Bejar, I. I. (1991). A methodology for scoring open-ended architectural design problems. J. Appl. Psychol., 76, 522-532.
17. Bennett, R. E. (1999). Using new technology to improve assessment. Educ. Meas. Issues Pract., 18(3), 5-12.
18. Bennett, R. E. (2006). Moving the field forward: some thoughts on validity and automated scoring. In Automated Scoring of Complex Tasks in Computer-Based Testing, edited by D. M. Williamson, I. I. Behar, and R. J. Mislevy, pp. 403-412. Mahwah, NJ: Lawrence Erlbaum Associates.*
19. Bennett, R. E. and Bejar, I. I. (1998). Validity and automated scoring: it’s not only the scoring. Educ. Meas., 17(4), 9-17.
20. Bennett, R. E., Morley, M., and Quardt, D. (2000). Three response types for broadening the conception of mathematical problem solving in computerized tests. Appl. Psychol. Meas., 24, 294-309.
21. Bennett, R. E., Jenkins, F., Persky, H., and Weiss, A. (2003). Assessing complex problem solving performances. Assess. Educ. Princ. Policy Pract., 10, 347-359.
22. Birenbaum, M., Kelly, A. E., and Tatsuoka, K. K. (1993). Diagnosing knowledge states in algebra using the rule-space model. J. Educ. Meas., 20, 221-230.
23. Burstein, J. C. (2003). The e-rater scoring engine: automated essay scoring with natural language processing. In Automated Essay Scoring: A Cross-Disciplinary Perspective, edited by M. D. Shermis and J. Burstein, pp. 113-122. Mahwah, NJ: Lawrence Erlbaum Associates.
24. Burstein, J. C. and Chodorow, M. (1999). Automated essay scoring for nonnative English speakers. In Proceedings of Computer-Mediated Language Assessment and Evaluation of Natural Language Processing, joint symposium of the Association of Computational Linguistics and the International Association of Language Learning Technologies, June 22, College Park, MD.
25. Chi, M. T. H., Glaser, R., and Farr, M., Eds. (1988). The Nature of Expertise. Hillsdale, NJ: Lawrence Erlbaum Associates.*
26. Chung, G. K. W. K. and Baker, E. L. (2003a). An exploratory study to examine the feasibility of measuring problem-solving processes using a click-through interface. J. Technol. Learn. Assess., 2(2) (available from http://jtla.org).
27. Chung, G. K. W. K. and Baker, E. L. (2003b). Issues in the reliability and validity of automated scoring of constructed responses. In Automated Essay Grading: A Cross-Disciplinary Approach, edited by M. D. Shermis and J. E. Burstein, pp. 23-40. Mahwah, NJ: Lawrence Erlbaum Associates.*
28. Chung, G. K. W. K., O’Neil, Jr., H. F., and Herl, H. E. (1999). The use of computer-based collaborative knowledge mapping to measure team processes and team outcomes. Comput. Hum. Behav., 15, 463-494.
29. Chung, G. K. W. K., Harmon, T. C., and Baker, E. L. (2001). The impact of a simulation-based learning design project on student learning. IEEE Trans. Educ., 44, 390-398.
30. Chung, G. K. W. K., Baker, E. L., and Cheak, A. M. (2002). Knowledge Mapper Authoring System Prototype, CSE Tech. Rep. 575. Los Angeles, CA: University of California/National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
31. Chung, G. K. W. K., Baker, E. L., Brill, D. G., Sinha, R., Saadat, F., and Bewley, W. L. (2003a). Automated assessment of domain knowledge with online knowledge mapping. Proc. I/ITSEC, 25, 1168-1179.
32. Chung, G. K. W. K., Delacruz, G. C., Dionne, G. B., and Bewley, W. L. (2003b). Linking assessment and instruction using ontologies. Proc. I/ITSEC, 25, 1811-1822.*
33. Chung, G. K. W. K., Sinha, R., de Souza e Silva, A. A., Michiuye, J. K., Cheak, A. M., Saadat, F. et al. (2004). CRESST Human Performance Knowledge Mapping Tool Authoring System, Deliverable to Office of Naval Research. Los Angeles, CA: University of California/National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
34. Chung, G. K. W. K., Dionne, G. B., and Kaiser, W. J. (2006). An Exploratory Study Examining the Feasibility of Using Bayesian Networks to Predict Circuit Analysis Understanding. Paper presented at the Annual Meeting of the National Council on Measurement in Education, April 9-11, San Francisco, CA.
35. Chung, G. K. W. K., Baker, E. L., Delacruz, G. C., Bewley, W. L., Elmore, J., and Seely, B. (in press). An approach to authoring problem-solving assessments. In Assessment of Problem Solving Using Simulations, edited by E. L. Baker, J. Dickieson, W. Wulfeck, and H. F. O’Neil. Mahwah, NJ: Lawrence Erlbaum Associates.
36. Clauser, B. E. (2000). Recurrent issues and recent advances in scoring performance assessments. Appl. Psychol. Meas., 24, 310-324.
37. Clauser, B. E., Subhiyah, R. G., Nungester, R. J., Ripkey, D. R., Clyman, S. G., and McKinley, D. (1995). Scoring a performance-based assessment by modeling the judgments of experts. J. Educ. Meas., 32, 397-415.
38. Clauser, B. E., Margolis, M. J., Clyman, S. G., and Ross, L. P. (1997). Development of automated scoring algorithms for complex performance assessments: a comparison of two approaches. J. Educ. Meas., 34, 141-161.
39. Clauser, B. E., Swanson, D. B., and Clyman, S. G. (1999). A comparison of the generalizability of scores produced by expert raters and automated scoring systems. Appl. Meas. Educ., 12, 281-299.
40. Clauser, B. E., Harik, P., and Clyman, S. G. (2000). The generalizability of scores for a performance assessment scored with a computer-automated scoring system. J. Educ. Meas., 37, 245-262.
41. Glaser, R. (1963). Instructional technology and the measurement of learning outcomes: some questions. Am. Psychol., 18, 519-521.*
42. Gruber, T. R. (1995). Toward principles for the design of ontologies used for knowledge sharing. Int. J. Hum.-Comput. Stud., 43, 907-928.*
43. Herl, H. E., Niemi, D., and Baker, E. L. (1996). Construct validation of an approach to modeling cognitive structure of U.S. history knowledge. J. Educ. Res., 89, 206-218.
44. Herl, H. E., O’Neil, Jr., H. F., Chung, G. K. W. K., and Schacter, J. (1999). Reliability and validity of a computer-based knowledge mapping system to measure content understanding. Comput. Hum. Behav., 15, 315-334.
45. Hively, W. (1974). Introduction to domain-referenced testing. Educ. Technol., 14(6), 5-10.
46. Hively, W., Patterson, H. L., and Page, S. H. (1968). A ‘universe defined’ system of arithmetic achievement tests. J. Educ. Meas., 5, 275-290.*
47. Huff, K. L. and Sireci, S. G. (2001). Validity issues in computerbased testing. Educ. Meas. Issues Pract., 20(3), 16-25.*
48. Kane, M., Crooks, T., and Cohen, A. (1999). Validating measures of performance. Educ. Meas. Issues Pract., 18(2), 5-17.*
49. Katz, I. R. (1998). A Software Tool for Rapidly Prototyping New Forms of Computer-Based Assessments, GRE Research Report 91-06aP. Princeton, NJ: ETS.
50. Katz, I. R. and James, C. M. (1998). Toward Assessment of Design Skill in Engineering, GRE Research Report 97-16. Princeton, NJ: ETS.
51. Katz, I. R., Martinez, M. E., Sheehan, K. M., and Tatsuoka, K. K. (1998). Extending the rule space methodology to a semantically rich domain: diagnostic assessment in architecture. J. Educ. Behav. Stat., 24, 254-278.
52. Kim, J.-O., Chung, G. K. W. K., and Delacruz, G. C. (2004). Examining the sensitivity of knowledge maps using repeated measures: a growth modeling approach. In Proceedings of Current Issues in Knowledge Mapping in Assessment and Instruction, symposium conducted at the American Educational Research Association Annual Meeting, April 12-16, San Diego, CA.
53. Klein, D. C. D., Chung, G. K. W. K., Osmundson, E., and Herl, H. E. (2002). Examining the Validity of Knowledge Mapping as a Measure of Elementary Students’ Scientific Understanding, CSE Technical Report No. 557. Los Angeles, CA: University of California/National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
54. Koedinger, K. R. and Nathan, M. J. (2004). The real story behind story problems: effects of representations on quantitative reasoning. J. Learn. Sci., 13, 129-164.
55. Lambiotte, J. G., Dansereau, D. F., Cross, D. R., and Reynolds, S. B. (1989). Multi-relational semantic maps. Educ. Psychol. Rev., 1, 331-367.
56. Landauer, T. K., Laham, D., and Foltz, P. W. (2003). Automated scoring and annotation of essays with the Intelligent Essay Assessor. In Automated Essay Scoring: A Cross-Disciplinary Perspective, edited by M. D. Shermis and J. Burstein, pp. 87-112. Mahwah, NJ: Lawrence Erlbaum Associates.
57. Lee, J. J. (2000). The Impact of Korean Language Accommodations on Concept Mapping Tasks for Korean American English Language Learners. Ph.D. dissertation. Los Angeles, CA: University of California.
58. Linn, R. L., Baker, E. L., and Dunbar, S. B. (1991). Complex, performance-based assessment: Expectations and validation criteria. Educ. Res., 20(8), 15-21.*
59. Margolis, M. J. and Clauser, B. E. (2006). A regression-based procedure for automated scoring of a complex medical performance assessment. In Automated Scoring of Complex Tasks in Computer-Based testing, edited by D. M. Williamson, I. I. Behar, and R. J. Mislevy, pp. 123-167. Mahwah, NJ: Lawrence Erlbaum Associates.
60. Messick, S. (1995). Standards of validity and the validity of standards in performance assessment. Educ. Meas. Issues Pract., 14(4), 5-8.*
61. Mislevy, R. J. and Gitomer, D. H. (1995). The role of probability-based inference in an intelligent tutoring system. User Model. User-Adapt. Interact., 5, 253-282.*
62. Mislevy, R. J. and Riconscente, M. M. (2005). Evidence-Centered Assessment Design: Layers, Structures, and Terminology, PADI Technical Report No. 9. Menlo Park, CA: SRI International.
63. Mislevy, R. J. and Riconscente, M. M. (2006). Evidence-centered assessment design: layers, concepts, and terminology. In Handbook of Test Development, edited by S. Downing and T. Haladyna, pp. 61-90. Mahwah, NJ: Lawrence Erlbaum Associates.*
64. Mislevy, R. J., Steinberg, L. S., Breyer, F. J., Almond, R. G., and Johnson, L. (2002). Making sense of data from complex assessments. Appl. Meas. Educ., 15, 363-389.*
65. Mislevy, R., Hamel, L., Fried, R. G., Gaffney, T., Haertel, G., Hafter, A. et al. (2003). Design Patterns for Assessing Science Inquiry, PADI Technical Report No. 1. Menlo Park, CA: SRI International.
66. Niemi, D., Vendlinski, T. P., Baker, E. L., and Wang, J. (in press). On-line tools to improve formative assessment. Br. J. Educ. Technol.
67. Novak, J. D. and Gowin, D. B. (1984). Learning How to Learn. New York: Cambridge University Press.
68. Osmundson, E., Chung, G. K. W. K., Herl, H. E., and Klein, D. C. D. (1999). Concept Mapping in the Classroom: A Tool for Examining the Development of Students’ Conceptual Understandings, CSE Technical Report No. 507. Los Angeles, CA: University of California/National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
69. Page, E. B. and Petersen, N. S. (1995). The computer moves into essay grading: updating the ancient test. Phi Delta Kappan, 76, 561-565.
70. Pellegrino, J., Chudowsky, N., and Glaser, R., Eds. (2001). Knowing What Students Know: The Science and Design of Educational Assessment. Washington, D.C.: National Academy Press.*
71. Popham, W. J. and Husek, T. R. (1969). Implications for criterion-referenced measurement. J. Educ. Meas., 6, 1-9.
72. Powers, D. E., Burstein, J. C., Chodorow, M., Fowles, M. E., and Kukich, K. (2000). Comparing the Validity of Automated and Human Essay Scoring, RR-00-10. Princeton, NJ: Educational Testing Service.
73. Powers, D. E., Burstein, J. C., Chodorow, M., Fowles, M. E., and Kukich, K. (2001). Stumping e-rater: Challenging the Validity of Automated Essay Scoring, RR-01-03. Princeton, NJ: Educational Testing Service.
74. Riconscente, M., Mislevy, R., Hamel, L., and PADI Research Group. (2005). An Introduction to PADI Task Templates, PADI Technical Report No. 3. Menlo Park, CA: SRI International.
75. Ruiz-Primo, M. A., Schultz, S. E., Li, M., and Shavelson, R. J. (2001). Comparison of the reliability and validity of scores from two concept-mapping techniques. J. Res. Sci. Teaching, 38, 260-278.
76. Scalise, K. and Gifford, B. (2006). Computer-based assessment in e-learning: a framework for constructing ‘intermediate constraint’ questions and tasks for technology platforms. J. Technol. Learn. Assess., 4(6) (available from http://www.jtla.org).
77. Schacter, J., Herl, H. E., Chung, G. K. W. K., Dennis, R. A., and O’Neil, Jr., H. F. (1999). Computer-based performance assessments: a solution to the narrow measurement and reporting of problem-solving. Comput. Hum. Behav., 15, 403-418.*
78. Stevens, R. H. and Casillas, A. (2006).Artificial neural networks. In Automated Scoring of Complex Tasks in Computer-Based Testing, edited by D. M. Williamson, I. I. Behar, and R. J. Mislevy, pp. 259-312. Mahwah, NJ: Lawrence Erlbaum Associates.
79. Tatsuoka, K. K. and Tatsuoka, M. M. (1983). Spotting erroneous rules of operation by the individual consistency index. J. Educ. Meas., 20, 221-230.
80. Tatsuoka, K. K. and Tatsuoka, M. M. (1997). Computerized cognitive diagnostic adaptive testing: effect on remedial instruction as empirical validation. J. Educ. Meas., 34, 3-20.
81. van der Linden, W. J. (2005). Linear Models for Optimal Test Design. New York: Springer-Verlag.
82. Vendlinski, T. and Stevens, R. (2002). Assessing student problem-solving skills with complex computer-based tasks. J. Technol. Learn. Assess., 1(3) (available from http://www.jtla. org).
83. Vendlinski, T., Niemi, D., and Wang, J. (2005). Learning assessment by designing assessments: an on-line formative assessment design tool. In Proceedings of the Society for Information Technology and Teacher Education International Conference 2005, edited by C. Crawford, R. Carlsen, I. Gibson, K. McFerrin, J. Price, and R. Weber, pp. 228-240. Norfolk, VA: Association for the Advancement of Computing in Education.
84. Williamson, D. M., Bejar, I. I., and Hone, A. S. (1999). ‘Mental model’ comparison of automated and human scoring. J. Educ. Meas., 36, 158-184.
85. Williamson, D. M., Almond, R. G., and Mislevy, R. J. (2000). Model criticism of Bayesian networks with latent variables. In Uncertainty in Artificial Intelligence: Proceedings of the 16th Conference, edited by C. Boutilier and M. Goldzmidt, pp. 634-643. San Francisco, CA: Morgan Kaufmann.
86. Williamson, D. M., Almond, R. G., Mislevy, R. J., and Levy, R. (2006). An application of Bayesian networks in automated scoring of computerized simulation tasks. In Automated Scoring of Complex Tasks in Computer-Based Testing, edited by D. M. Williamson, I. I. Behar, and R. J. Mislevy, pp. 201-257. Mahwah, NJ: Lawrence Erlbaum Associates.
87. Yang, Y., Buckendahl, C. W., Juszkiewicz, P. J., and Bhola, D. S. (2002). A review of strategies for validating computer-automated scoring. Appl. Meas. Educ., 15, 391-412.
88. Yin, Y. and Shavelson, R. J. (2004). Application of Generalizability Theory to Concept-Map Assessment Research, CSE Technical Report No. 640. Los Angeles, CA: University of California/National Center for Research on Evaluation, Standards, and Student Testing (CRESST).