Technology supports for assessment design

International Encyclopedia of Education

Volumn , Issue , 2010, Pages 56-65

  Mislevy, R J ^a   Winters, F I ^a   Bejar, I I ^b   Bennett, R E ^b   Haertel, G D ^c  

^a UNIVERSITY OF MARYLAND   (United States)

^b EDUCATIONAL TESTING SERVICE   (United States)

^c SRI INTERNATIONAL   (United States)

Author keywords

Assessment design; Automated scoring; Design pattern; Evidence centered design; Item model; Task design; Task model

Indexed keywords

EID: 79961084149     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1016/B978-0-08-044894-7.00703-X     Document Type: Chapter

References (60)

1. Alexander C., Ishikawa S., Silverstein M., et al. A Pattern Language: Towns, Buildings, Construction 1977, Oxford University Press, New York.
2. Almond R.G., Steinberg L.S., Mislevy R.J. Enhancing the design and delivery of assessment systems: A four-process architecture. Journal of Technology, Learning, and Assessment 2002, 1(5). (accessed August 2009). http://www.bc.edu/research/intasc/jtla/journal/v1n5.shtml.
3. Bejar I.I. Generative testing: From conception to implementation. Item Generation for Test Development 2002, 199-218. Erlbaum, Mahwah, NJ. S.H. Irvine, P.C. Kyllonen (Eds.).
4. Bejar I.I., Lawless R.R., Morley M.E., et al. A feasibility study of on-the-fly adaptive testing. Journal of Technology, Learning and Assessment 2003, 2(3):4-29.
5. Bennett R.E. Moving the field forward: Some thoughts on validity and automated scoring. Automated Scoring of Complex Tasks in Computer-Based Testing 2006, 403-412. Erlbaum, Hillsdale, NJ. D.M. Williamson, R.J. Mislevy, I.I. Bejar (Eds.).
6. Bennett R.E., Ben-Simon A. Toward theoretically meaningful automated essay scoring 2006, National Institute for Testing and Evaluation (NITE), Jerusalem, Israel, (accessed August 2009). http://clickit.ort.org.il/files/upl/224004906/934404432.pdf.
7. Bennett R.E., Jenkins F., Persky H., Weiss A. Assessing complex problem-solving performances. Assessment in Education 2003, 10:347-359.
8. Braun H., Bejar I.I., Williamson D.M. Rule based methods and mental modeling. Automated Scoring of Complex Tasks in Computer Based Testing 2006, 83-122. Erlbaum, Mahwah, NJ. D.M. Williamson, R.J. Mislevy, I.I. Bejar (Eds.).
9. Burstein J. The e-rater® scoring engine: Automated essay scoring with natural language processing. Automated Essay Scoring: A Cross-Disciplinary Perspective 2003, Erlbaum, Mahwah, NJ. M.D. Shermis, J.C. Burstein (Eds.).
10. Conati C., Gertner A., VanLehn K. Using Bayesian networks to manage uncertainly in student modeling. User Modeling and User-Adapted Interaction 2002, 12(4):371-417.
11. Conejo R., Guzmán E., Millán E., et al. SIETTE: A web-based tool for adaptive testing. International Journal of Artificial Intelligence in Education 2004, 14:29-61.
12. Davidson F., Lynch B.K. Testcraft: A Teacher's Guide to Writing and Using Language Test Specifications 2001, Yale University Press, New Haven, CT.
13. Deane P., Sheehan K.M., Sabatini J., Futagi Y., Kostin I. Differences in text structure and its implications for assessment of struggling readers. Scientific Studies of Reading 2006, 10(3):257-275.
14. Embretson S.E. A cognitive design system approach to generating valid tests: Application to abstract reasoning. Psychological Methods 1998, 3:380-396.
15. Embretson S.E. Generating items during testing: Psychometric issues and models. Psychometrika 1999, 64:407-433.
16. Enright M.K., Sheehan K. Modeling the difficulty of quantitative reasoning items: Implications for item generation. Item Generation for Test Development 2002, 129-158. Erlbaum, Mahwah, NJ. S.H. Irvine, P. Kyllonen (Eds.).
17. Gamma E., Helm R., Johnson R., Vlissides J. Design Patterns: Elements of Reunsable Object-Oriented Software 1995, Pearson, Upper Saddle River, NJ.
18. Graf E.A., Peterson S., Steffen M., Lawless R. Psychometric and Cognitive Analysis as a Basis for the Revision of Quantitative Item Models 2005, Educational Testing Service, Princeton, NJ, (No. RR-05-25).
19. Hamel L., Schank P. A wizard for PADI assessment design 2006, SRI International, Menlo Park, CA, PADI Technical Report 11.
20. Higgins D., Futagi Y., Deane P. Multilingual generalization of the Model Creator software for math item generation (RR-05-02) 2005, Educational Testing Service, Princeton, NJ, ETS Research Report.
21. Hively W., Patterson H.L., Page S.H. A "universe-defined" system of arithmetic achievement tests. Journal of Educational Measurement 1968, 5:275-290.
22. Hornke L.F. Item generation models for higher cognitive functions. Item Generation for Test Development 2002, 159-178. Earlbaum, Mahwah, NJ. S.H. Irvine, P.C. Kyllonen (Eds.).
23. Katz I.R. FRADS: A system for facilitating rapid prototyping by end users. Proceedings of the Sixth Annual International Conference on Human-Computer Interaction 1995, Elsevier, Amsterdam. Y. Anzai, K. Ogawa (Eds.).
24. Kurland L., Gertner A., Bartee T., Chisholm M., McQuade S. Using cognitive task analysis and eye tracking to understand imagery analysis 2005, Mitre Corporation, Bedford, MA, Mitre Corporation Technical Report #05-1365.
25. Landauer T.K., Laham D., Foltz P.W. The intelligent essay assessor. IEEE Intelligent Systems 2000, 15:27-31.
26. Leucht R.M. From design to delivery: Engineering the mass production of complex performance assessments 2002.
27. Messick S. The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher 1994, 23(2):13-23.
28. Mislevy R.J., Hamel L., Fried R.G., et al. Design patterns for assessing inquiry in science 2003, PADI Research Report #1., SRI International, Menlo Park, CA, (accessed August 2009). http://padi.sri.com/downloads/TR1_Design_Patterns.pdf.
29. Mislevy R.J., Riconscente M. Evidence-centered assessment design: Layers, concepts, and terminology. Handbook of Test Development 2006, 61-90. Erlbaum, Mahwah, NJ. S. Downing, T. Haladyna (Eds.).
30. Mislevy R., Steinberg L.S., Almond R.G. On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives 2003, 1(1):3-62.
31. Newstead S.E., Bradon P., Handley S.J., Dennis I., Evans J.S.B.T. Predicting the difficulty of complex logical reasoning problems. Thinking and Reasoning 2006, 12(1):62-90.
32. Odendahl N. Online delivery and scoring of constructed-response assessments 1999.
33. Page E.B. Project essay grade (PEG). Automated Essay Scoring: A Cross-Disciplinary Perspective 2003, 262-273. Erlbaum, Mahwah, NJ. M.D. Shermis, J.C. Burstein (Eds.).
34. Passoneau R., Hemat L., Plante J., Sheehan K.M. Electronic Sources as Input to GRE Reading Comprehension Item Development: Source Finder Prototype Evaluation 2002, (RR-02-12)., Educational Testing Service, Princeton, NJ, (accessed August 2009). http://www.ets.org.
35. Riconscente M.M., Mislevy R.J., Hamel L. An introduction to PADI task templates. Principled Assessment Designs for Inquiry (PADI) Technical Report 3 2005, PADI Research Group, SRI International, Menlo Park, CA.
36. Rupp A.A. Feature selection for choosing and assembling measurement models: A building-block-based organization. International Journal of Testing 2002, 2:311-360.
37. Sheehan K.M., Kostin I., Futagi Y., Hemat R., Zuckerman D. Inside Source Finder: Predicting the Acceptability Status of Candidate Reading Comprehension Source Documents 2006, (RR-06-24), Educational Testing Service, Princeton, NJ, (accessed August 2009). http://www.ets.org.
38. Shute V.J., Torreano L.A., Willis R.E. DNA: Toward an automated knowledge elicitation and organization tool. Computers as Cognitive Tools: No More Walls, II 2000, 309-338. Erlbaum, Mahwah, NJ. S. Lajoie (Ed.).
39. Singley M.K., Bennett R.E. Item generation and beyond: Applications of schema theory to mathematics assessment. Item Generation for Test Development 2002, 361-384. Erlbaum, Hillsdale, NJ. S. Irvine, P. Kyllonen (Eds.).
40. van der Linden W.J. Linear Models for Optimal Test Design 2005, Springer, New York.
41. Whalen S.J., Bejar I.I. Relational databases in assessment: An application to online scoring. Journal of Educational Computing Research 1998, 18:1-13.
42. Williams K.E. An automated aid for the conduct of a detailed cognitive task analysis for modeling human computer interaction performance. Special Series on Cognitive Task Analysis 2000, Erlbaum, Hillsdale, NJ. R. Schraagen, S.F. Chipman, V.L. Shalin (Eds.).
43. Williamson D.M., Mislevy R.J., Bejar I.I. Automated Scoring of Complex Tasks in Computer-Based Testing 2006, Erlbaum, Mahwah, NJ.
44. Zachary W.W., Ryder J.M., Hicinbothom J.H. Building cognitive task analyses and models of a decision-making team in a complex real-time environment. Cognitive Task Analysis 2000, 365-384. Erlbaum, Mahwah, NJ. S.F. Chipman, V.L. Shalin, J.M. Schraagen (Eds.).
45. Atlas of Science Literacy 2001, American Association for the Advancement of Science, AAAS, Washington DC.
46. Baker E.L. Design of automated authoring systems for tests 2002, 79-89. Published of Collection: National Research Council, Board on Testing and Assessment, Center for Education, Division of Behavioral and Social Sciences and Education, Washington, DC.
47. Bejar I.I. Model-based item generation: A review of recent research. Measuring Psychological Constructs: Advances in Model-Based Approaches 2009, American Psychological Association Books, Washington, DC. S.E. Embretson (Ed.).
48. Kane M.T. An argument-based approach to validity. Psychological Bulletin 1992, 112(3):527-535.
49. Messick S. Validity. Educational Measurement 1989, 13-103. American Council on Education/MacMillan, New York. 3rd edn. R.L. Linn (Ed.).
50. Mislevy R., Haertel G. Implications of Evidence-Centered Design for Educational Testing 2006, SRI International, Menlo Park, CA, Draft PADI Technical Report 17.
51. Mislevy R.J., Haertel G.D. Implications of evidence-centered design for educational assessment. Educational Measurement: Issues and Practice 2006, 26(4). 6-20.
52. Simon H.A. Complex systems: The interplay of organizations and markets in contemporary society. Computational and Mathematical Organization Theory 2001, 7(2):79-85.
53. VanLehn K., Niu Z. Bayesian student modeling, user interfaces and feedback: A sensitivity analysis. International Journal of Artificial Intelligence in Education 2001, 12(2):154-184.
54. Williams K.E., Hultman E.G., Graesser A. CAT: A tool for eliciting knowledge on how to perform procedures. Behavior Research Methods, Instruments and Computers 1998, 30(4):565-572.
55. Wilson M.R. Constructing Measures: An Item Response Modeling Approach 2005, Erlbaum, Mahwah, NJ.
56. - ANDES physics tutor. http://www.andestutor.org.
57. - Atlas of Science Literacy. http://www.project2061.org.
58. - Workshop Announcement: Getting Your Distance Learning Used Effectively. (iGEN Cognitive Agent Software). http://www.cognitiveagent.com.
59. - System of Intelligent Evaluation using Tests. http://www.marte.lcc.uma.es.
60. - QuestionMark Perception. http://www.questionmark.com.