You can’t fatten a hog by weighing it-or can you? Evaluating an assessment for learning system called ACED

International Journal of Artificial Intelligence in Education

Volumn 18, Issue 4, 2008, Pages 289-316

  Shute, Valerie J ^a   Hansen, Eric G ^b   Almond, Russell G ^b  

^a FLORIDA STATE UNIVERSITY   (United States)

^b EDUCATIONAL TESTING SERVICE   (United States)

Author keywords

Adaptive sequencing; Assessment for learning; Bayesian network; Formative assessment; Mathematics

Indexed keywords

EID: 85013619076     PISSN: 15604292     EISSN: 15604306     Source Type: Journal    
DOI: None     Document Type: Article

References (70)

1. Almond, R. G. (1995). Graphical Belief Modeling. Boca Raton, FL: CRC Press, Inc.
2. Almond, R. G. (2007a). An illustration of the use of Markov decision processes to represent student growth (learning). ETS Research Report, RR-07-40 (pp. 1-61), Princeton, NJ.
3. Almond, R. G. (2007b). I can name that Bayesian network in two matrixes! CEUR Workshop Proceedings, 268, Retrieved January 7, 2008, from http://sunsite.informatik.rwth-aachen.de/Publications/CEUR-WS/Vol-268/paper1.pdf
4. Almond, R. G., & Mislevy, R. J. (1999). Graphical models and computerized adaptive testing. Applied Psychological Measurement, 23 223-238.
5. Almond, R. G., DiBello, L. V., Moulder, B., & Zapata-Rivera, J-D. (2007). Modeling diagnostic assessment with Bayesian networks. Journal of Educational Measurement, 44 341-359.
6. Almond, R. G., Yan, D., Matukhin, A., & Chang, D. (2006). StatShop testing. ETS, Research Memorandum 06-04, Princeton, NJ.
7. Azevedo, R., & Bernard, R. M. (1995). A meta-analysis of the effects of feedback in computer-based instruction. Journal of Educational Computing Research, 13(2), 111-127.
8. Bangert-Drowns, R. L., Kulik, C. C., Kulik, J. A., & Morgan, M. T. (1991). The Instructional effect of feedback in test-like events. Review of Educational Research, 61(2), 213-238.
9. Barr, A., & Feigenbaum, E. (1981). he Handbook of Artificial Intelligence, Volume 1. Los Altos, CA: William Kaufmann, Inc.
10. Bass, K. M, & Glaser, R. (2004). Developing assessments to inform teaching and learning (CSE Rep. No. 628). Los Angeles, CA: UCLA, Graduate School of Education and Information Studies. Retrieved January 7, 2008, from http://eric.ed.gov/ERICDocs/data/ericdocs2sql/content_storage_01/0000019b/80/1b/9d/fb.pdf.
11. Birenbaum, M., Breuer, K., Cascallar, E., Dochy, F., Dori, Y., Ridgway, J., & Wiesemes, R. (2006 March/April.). A learning integrated assessment system. In R. Wiesemes & G. Nickmans (Eds.) EARLI Series of Position Papers.
12. Black, P., & Wiliam, D. (1998a). Assessment and classroom learning. Assessment in Education: Principles, Policy & Practice, 5(1), 7-71.
13. Black, P., & Wiliam, D. (1998b). Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan, 80(2), 139-148.
14. Brookhart, S. (2001). Successful students’ formative and summative uses of assessment information. Assessment in Education: Principles, Policy & Practice, 8(2), 153-169.
15. Clark, L. A., & Watson, D. (1995). Constructing validity: Basic issues in objective scale development. Psychological Assessment, 7 309-319.
16. Clarke, S. (2001). Unlocking formative assessment: Practical strategies for enhancing pupils’ learning in the primary classroom. London: Hodder & Stoughton.
17. Conati, C., Gertner, A. S., & VanLehn, K. (2002). Using Bayesian networks to manage uncertainty in student modeling. User Modeling and User-Adapted Interaction, 12(4), 371-417.
18. Conejo, R., Guzmán, E., Millán, E., Trella, M., Pérez-de-la-Cruz, J. L., & Ríos, A. (2004). SIETTE: A Web-Based Tool for Adaptive Teaching. International Journal of Artificial Intelligence in Education, 14 29-61.
19. Corbett, A. T., & Anderson, J. R. (2001). Locus of feedback control in computer-based tutoring: Impact on learning rate, achievement and attitudes. Proceedings of ACM CHI 2001 Conference on Human Factors in Computing Systems (pp. 245-252). New York: ACM Press.
20. Good, I. J. (1985).: Weight of evidence: A brief survey. In J. Bernardo, M. DeGroot, D. Lindley & A. Smith (Eds.) Bayesian Statistics 2 (pp. 249-269). Amsterdam: North Holland.
21. Good, I. J., & Card, W. (1971). The diagnostic process with special reference to errors. Method of Inferential Medicine, 10 176-188.
22. Hansen, E. G., & Mislevy, R. J. (2005). Accessibility of computer-based testing for individuals with disabilities and English language learners within a validity framework. In M. Hricko & S. Howell (Eds.) Online assessment and measurement: Foundation, challenges, and issues. Hershey, PA: Idea Group Publishing, Inc.
23. Hansen, E. G., & Shute, V. J. (2007, April). Towards accessible educational products: The usability of an assessment-for-learning system in mathematics for individuals with visual disabilities. Paper presented at the annual National Council on Measurement in Education meeting, Chicago, IL, April, 2007.
24. Hansen, E. G., Mislevy, R. J., Steinberg, L. S., Lee, M. J., & Forer, D. C. (2005). Accessibility of tests for individuals with disabilities within a validity framework. System: An International Journal of Educational Technology and Applied Linguistics, 33(1), 107-133.
25. Harlen, W. (2005). Teachers’ summative practices and assessment for learning-tensions and synergies. Curriculum Journal, 16(2), 207-223.
26. Hensen, R., & Douglas, J. (2005). Test construction for cognitive diagnosis. Applied Psychological Measurement, 29, 262-277.
27. Jameson, A. (2007). Adaptive interfaces and agents. In J. A. Jacko & A. Sears (Eds.) The human-computer interaction handbook: Fundamentals, evolving technologies & emerging applications (2nd ed.). Mahwah, NJ: Lawrence Erlbaum.
28. Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254-284.
29. Leahy, S., Lyon, C., Thompson, M., & Wiliam, D. (2005). Classroom assessment: Minute by minute, day by day. Educational Leadership, 63(3), 19-24.
30. Lepper, M. R., & Chabay, R. W. (1985). Intrinsic motivation and instruction: Conflicting views on the role of motivational processes in computer-based education. Educational Psychologist, 20(4), 217-230.
31. Madigan, D., & Almond, R. G. (1996). On test selection strategies for belief networks. In D. Fisher & H.-J. Lenz (Eds.) Learning from data: AI and statistics IV (pp. 89-98). New York: Springer-Verlag.
32. Madigan, D., Mosurski, K., & Almond, R. G. (1997). Graphical explanation in belief networks. Journal of Computational and Graphical Statistics, 6(2), 160-181.
33. Mason, B. J., & Brüning, R. (2001). Providing feedback in computer-based instruction: What the research tells us. Retrieved January 2, 2008, from http:IIdwb4.unl.eduIdwbIResearchIMBIMasonBruning.html.
34. McTighe, J., & O’Connor, K. (2005). Seven practices for effective learning. Educational Leadership, 63(3), 10-17.
35. Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Education Researcher, 23(2), 13-23.
36. Millán, E., & Pérez-de-la-Cruz, J. L. (2002). A Bayesian diagnostic algorithm for student modeling and its evaluation. User Modeling and User-Adaptive Interaction, 12 281-330.
37. Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2002). On the role of task model variables in assessment design. In S. H. Irvine & P. C. Kyllonen (Eds.) Item generation for test development (pp. 97-128). Mahwah, New Jersey: Lawrence Erlbaum.
38. Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1, 3-62.
39. Moreno, R. (2004). Decreasing cognitive load for novice students: Effects of explanatory versus corrective feedback in discovery-based multimedia. Instructional Science, 32 99-113.
40. Narciss, S., & Huth, K. (2004). How to design informative tutoring feedback for multi-media learning. In H. M. Niegemann, D. Leutner & R. Brunken (Eds.) Instructional design for multimedia learning (pp. 181-195). Munster, New York: Waxmann.
41. Nunnally, J. (1978). Psychometric theory. New York: McGraw-Hill.
42. Owen, R. J. (1975). A Bayesian sequential procedure for quantal response in the context of adaptive mental testing. Journal of the American Statistical Association, 70(350), 351-371.
43. Peat, M., & Franklin, S. (2002). Supporting student learning: the use of computer-based formative assessment modules. British Journal of Educational Technology, 33(5), 515-523.
44. Popham, W. J. (2001). Teaching to the test. Educational Leadership, 58(6), 16-20.
45. Razzaq, L., Feng, M., Nuzzo-Jones, G., Heffernan, N. T., Koedinger, K. R., Junker, B., et al. (2005). The Assistment Project: Blending assessment and assisting. In C. K. Looi, G. McCalla, B. Bredeweg & J. Breuker (Eds.) Artificial Intelligence in Education: Supporting Learning through Intelligent and Socially Informed Technology (pp. 555-562). Amsterdam: IOS Press.
46. Schwarz, R. D., & Sykes, R. (2004, June). Psychometric foundations for formative assessment. Paper presented at the National Educational Computing Conference, New Orleans, LA.
47. Shafer, G. (1976). A Mathematical Theory of Evidence. Princeton, NJ: Princeton University Press.
48. Shepard, L. A. (2003). Reconsidering large-scale assessment to heighten its relevance to learning. In J. M. Atkin & J. E. Coffey (Eds.) Everyday assessment in the science classroom. Arlington, VA: NSTA Press.
49. Shute, V. J. (2007). Tensions, trends, tools, and technologies: Time for an educational sea change. In C. A. Dwyer (Ed.) The future of assessment: Shaping teaching and learning (pp. 139-187). New York: Lawrence Erlbaum Associates, Taylor & Francis Group.
50. Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153-189.
51. Shute, V. J., & Zapata-Rivera, D. (in press). Educational measurement and intelligent systems. In E. Baker, B. McGaw & P. Peterson (Eds.) Third Edition of the International Encyclopedia of Education. Oxford, UK: Elsevier Publishers.
52. Shute, V. J., & Zapata-Rivera, D. (2008). Adaptive technologies. In J. M. Spector, D. Merrill, J. van Merrienboer & M. Driscoll (Eds.) Handbook of Research on Educational Communications and Technology (3rd Edition) (pp. 277-294). New York, NY: Lawrence Erlbaum Associates, Taylor & Francis Group.
53. Shute, V. J., Graf, E. A., & Hansen, E. (2005). Designing adaptive, diagnostic math assessments for individuals with and without visual disabilities. In L. PytlikZillig, R. Bruning & M. Bodvarsson (Eds.) Technology-based education: Bringing researchers and practitioners together (pp. 169-202). Greenwich, CT: Information Age Publishing.
54. Shute, V. J., Hansen, E. G., & Almond, R. G. (2007). An assessment for learning system called ACED: Designing for learning effectiveness and accessibility. ETS Research Report, RR-07-26 (pp. 1-54), Princeton, NJ.
55. Snow, R. E., & Mandinach, E. B. (1991). Integrating assessment and instruction: A research and development agenda. Princeton, NJ: Educational Testing Services.
56. Stiggins, R. J. (2002). Assessment crisis: The absence of assessment FOR learning. Phi Delta Kappan Professional Journal, 83(10), 758-765.
57. Stiggins, R. (2006). Assessment for learning: A key to motivation and achievement. Edge: The Latest Information for the Education Practitioner, 2(2), 1-19.
58. Symonds, K. W. (2004). After the test: Closing the achievement gaps with data. Naperville, IL: Learning Point Associates. Retrieved January 3, 2008, from http://www.ncrel.org/gap/studies/basrc.pdf
59. Taras, M. (2002). Using assessment for learning and learning from assessment. Assessment and Evaluation in Higher Education, 27(6), 501-510.
60. Tatsuoka, K. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20 345-354.
61. Tatsuoka, K. K. (1985). A probabilistic model for diagnostic misconceptions in the pattern classification approach. Journal of Educational Statistics, 10 55-73
62. Tiknaz, Y., & Sutton, A. (2006). Exploring the role of assessment tasks to promote formative assessment in key Stage 3 Geography: Evidence from twelve teachers. Assessment in Education: Principles, Policy and Practice, 13(3), 327-343.
63. Van der Linden, W. J. (1998). Bayesian item selection criteria for adaptive testing. Psychometrika, 63(2), 201-216.
64. VanLehn, K. (2006). The behavior of tutoring systems. International Journal of Artificial Intelligence in Education, 16(3), 227-265.
65. Vygotsky, L. (1967). Play and its role in the mental development of the child. Soviet Psychology, 5(3), 6-18.
66. Wainer, H. (1990). Computer adaptive testing: A primer. Hillsdale, NJ: Lawrence Erlbaum Associates.
67. Wainer, H., Dorans, N. J., Flaugher, R., Green, B. F., Mislevy, R. J., Steinberg, L., & Thissen, D. (2000). Computerized adaptive testing: A primer (2nd Edition). Mahwah, NJ: Erlbaum Associates.
68. Wiggins, G. (1998). Educative assessment: Designing assessments to inform and improve student performance. San Francisco: Jossey-Bass.
69. Wiliam, D., Lee, C., Harrison, C., & Black, P. (2004). Teachers developing assessment for learning: impact on student achievement. Assessment in Education: Principles, Policy & Practice, 11(1), 49-65.
70. Zapata-Rivera, D., Vanwinkle, W., Shute, V. J., Underwood, J. S., & Bauer, M. (2007). English ABLE. In R. Luckin, K. Koedinger & J. Greer (Eds.) Artificial Intelligence in Education-Building Technology Rich Learning Contexts That Work (Vol. 158, pp. 323-330). Amsterdam: IOS Press.