Evidence-centered design of epistemic games: Measurement principles for complex learning environments

Journal of Technology, Learning, and Assessment

Volumn 8, Issue 4, 2010, Pages 1-48

  Rupp, André A ^a   Gushta, Matthew ^a   Mislevy, Robert J ^a   Shaffer, David Williamson ^b  

^a UNIVERSITY OF MARYLAND   (United States)

^b UNIVERSITY OF WISCONSIN MADISON   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 77949942432     PISSN: None     EISSN: 15402525     Source Type: Journal    
DOI: None     Document Type: Article

References (71)

1. th International Objective Measurement Workshop, Berkeley, CA, April 5-7.
2. Allen, M.J., & Yen, W.M. (2002). Introduction to Measurement Theory. Long Grove, IL: Waveland Press.
3. Almond, R.G., Williamson, D.M., Mislevy, R.J., & Yan, D. (in press). Bayes nets in educational assessment. New York: Springer.
4. Almond, R.G., Mulder, J., Hemat, L.A., & Yan, D. (2006). Bayesian network models for local dependence among observable outcome variables (RR-06-36). Princeton, NJ: Educational Testing Service.
5. American Psychological Association, National Council on Measurement in Education, & American Educational Research Association (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
6. Bauer, D.J. (2007). Observations on the use of growth mixture models in psychological research. Multivariate Behavioral Research, 42, 757-786.
7. Bagley, E., & Shaffer, D.W. (2009). When people get in the way: Promoting civic thinking through epistemic gameplay. International Journal of Gaming and Computer-mediated Simulations, 1, 36-52.
8. Baker, E.L., Dickieson, J., Wulfeck, W., & O'Neil, H.F. (Eds.) (2007). Assessment of problem solving using simulations. Mahwah, NJ: Erlbaum.
9. Bennett, W. Jr., Lance, C.E., & Woehr, D.J. (Eds.) (2006). Performance measurement: Current perspectives and future challenges. Mahwah, NJ: Erlbaum.
10. Borsboom, D., Mellenbergh, G. J. (2007). Test validity in cognitive assessment. In J. P. Leighton & M.J. Gierl (Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 85-118). Cambridge: Cambridge University Press.
11. Brecht, J., Cheng, B., Mislevy, R., Haertel, G., & Haynie, K. (2009). The PADI System as a complex of epistemic forms and games (PADI Technical Report 21). Menlo Park, CA: SRI International.
12. Brennan, R.L. (2001a). An essay on the history and future of reliability from the perspective of replications. Journal of Educational Measurement, 38(4), 295-317.
13. Brennan, R.L. (2001b). Generalizability theory. New York: Springer.
14. Campbell, D.T., & Fiske, D.W. (1959) Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
15. Clark, D., Nelson, B., Sengupta, P., & D'Angelo, C. (2009, September). Rethinking science learning through digital games and simulations: Genres, examples, and evidence. Presented at the Learning science: Computer games, simulations, and education workshop sponsored by the National Academy of Sciences, Washington, D.C.
16. de Ayala, R.J. (2009). The theory and practice of item response theory. New York: The Guilford Press.
17. de Boeck, P. & Wilson, M. (eds.) (2004). Explanatory item response models: A generalized linear and non linear approach. NewYork: Springer.
18. Downing, S.M., & Haladyna, T.M. (Eds.). (2006). Handbook of test development. Mahwah, NJ: Erlbaum.
19. Duncan, T.E., Duncan, S.C., & Strycker, L.A. (2006). An introduction to latent variable growth curve modelling: Concepts, issues, and applications. Mahwah, NJ: Erlbaum.
20. Ehrlich, T. (2000). Civic responsibility and higher education. Phoenix, AZ: American Council on Education and The Oryx Press.
21. Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Mahwah, NJ: Erlbaum.
22. Fitzmaurice, G., Davidian, M., Verbeke, G., & Molenberghs, G. (2008). Longitudinal data analysis: A handbook of modern statistical methods. New York: Chapman & Hall/CRC.
23. Frisbie, D. (1988). Reliability of scores from teacher-made tests. Educational Measurement: Issues and Practice, 10, 55-65.
24. Gee, J.P. (2003). What video games have to teach us about learning and literacy. New York: Palgrave/Macmillan.
25. Gibson, D., Aldrich, C., & Prensky, M. (Eds.). (2006). Games and simulations in online learning: Research and development frameworks. Hershey, PA: Information Science Publishing.
26. Haberman, S.J. (2008). When can subscores have value? Journal of Educational and Behavioral Statistics, 33, 204-229.
27. Hale, J. (1999). Performance-based certification: How to design a valid, defensible, and cost-effective program. New York: Pfeiffer.
28. Joint Committee on Testing Practices (2004). Code of fair testing practices in education. Retrieved on November 11, 2008 from http://www.apa.org/science/fairtestcode.html.
29. Kamada, T. & Kawai S. (1989). An algorithm for drawing general undirected graphs. Information Processing Letters, 31, 7-15.
30. th ed.) (pp. 18-64). Westport, CT: Praeger.
31. th annual Maryland Assessment Conference entitled The Concept of Validity: Revisions, New Directions and Applications at the University of Maryland, College Park, MD, October 9-10.
32. Landauer, T.K., McNamara, D.S., Dennis, S., & Kintsch, W. (2007). Handbook of latent semantic analysis. Mahwah, NJ: Erlbaum.
33. Leighton, J.P. (2004). Avoiding misconception, misuse, and missed opportunities: The collection of verbal reports in educational achievement testing. Educational Measurement: Issues and Practice, 23(4), 6-15.
34. Leighton, J.P., & Gierl, M. (2007). Verbal reports as data for cognitive diagnostic assessment. In J. P. Leighton & M. J. Gierl (Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 146-172). Cambridge: Cambridge University Press.
35. th ed.) (pp. 13-103). New York: Macmillan.
36. Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741-749.
37. Mislevy, R.J. (2004). Can there be reliability without " reliability" ? Journal of Educational and Behavioral Statistics, 29, 241-244.
38. th edition) (pp. 257-305). Portsmouth, NH: Greenwood Publishing Group.
39. th annual Maryland Assessment Conference entitled The Concept of Validity: Revisions, New Directions and Applications at the University of Maryland, College Park, MD, October 9-10.
40. Mislevy, R.J., & Braun, H.I. (2003, May). Intuitive test theory. Presented at the Annual Dinner Meeting of the Princeton Association for Computing Machinery (ACM) and Institute of Electrical and Electronics Engineers (IEEE) Computer Society Chapters, Kingston, NJ, May 22.
41. Mislevy, R.J., Almond, R.G., & Steinberg, L.S. (2003). On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1, 3-62.
42. Mislevy, R.J., Steinberg, L.S., Almond, R.G., & Lukas, J.F. (2006). Concepts, terminology, and basic models of evidence-centered design. In D. M. Williamson, I. I. Bejar, & R. J. Mislevy (Eds.), Automated scoring of complex tasks in computer-based testing (pp. 15-48). Mahwah, NJ: Erlbaum.
43. Moss, P. (1994). Can there be validity without reliability? Educational Researcher, 23(2), 5-12.
44. National Assessment Governing Board (2006). Civics framework for the 2006 National Assessment of Educational Progress. Retrieved November 11, 2008 from http://www.nagb.org/civics_06.pdf.
45. st Century Skills Working Group sponsored by the MacArthur Foundation, Tempe, AZ, April 14-16.
46. st century approach for assessing reading competency (RR-09-04). Princeton, NJ: Educational Testing Service.
47. st century skills, education, and competitiveness: A resource and policy guide. Tuscon, AZ. Available online at www.21stcenturykills.org.
48. Phelan, M.J. (1990). Estimating the transition probabilities from censored Markov renewal processes. Statistics & Probability Letters, 10, 43-47.
49. Plass, J.L., Homer, B.D., & Hayward, E. (2009). Design factors for educationally effective animations and simulations. Journal of Computing in Higher Education, 21(1), 31-61.
50. st Century co-sponsored by ETS and IES, Philadelphia, PA, April 16-19.
51. Rupp, A.A., & Mislevy, R.J. (2007). Cognitive foundations of structured item response theory models. In J. Leighton & M. J. Gierl (Eds.), Cognitively diagnostic assessment for education: Theory and applications (pp. 205-241). Cambridge: Cambridge University Press.
52. Rupp, A.A., & Templin, J. (2008). Unique characteristics of diagnostic classification models: A comprehensive review of the current state-of-the-art. Measurement: Interdisciplinary Research and Perspectives, 6, 219-262.
53. Rupp, A.A., Templin, J., & Henson, R. (in press). Diagnostic measurement: Theory, methods, and applications. New York: Guilford Press.
54. Rupp, A.A., Choi, Y.-Y., Gushta, M., Mislevy, R.J., Thies, M.-C., Bagley, E., Hatfield, D., Nash, P., & Shaffer, D. (2009a, May). Statistical research for data structures from epistemic games: A brainstorm of ideas. Presented at the third meeting of the Assessment Group for Sociocultural and Games-based Learning sponsored by the John D. and Catherine T. MacArthur foundation, Phoenix, AZ, May 14-17.
55. Rupp, A.A., Choi, Y., Gushta, M., Mislevy, R. J., Bagley, E., Nash, P., Hatfield, D., Svarowski, G., & Shaffer, D. (2009b). Modeling learning progressions in epistemic games with epistemic network analysis: Principles for data analysis and generation. Proceedings from the Learning Progressions in Science Conference held in Iowa City, IA, June 24-26.
56. Seeratan, K., & Mislevy, R. (2009). Design patterns for assessing internal knowledge representations (PADI Technical Report 22). Menlo Park, CA: SRI International.
57. Shaffer, D.W. (2006a). How computer games help children learn. New York: Palgrave/Macmillan.
58. Shaffer, D.W. (2006b). Epistemic frames for epistemic games. Computers and Education, 46(3), 223-234.
59. st century assessment of learning. International Journal of Learning Media, 1(2), 33-53.
60. Shute, V.J. (2008). Focus on formative feedback. Review of Educational Research, 78, 153-189.
61. Shute, V.J., & Spector, J.M. (2008). SCORM 2.0 white paper: Stealth assessment in virtual worlds. Unpublished manuscript. Retrieved November 14, 2009 from http://www.adlnet.gov/Technologies/Evaluation/Library/Additional%20Resources/LETSI%20White%20 Papers/Shute%20-%20Stealth%20Assessment%20in%20Virtual%20 Worlds.pdf.
62. st century learning: The benefits of blinking. In J. Gee (Ed.), Games, learning, assessment. Boston, MA: MIT Press.
63. Shute, V.J., Ventura, M. Bauer, M., and Zapata-Rivera, D. (2008). Monitoring and fostering learning through games and embedded assessments (Research report RR-08-69). Princeton, NJ: Educational Testing Service.
64. Sinharay, S., Haberman, S.J., & Puhan, G. (2007). Subscores based on classical test theory: To report or not to report. Educational Measurement: Issues and Practice, 26(4), 21-28.
65. Storey, J.K., Kopp, K.J., Wiemer, K., Chipman, P., & Graesser, A.C. (in press). Using AutoTutor to teach scientific critical thinking skills. Behavior Research Methods.
66. Tatsuoka, K.K. (1983). Rule-space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345-354.
67. Templin, J., & Henson, R. (2009, April). Practical issues in using diagnostic estimates: Measuring the reliability and validity of diagnostic estimates. Presented at the annual meeting of the National Council on Measurement in Education (NCME), San Diego, CA.
68. Wainer, H., Bradlow, E.T., & Wang, X. (2007). Testlet response theory and its applications. Cambridge: Cambridge University Press.
69. Wasserman, S. & Faust, K. (1994). Social Network Analysis: Methods and Applications. Cambridge: Cambridge University Press.
70. Wild, C.L., & Rawasmany, R. (Eds.). (2007). Improving testing: Applying process tools and techniques to assure quality. Mahwah, NJ: Erlbaum.
71. Williamson, D.M., Mislevy, R.J., & Bejar, I. (2006). Automated scoring of complex tasks in computer-based testing. Mahwah, NJ: Erlbaum.