Science assessments for all: Integrating science simulations into balanced state science assessment systems

Journal of Research in Science Teaching

Volumn 49, Issue 3, 2012, Pages 363-393

  Quellmalz, Edys S ^a   Timms, Michael J ^b   Silberglitt, Matt D ^c   Buckley, Barbara C ^a  

^a WestEd   (United States)

^b Australian Council for Educational Research   (Australia)

^c WestEd   (United States)

Author keywords

assessment; balanced assessment systems; evidence centered design; model based learning; simulations

Indexed keywords

ASSESSMENT; ASSESSMENT SYSTEM; BALANCED ASSESSMENT; CONTENT KNOWLEDGE; ENGLISH LANGUAGES; GENERAL POPULATION; ITEM RESPONSE THEORY; MIDDLE SCHOOL; MODEL-BASED LEARNING; MULTI-LEVEL; PSYCHOMETRIC QUALITY; RELIABILITY AND VALIDITY; SIMULATION-BASED; SIMULATIONS; STUDENT CHARACTERISTICS; TECHNICAL QUALITY; TESTING PROGRAMS;

COMPUTER SIMULATION; CURRICULA; DESIGN; RATING; SCHOOL BUILDINGS; STUDENTS; SURVEYS; TESTING;

HANDICAPPED PERSONS;

EID: 84856724583     PISSN: 00224308     EISSN: 10982736     Source Type: Journal    
DOI: 10.1002/tea.21005     Document Type: Article

References (100)

1. American Association for the Advancement of Science (AAAS). (1993). Benchmarks for science literacy. New York, NY: Oxford University Press.
2. American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (1999). Standards for psychological testing. Washington, DC: American Educational Research Association.
3. Bangert-Downs, R. L., Kulik, C. L. C., Kulik, J. A., &, Morgan, M. T., (1991). The instructional effect of feedback in test-like events. Review of Educational Research, 61 (2), 213-238.
4. Bell, P., &, Linn, M. C., (2000). Beliefs about science: How does science instruction contribute? In, B. K. Hofer, &, P. R. Pintrich, (Eds.), Personal epistemology: The psychology of beliefs about knowledge and knowing. Mahwah, NJ: Lawrence Erlbaum Associates.
5. Betrancourt, M., (2005). The animation and interactivity principles. In, R. E. Mayer, (Ed.), Handbook on multimedia learning. New York: Cambridge University Press.
6. Black, P., (1998). Testing: Friend or foe? Theory and practice of assessment and testing. New York: Falmer Press.
7. Black, P., &, Wiliam, D., (1998). Assessment and classroom learning. Assessment in Education, 5 (1), 7-74.
8. Blumenfeld, P. C., Marx, R. W., Krajcik, J. S., &, Soloway, E., (1996). Learning with peers: From small group cooperation to collaborative communities. Educational Researcher, 25 (8), 37-40.
9. Bolt, S. E., &, Thurlow, M. L., (2004). Five of the most frequently allowed testing accommodations in state policy. Remedial and Special Education, 25 (3), 141-152. (Pubitemid 38750490)
10. Bransford, J. D., Brown, A. L., &, Cocking, R. R., (2000). How people learn: Brain, mind, experience, and school. Washington, DC: National Academy Press.
11. Buckley, B. C., (2000). Interactive multimedia and model-based learning in biology. International Journal of Science Education, 22 (9), 895-935.
12. Buckley, B. C., (in press) Supporting and assessing complex biology learning with computer-based simulations and representations. In, D. Treagust, &, C-.Y. Tsui, (Eds.), Multiple representations in biological education. New York: Springer.
13. Buckley, B. C., Gobert, J., Horwitz, P., &, O'Dwyer, L., (2010). Looking inside the black box: Assessing model-based learning and inquiry in biologica. International Journal of Learning Technologies, 5 (2): 166-190.
14. Buckley, B. C., Gobert, J., Kindfield, A. C. H., Horwitz, P., Tinker, B., Gerlits, B., â∥, Willett, J., (2004). Model-based teaching and learning with hypermodels: What do they learn? How do they learn? How do we know ? Journal of Science, Education and Technology, 13 (1), 23-41.
15. Burling, K., Beck, R., Jude, J., Murray, E., Harms, M., &, Dolan, B., (2006). Constructing innovative computer-administered tasks and items according to universal design: Illustrative examples with pilot data. Paper presented at the Annual Meeting of the National Council on Measurement in Education.
16. Case, B. J., (2008). Accommodations to improve instruction, learning, and assessment. In, M. Karchmer, R. Johns, &, M. Ross, (Eds.), Testing deaf students in an age of accountability. Washington, DC: Gallaudet Research Institute.
17. Case, B. J., Brooks, T., Wang, S., &, Young, M., (2005). Administration mode comparability study. San Antonio, TX: Harcourt Assessment, Inc.
18. CAST. (2008). Universal design for learning guidelines version 1.0. Wakefield, MA: Author.
19. Chiu, C. W. T., &, Pearson, P. D., (1999). Synthesizing the effects of test accommodations for special education and limited English proficiency students. Paper presented at the National Conference on Large Scale Assessment, Snowbird, UT.
20. Clement, J., (1989). Learning via model construction and criticism: Protocol evidence on sources of creativity in science. In, J. A. Glover, R. R. Ronning, &, C. R. Reynolds, (Eds.), Handbook of creativity: Assessment, theory and research (pp. 341-381). New York: Plenum Press.
21. College Board. (2009). Science: College Boards standards for college success. Retrieved from.
22. Collins, A., Brown, J. S., &, Newman, S., (1989). Cognitive apprenticeship: Teaching the craft of reading, writing, and mathematics. In:, L. B. Resnick, (Ed.), Knowing, learning, and instruction: Essays in honor of Rober Glaser (pp. 453-494). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
23. Covington, M. V., (1999). Caring about learning: The nature and nurturing of subject-matter appreciation. Educational Psychologist, 34 (2), 127-136.
24. Darling-Hammond, L., (2010). Performance counts: Assessment systems that support high-quality learning. Washington, DC: CCSSO.
25. Darling-Hammond, L., &, Pecheone, R., (2010). Developing an internationally comparable balanced assessment system that supports high-quality learning. Presented at the National Conference on Next Generation K-12 Assessment Systems, Center for K-12 Assessment & Performance Management with the Education Commission of the States (ECS) and the Council of Great City Schools (CGCS), Washington, DC
26. Dassa, C., Vazquez-Abad, J., &, Ajar, D., (1993). Formative assessment in a classroom setting: From practice to computer innovations. Alberta Journal of Educational Research, 39 (1), 111-125.
27. Donovan, M. S., &, Bransford, J. D., (2005). How students learn: Science in the classroom. Washington, DC: The National Academies Press.
28. Duschl, R. A., &, Osborne, J., (2002). Supporting and promoting argumentation discourse in science education. Studies in Science Education, 38, 39-72.
29. Duschl, R. A., Schweingruber, H. A., &, Shouse, A. W., (2007). Taking science to school: Learning and teaching science in grades K-8. Washington, DC: The National Academies Press.
30. Edelson, D., &, Reiser, B. J., (2006). Making authentic practices accessible to learners: Design challenges and strategies. In, K. Sawyer, (Ed.), Cambridge handbook of the learning sciences (pp. 335-354). New York: Cambridge University Press.
31. Forsyth, B., &, Lessler, J. T., (2004). Cognitive laboratory methods: A taxonomy. In, P. P. Biemer, R. M. Groves, L. E. Lyberg, N. A. Mathiowetz, &, S. Sudman, (Eds.), Measurement errors in surveys. Hoboken, NJ, USA: Wiley.
32. Geier, R., Blumenfeld, P., Marx, R., Krajcik, J., Fishman, B., &, Soloway, E., (2008). Standardized test outcomes of urban students participating in standards and project based science curricula. Journal of Research in Science Teaching, 45 (8), 922-939.
33. George, D., &, Mallery, P., (2003). SPSS for Windows step by step: A simple guide and reference. 11.0 update (4th ed.) Boston: Allyn & Bacon.
34. Gerard, L. F., Spitulnik, M., &, Linn, M. C., (2010). Teacher use of evidence to customize inquiry science instruction. Journal of Research in Science Teaching, 47 (9), 1037-1063.
35. Gobert, J., (2000). A typology of models for plate tectonics: Inferential power and barriers to understanding. International Journal of Science Education, 22 (9), 937-977.
36. Gobert, J. D., &, Clement, J. J., (1999). Effects of student-generated diagrams versus student-generated summaries on conceptual understanding of causal and dynamic knowledge in plate tectonics. Journal of Research in Science Teaching, 36 (1), 39-53. (Pubitemid 129534707)
37. Goldstone, R. L., (2006). The complex systems see-change in education. The Journal of Learning Sciences, 15, 35-43. (Pubitemid 43242262)
38. Goldstone, R. L., &, Wilensky, U., (2008). Promoting transfer through complex systems principles. Journal of the Learning Sciences, 17, 465-516.
39. Harns, M., Burling, K., Hanna, E., &, Dolan, B., (2006). Constructing innovative computer-administered tasks and items according to universal design: Establishing guidelines for test developers. Paper presented at the Annual Meeting of the National Council on Measurement in Education.
40. Heiser, J., (2004). External representations as insights to cognition: Production and comprehension of text and diagrams in instructions. Unpublished doctoral dissertation. Stanford University.
41. Herman, J., Dai, Y., Htut, A. M., Martinez, M., &, Rivera, N., (2010). CRESST evaluation report: Evaluation of the Enhanced Assessment Grants (EAGs). Los Angeles: CRESST.
42. Herman, J. L., Osmundson, E., Ayalya, C., Schneider, S., &, Timms, M., (2005). The nature and impact of teachers formative assessment practices. Paper presented at the American Educational Research Association.
43. Hestenes, D., Wells, M., &, Swackhamer, G., (1992). Force concept inventory. The Physics Teacher, 30, 141-158.
44. Horwitz, P., Gobert, J. D., Buckley, B. C., &, O'Dwyer, L. M., (2010). Learning genetics with dragons: From computer-based manipulatives to hypermodels. In, M. J. Jacobson, &, P. Reimann, (Eds.), Designs for learning environments of the future: International perspectives from the learning sciences. New York: Springer.
45. Jacobson, M. J., (2001). Problem solving, cognition, and complex systems: Differences between experts and novices. Complexity, 6 (3), 41-49. (Pubitemid 33271961)
46. King, K., (2011). Balanced, multilevel science assessment systems. Presented at the National Conference on Student Assessment. Orlando, FL.
47. Kolodner, J. L., Camp, P. J., Crismond, D., Fasse, B., Gray, J., Holbrook, J., Puntambekar, S., &, Ryan, M., (2003). Problem-based learning meets case-based reasoning in the middle-school science classroom: Putting learning by design into practice. The Journal of the Learning Sciences, 12 (4), 495-547. (Pubitemid 37244610)
48. Kopriva, R., (2008). Improving testing for English language learners. New York: Routledge-Taylor & Francis Group.
49. Krajcik, J., Blumenfeld, P. C., Marx, R. W., Bass, K. M., &, Fredricks, J., (1998). Inquiry in project-based science classrooms: Initial attempts by middle school students. Journal of the Learning Sciences 7 (3&4), 313-350. (Pubitemid 128497025)
50. Lee, O., (2002). Science inquiry for elementary students from diverse backgrounds. In, W. G. Secada, (Ed.), Review of research in education (Vol. 26, pp. 23-69). Washington, DC: American Educational Research Association.
51. Lehrer, R., &, Schauble, L., (2002). Symbolic communication in mathematics and science: Co-constituting inscription and thought. In, E. D. A. J. Byrnes, (Ed.), Language, literacy, and cognitive development. The development and consequences of symbolic communication (pp. 167-192). Mahwah, NJ: Lawrence Erlbaum Associates.
52. Lehrer, R., Schauble, L., Strom, D., &, Pligge, M., (2001). Similarity of form and substance: Modeling material kind. In, D. K. S. Carver, (Ed.), Cognition and instruction: 25 years of progress (pp. 39-74). Mahwah, NJ: Lawrence Erlbaum Associates.
53. Linn, M. C., Bell, B., &, Davis, E. A., (2004). Internet environments for science education. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
54. Maehr, M. L., &, Midgley, C., (1996). Transforming school cultures. Boulder, CO: Westview Press.
55. Martin, J., &, VanLehn, K., (1995). Student assessment using bayesian nets. Journal of Human-Computer Studies, 42 (6), 575-591.
56. Marx, R. W., Blumenfeld, P. C., Krajcik, J. S., Fishman, B., Soloway, E., Geier, R., &, Tal, R. T., (2004). Inquiry-based science in the middle grades: Assessment of learning in urban systemic reform. Journal of Research in Science Teaching, 41 (10), 1063-1080. (Pubitemid 40002957)
57. Mayer, R. E., (2005). Cognitive theory of multimedia learning. In, R. E. Mayer, (Ed.), The Cambridge handbook of multimedia learning. New York: Cambridge University Press.
58. Mayer, R. E., &, Anderson, R. B., (1992). The instructive animation: Helping students build connections between words and pictures in multimedia learning. Journal of Educational Psychology, 84 (4), 444-452.
59. Metz, K. E., (2004). Children's understanding of scientific inquiry: Their conceptualization of uncertainty in investigations of their own design. Cognition and Instruction, 22 (2): 219-290. (Pubitemid 38808945)
60. Messick, S., (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23 (2), 12-23.
61. Minnesota Department of Education. (2010). Draft test specifications for science. Available at:.
62. Mislevy, R. J., (1994). Evidence and inference in educational assessment. Psychometrika, 59, 439-483.
63. Mislevy, R., &, Haertel, G. D., (2007). Implications of evidence-centered designs for educational testing. Educational Measurement: Issues and Practices, 25 (4), 6-20.
64. National Assessment Governing Board (NAGB). (2008). Science framework for the 2009 national assessment of educational progress. Washington, DC: Author.
65. National Research Council (NRC). (1996). National science education standards. Washington, DC: National Academy Press.
66. National Research Council (NRC). (2006). Systems for state science assessment. Washington, DC: The National Academies Press.
67. National Research Council (NRC). (2011). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. Washington, DC: The National Academies Press.
68. Nersessian, N. J., (2008). Creating scientific concepts. Cambridge MA: MIT Press.
69. Nolin, M., &, Chandler, K., (1996). Use of cognitive laboratories and recorded interviews in the national household education survey (No. NCES-96-332). Rockville, MD: Westat.
70. Norman, D. A., (1993). Things that makes us smart. Reading, MA: Adddison-Wesley.
71. Pashler, H., Bain, P., Bottge, B., Graesser, A., Koedinger, K., McDaniel, M., &, Metcalfe, J., (2007). Organizing instruction and study to improve student learning (No. NCER 2007-2004). Washington, DC: National Center for Education Research, Institute of Education Sciences, U.S. Department of Education.
72. Pellegrino, J., Chudowsky, N., &, Glaser, R., (2001). Knowing what students know: The science and design of educational assessment. Washington, DC: National Academy Press.
73. Perie, M., Marion, S., &, Gong, B., (2009). Moving toward a comprehensive assessment system: A framework for considering interim assessments. Educational Measurement: Issues & Practice, 28 (3), 5-13.
74. Phillips, S. E., (1993). Legal implications of high-stakes assessments: What states should know. Oak Brook, IL: North Central Regional Laboratory.
75. Quellmalz, E. S., DeBarger, A., Haertel, G., &, Kreikemeier, P., (2005). Validities of science inquiry assessments: Final report. Menlo Park, CA: SRI International.
76. Quellmalz, E. S., &, Haertel, G., (2004). Technology supports for state science assessment systems. Unpublished manuscript, Washington DC.
77. Quellmalz, E. S., &, Moody, M., (2004). Models for multi-level state science assessment systems. Report commissioned by the National Research Council Committee on Test Design for K-12 Science Achievement.
78. Quellmalz, E. S., &, Pellegrino, J. W., (2009). Technology and testing. Science, 323, 75-79.
79. Quellmalz, E., Schank, P., Hinojosa, T., &, Padilla, C., (1999). Performance assessment links in science (pals) (No. ERIC Digest Series E-DO-TM-99-04). College Park, MD: ERIC Clearinghouse on Assessment and Evaluation.
80. Quellmalz, E. S., Timms, M. J., &, Schneider, S. A., (2009). Assessment of student learning in science simulations and games. Washington, D. C.: National Research Council.
81. Quellmalz, E., Timms, M., &, Silberglitt, M. D., (2011). Integrating simulation-based assessments into state science assessment systems. San Francisco: WestEd.
82. Quintana, C., Reiser, B. J., Davis, E. A., Krajcik, J., Fretz, E., Duncan, R. G., Kyza, E., Edelson, D., &, Soloway, E., (2004). A scaffolding design framework for software to support science inquiry. Journal of the Learning Sciences, 13 (3), 337-386. (Pubitemid 39057719)
83. Rebetez, C., Sangin, M., Betrancourt, M., &, Dillenbourg, P., (2004). Effects of collaboration in the context of learning from animations. Proceedings of EARLI SIG meeting on Comprehension of Text and Graphics: Basic and applied issues, 9-11, September 2004, Valencia (Spain), 187-192.
84. Reiser, B. J., (2004). Scaffolding complex learning: The mechanisms of structuring and problematizing student work. Journal of the Learning Sciences, 13 (3), 273-304. (Pubitemid 39063558)
85. Reiser, B. J., Tabak, I., Sandoval, W. A., Smith, B. K., Steinmuller, F., &, Leone, A. J., (2001). BGuILE: Strategic and conceptual scaffolds for scientific inquiry in biology classrooms. In, S. M. Carver, &, D. Khlar, (Eds.), Cognition and instruction: Twenty-five years of progress (pp. 263-305). Mahwah, NJ: Erlbaum.
86. Rieber, L. P., Tzeng, S., &, Tribble, K., (2004). Discovery learning, representation, and explanation within a computer-based simulation. Computers and Education, 27 (1), 45-58.
87. Rivet, A., &, Krajcik, J. S., (2004). Achieving standards in urban systemic reform: An example of a sixth grade project-based science curriculum. Journal of Research in Science Teaching, 41 (7), 669-692.
88. Rose, D. H., &, Meyer, A., (2000). Universal design for learning. Journal of Special Education Technology, 15 (1), 67-70.
89. Scalise, K., Timms, M., Moorjani, A., Clark, L., Holtermann, K., &, Irvin, P. S., (2011). Student learning in science simulations: Design features that promote learning gains. Journal of Research in Science Teaching, 48 (9), 1050-1078.
90. Schwartz, D. L., &, Heiser, J., (2006). Spatial representations and imagery in learning. In, R. K. Sawyer, (Ed.), The Cambridge handbook of the learning sciences. Cambridge: Cambridge University Press.
91. Simon, H. A., (1980). Problem solving and education. In, D. T. Tuma, &, F. Reif, (Eds.), Problem solving and education: Issues in teaching and research (pp. 81-96). Hillsdale, NJ: Erlbaum.
92. Stewart, J., Cartier, J. L., &, Passmore, C. M., (2005). Developing understanding through model-based inquiry. In, M. S. Donovan, &, J. D. Bransford, (Eds.), How students learn (pp. 515-565). Washington, DC: The National Academies Press.
93. Stewart, I., &, Golubitsky, M., (1992). Fearful symmetry: Is God a geometer ? Cambridge MA: Blackwell Cambridge.
94. Stiggins, R., (2006). Balanced assessment systems: Redefining excellence in assessment. Educational Testing Service. Portland: Oregon.
95. Twing, J. S., &, Dolan, R. P., (2008). UD-CBT guidelines. Retrieved March 5th, 2008 from.
96. Vattam, S. S., Goel, A. K., Rugaber, S., Hmelo-Silver, C. E., Jordan, R., Gray, S., &, Sinha, S., (2011). Understanding complex natural systems by articulating structure-behavior-function models. Journal of Educational Technology & Society, 14 (1), 66-81.
97. Vye, N. J., Schwartz, D. L., Bransford, J. D., Barron, B. J., Zech, L., Cognition Technology Group at Vanderbilt. (1998). Smart environments that support monitoring, reflection, and revision. In, D. Hacker, J. Dunlosky, &, A. C. Graesser, (Eds.), Metacogition in educational theory and practice (pp. 305-346). Mahwah, NJ: Erlbaum.
98. Wang, S., (2005). Online or paper: Does delivery affect results ? San Antonio, TX: Harcourt Assessment, Inc.
99. White, B. Y., &, Frederiksen, J. R., (1998). Inquiry, modeling, and metacognition: Making science accessible to all students. Cognition and Instruction, 16 (1): 3-118. (Pubitemid 128343737)
100. Zucker, S., Sassman, C., &, Case, B. J., (2004). Cognitive labs. San Antonio, TX: Harcourt Assessment.