Next-generation environments for assessing and promoting complex science learning

Journal of Educational Psychology

Volumn 105, Issue 4, 2013, Pages 1100-1114

  Quellmalz, Edys S ^a   Davenport, Jodi L ^a   Timms, Michael J ^b   DeBoer, George E ^c   Jordan, Kevin A ^a   Huang, Chun Wei ^a   Buckley, Barbara C ^a  

^a WestEd   (United States)

^b Australian Council for Educational Research   (Australia)

^c AAAS   (United States)

Author keywords

Educational assessment; Multimedia; Psychometrics; Science education; Technology enhanced assessment

Indexed keywords

EID: 84887561178     PISSN: 00220663     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0032220     Document Type: Article

References (74)

1. Achieve. (2012). Next generation science standards. Retrieved from http://www.nextgenscience.org/nnext-generation-science-standards
2. Adams, W. K., Reid, S., Lemaster, R., McKagan, S. B., Perkins, K. K., Dubson, M., & Wieman, C. E. (2008). A study of educational simulations:Part 1-Engagement and learning. Journal of Interactive Learning Research, 19, 397-419.
3. Ainsworth, S. (2008). The educational value of multiple-representations when learning complex scientific concepts. In J. K. Gilbert, M. Reiner, & M. Nakhleh (Eds.), Visualization: Theory and practice in science education (pp. 191-208). New York, NY: Springer. doi:10.1007/978-1-4020-5267-5_9
4. American Association for the Advancement of Science. (1993). Benchmarks for science literacy. New York, NY: Oxford University Press.
5. Betrancourt, M. (2005). The animation and interactivity principles. In R. E. Mayer (Ed.), Handbook on multimedia learning (pp. 287-296). New York, NY: Cambridge University Press. doi:10.1017/CBO9780511816819.019
6. Bransford, J. D., Brown, A. L., & Cocking, R. R. (2000). How people learn: Brain, mind, experience, and school. Washington, DC: National Academy Press.
7. Brennan, R. (2001). mGENOVA [Computer software]. Retrieved from http://www.education.uiowa.edu/centers/casma/computer-programs.aspx
8. Buckley, B. C., Gobert, J., Horwitz, P., & O'Dwyer, L. (2010). Looking inside the black box: Assessing model-based learning and inquiry in BioLogica. International Journal of Learning Technology, 5, 166-190. doi:10.1504/IJLT.2010.034548
9. Buckley, B. C., & Quellmalz, E. S. (2013). Supporting and assessing complex biology learning with computer-based simulations and representations. In D. Treagust & C.-Y. Tsui (Eds.), Multiple representations in biological education (pp. 247-267). Dordrecht, the Netherlands:Springer. doi:10.1007/978-94-007-4192-8_14
10. Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105. doi:10.1037/h0046016
11. Clark, R. C., & Mayer, R. E. (2011). E-learning and the science of instruction: Proven guidelines for consumers and designers of multimedia learning. San Francisco, CA: Pfeiffer. doi:10.1002/9781118255971
12. Clement, J. (1989). Learning via model construction and criticism: Protocol evidence on sources of creativity in science. In J. A. Glover, R. R. Ronning, & C. R. Reynolds (Eds.), Handbook of creativity: Assessment, theory and research (pp. 341-381). New York, NY: Plenum Press.
13. College Board. (2009). Science: College Boards standards for college success. Retrieved from http://professionals.collegeboard.com/profdownload/cbscs-science-standards-2009.pdf
14. Collins, A., Brown, J. S., & Newman, S. (1989). Cognitive apprenticeship:Teaching the craft of reading, writing, and mathematics. In L. B. Resnick (Ed.), Knowing, learning, and instruction: Essays in honor of Robert Glaser (pp. 453-494). Hillsdale, NJ: Erlbaum.
15. Darling-Hammond, L. (2010). Performance counts: Assessment systems that support high-quality learning. Washington, DC: CCSSO.
16. Darling-Hammond, L., & Pecheone, R. (2010). Developing an internationally comparable balanced assessment system that supports high-quality learning. Presented at the National Conference on Next Generation K-12 Assessment Systems, Center for K-12 Assessment & Performance Management, with the Education Commission of the States (ECS) and the Council of Great City Schools (CGCS), Washington, DC.
17. Dede, C. (2009). Immersive interfaces for engagement and learning. Science, 323, 66-69. doi:10.1126/science.1167311
18. de Jong, T. (2006). Technological advances in inquiry learning. Science, 312, 532-533. doi:10.1126/science.1127750
19. Geier, R., Blumenfeld, P., Marx, R., Krajcik, J., Fishman, B., & Soloway, E. (2008). Standardized test outcomes of urban students participating in standards and project based science curricula. Journal of Research in Science Teaching, 45, 922-939. doi:10.1002/tea.20248
20. Hegarty, M. (2004). Dynamic visualizations and learning: Getting to the difficult questions. Learning and Instruction, 14, 343-351. doi:10.1016/ j.learninstruc.2004.06.007
21. Hmelo-Silver, C. E., Jordan, R., Liu, L., Gray, S., Demeter, M., Rugaber, S.,.. Goel, A. (2008). Focusing on function: Thinking below the surface of complex science systems. Science Scope, 31, 27-35.
22. Horwitz, P., Gobert, J. D., Buckley, B. C., & O'Dwyer, L. M. (2010). Learning genetics with dragons: From computer-based manipulatives to hypermodels. In M. J. Jacobson & P. Reimann (Eds.), Designs for learning environments of the future: International perspectives from the learning sciences (pp. 61-87). New York, NY: Springer. doi:10.1007/ 978-0-387-88279-6_3
23. Ioannidou, A., Repenning, A., Webb, D., Keyser, D., Luhn, L., & Daetwyler, C. (2010). Mr. Vetro: A collective simulation for teaching health science. International Journal of Computer-Supported Collaborative Learning, 5, 141-166. doi:10.1007/s11412-010-9082-8
24. Jacobson, M. J. (2001). Problem solving, cognition, and complex systems:Differences between experts and novices. Complexity, 6, 41-49. doi:10.1002/cplx.1027
25. King, K. (2011). Balanced, multilevel science assessment systems. Presented at the National Conference on Student Assessment, Orlando, FL.
26. Kolodner, J. L., Camp, P. J., Crismond, D., Fasse, B., Gray, J., Holbrook, J.,.. Ryan, M. (2003). Problem-based learning meets case-based reasoning in the middle-school science classroom: Putting learning by design into practice. The Journal of the Learning Sciences, 12, 495-547. doi:10.1207/S15327809JLS1204_2
27. Koomen, M. (2006). The development and implementation of a computerbased assessment of science literacy in PISA 2006. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
28. Krajcik, J., Marx, R., Blumenfeld, P., Soloway, E., & Fishman, B. (2000). Inquiry-based science supported by technology: Achievement and motivation among urban middle school students. Paper presented at the American Educational Research Association, New Orleans, LA.
29. Kriz, S., & Hegarty, M. (2007). Top-down and bottom-up influences on learning from animations. International Journal of Human-Computer Studies, 65, 911-930. doi:10.1016/j.ijhcs.2007.06.005
30. Kühl, T., Scheiter, K., Gerjets, P., & Edelmann, J. (2011). The influence of text modality on learning with static and dynamic visualizations. Computers in Human Behavior, 27, 29-35. doi:10.1016/j.chb.2010.05.008
31. Larkin, J., & Simon, H. (1987). Why a diagram is (sometimes) worth ten thousand words. Cognitive Science, 11, 65-100. doi:10.1111/j.1551-6708.1987.tb00863.x
32. Lee, H., Plass, J. L., & Homer, B. D. (2006). Optimizing cognitive load for learning from computer-based science simulations. Journal of Educational Psychology, 98, 902-913. doi:10.1037/0022-0663.98.4.902
33. Lee, O. (2002). Science inquiry for elementary students from diverse backgrounds. In W. G. Secada (Ed.), Review of research in education (Vol. 26, pp. 23-69). Washington, DC: American Educational Research Association.
34. Lehrer, R., & Schauble, L. (2002). Symbolic communication in mathematics and science: Co-constituting inscription and thought. In E. D. A. J. Byrnes (Ed.), Language, literacy, and cognitive development. The development and consequences of symbolic communication (pp. 167-192). Mahwah, NJ: Erlbaum.
35. Lehrer, R., Schauble, L., Strom, D., & Pligge, M. (2001). Similarity of form and substance: Modeling material kind. In D. K. S. Carver (Ed.), Cognition and instruction: 25 years of progress (pp. 39-74). Mahwah, NJ: Erlbaum.
36. Linn, M. C., & Eylon, B.-S. (2011). Science learning and instruction:Taking advantage of technology to promote knowledge integration. New York, NY: Routledge.
37. Liu, O. L., Lee, H.-S., & Linn, M. C. (2011). Measuring knowledge integration: Validation of four-year assessments. Journal of Research in Science Teaching, 48, 1079-1107. doi:10.1002/tea.20441
38. Loehlin, J. C. (1998). Latent variable model: An introduction to factor, path, and structural analysis. Mahwah, NJ: Erlbaum.
39. Lowe, R., & Schnotz, W. (2008). Learning with animation: Research implications for design. Cambridge, NY: Cambridge University Press.
40. Lowe, R. K., & Schnotz, W. (Eds.). (2007). Learning with animation. New York, NY: Cambridge University Press.
41. Marx, R. W., Blumenfeld, P. C., Krajcik, J. S., Fishman, B., Soloway, E., Geier, R., & Tal, R. T. (2004). Inquiry-based science in the middle grades: Assessment of learning in urban systemic reform. Journal of Research in Science Teaching, 41, 1063-1080. doi:10.1002/tea.20039
42. Mayer, R. E. (Ed.). (2005a). The Cambridge handbook of multimedia learning. New York, NY: Cambridge University Press. doi:10.1017/CBO9780511816819
43. Mayer, R. E. (2005b). Cognitive theory of multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 31-48). New York, NY: Cambridge University Press. doi:10.1017/CBO9780511816819.004
44. Mayer, R. E., & Johnson, C. (2008). Revising the redundancy principle in multimedia learning. Journal of Educational Psychology, 100, 380-386. doi:10.1037/0022-0663.100.2.380
45. Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23, 12-23.
46. Metz, K. E. (2004). Children's understanding of scientific inquiry: Their conceptualization of uncertainty in investigations of their own design. Cognition and Instruction, 22, 219-290. doi:10.1207/s1532690xci2202_3
47. Minnesota Department of Education. (2010). Draft test specifications for science. Retrieved from http://education.state.mn.us/MDE/Accountability_Programs/Assessment_and_Testing/Assessments/MCA/TestSpecs/index.html
48. Mislevy, R. J., Almond, R. G., & Lukas, J. F. (2003). A brief introduction to evidence-centered design. Princeton, NJ: Educational Testing Service.
49. National Assessment Governing Board. (2008). Science framework for the 2009 national assessment of educational progress. Washington, DC:National Assessment Governing Board.
50. National Center for Education Statistics. (2012). The nation's report card:Science in action: Hands-on and interactive computer tasks from the 2009 science assessment (Report No. NCES 2012-468). Washington, DC: Institute of Education Sciences, U.S. Department of Education.
51. National Research Council. (2011). A framework for k-12 science education:Practices, crosscutting concepts, and core ideas. Washington, DC:The National Academies Press.
52. Nersessian, N. J. (2008). Creating scientific concepts. Cambridge, MA: MIT Press.
53. Papaevripidou, M., Constantinou, C. P., & Zacharia, Z. C. (2007). Modeling complex marine ecosystems: An investigation of two teaching approaches with fifth graders. Journal of Computer Assisted Learning, 23, 145-157. doi:10.1111/j.1365-2729.2006.00217.x
54. Pedone, R., Hummel, J. E., & Holyoak, K. J. (2001). The use of diagrams in analogical problem solving. Memory & Cognition, 29, 214-221. doi:10.3758/BF03194915
55. Pellegrino, J., Chudowsky, N., & Glaser, R. (2001). Knowing what students know: The science and design of educational assessment. Washington, DC: National Academy Press.
56. Plass, J. L., Homer, B. D., & Hayward, E. O. (2009). Design factors for educationally effective animations and simulations. Journal of Computing in Higher Education, 21, 31-61. doi:10.1007/s12528-009-9011-x
57. Quellmalz, E. S., & Haertel, G. (2004). Technology supports for state science assessment systems. Unpublished manuscript.
58. Quellmalz, E. S., & Pellegrino, J. W. (2009). Technology and testing. Science, 323, 75-79. doi:10.1126/science.1168046
59. Quellmalz, E., Schank, P., Hinojosa, T., & Padilla, C. (1999). Performance assessment links in science (PALS; Report No. EDO-TM-99-04). College Park, MD: ERIC Clearinghouse on Assessment and Evaluation.
60. Quellmalz, E. S., Timms, M. J., & Buckley, B. (2010). The promise of simulation-based science assessment: The calipers project. International Journal of Learning Technology, 5, 243-263. doi:10.1504/IJLT.2010.037306
61. Quellmalz, E. S., Timms, M. J., Buckley, B. C., Davenport, J., Loveland, M., & Silberglitt, M. D. (2011). 21st century dynamic assessment. In J. Clarke-Midura, M. Mayrath, & C. Dede (Eds.), Technology-based assessments for 21st century skills: Theoretical and practical implications from modern research (pp. 55-89). Charlotte, NC: Information Age.
62. Quellmalz, E. S., Timms, M. J., Silberglitt, M. D., & Buckley, B. C. (2012). Science assessments for all: Integrating science simulations into balanced state science assessment systems. Journal of Research in Science Teaching, 49, 363-393.
63. Rieber, L. P., Tzeng, S., & Tribble, K. (2004). Discovery learning, representation, and explanation within a computer-based simulation. Computers and Education, 27, 45-58.
64. Rivet, A., & Krajcik, J. S. (2004). Achieving standards in urban systemic reform: An example of a sixth grade project-based science curriculum. Journal of Research in Science Teaching, 41, 669-692. doi:10.1002/tea.20021
65. Schwartz, D. L., & Black, T. (1999). Inferences through imagined actions:Knowing by simulated doing. Journal of Experimental Psychology: Learning, Memory, and Cognition, 25, 116-136. doi:10.1037/0278-7393.25.1.116
66. Schwartz, D. L., & Heiser, J. (2006). Spatial representations and imagery in learning. In R. K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 283-298). Cambridge, England: Cambridge University Press.
67. Simon, H. A. (1980). Problem solving and education. In D. T. Tuma & F. Reif (Eds.), Problem solving and education: Issues in teaching and research (pp. 81-96). Hillsdale, NJ: Erlbaum.
68. Slotta, J. D., & Chi, M. T. H. (2006). Helping students understand challenging topics in science through ontology training. Cognition and Instruction, 24, 261-289. doi:10.1207/s1532690xci2402_3
69. Smith, C. L., Wiser, M., Anderson, C. W., & Krajcik, J. (2006). Implications of research on children's learning for standards and assessment: A proposed learning progression for matter and the atomic-molecular theory. Measurement: Interdisciplinary Research and Perspectives, 4, 1-98. doi:10.1080/15366367.2006.9678570
70. Stewart, J., Cartier, J. L., & Passmore, C. M. (2005). Developing understanding through model-based inquiry. In M. S. Donovan & J. D. Bransford (Eds.), How students learn (pp. 515-565). Washington, DC:The National Academies Press.
71. Tversky, B., Heiser, J., Lozano, S., MacKenzie, R., & Morrison, J. (2008). Enriching animations. In R. K. Lowe & W. Schnotz (Eds.), Learning with animation: Research and design implications (pp. 263-285). New York, NY: Cambridge University Press.
72. van Merriënboer, J. J. G., & Kester, L. (2005). The four-component instructional design model: Multimedia principles in environments for complex learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 71-94). New York, NY: Cambridge University Press. doi:10.1017/CBO9780511816819.006
73. Vattam, S. S., Goel, A. K., Rugaber, S., Hmelo-Silver, C. E., Jordan, R., Gray, S., & Sinah, S. (2011). Understanding complex natural systems by articulating structure-behavior-function models. Journal of Educational Technology & Society, 14, 66-81.
74. Zacharia, Z. C. (2007). Comparing and combining real and virtual experimentation:An effort to enhance students' conceptual understanding of electric circuits. Journal of Computer Assisted Learning, 23, 120-132. doi:10.1111/j.1365-2729.2006.00215.x