Comparing three online testing modalities: Using static, active, and interactive online testing modalities to assess middle school students' understanding of fundamental ideas and use of inquiry skills related to ecosystems

Journal of Research in Science Teaching

Volumn 51, Issue 4, 2014, Pages 523-554

  DeBoer, George E ^a   Quellmalz, Edys S ^b   Davenport, Jodi L ^b   Timms, Michael J ^c   Herrmann Abell, Cari F ^a   Buckley, Barbara C ^b   Jordan, Kevin A ^b   Huang, Chun Wei ^b   Flanagan, Jean C ^d  

^a AAAS   (United States)

^b WestEd   (United States)

^c Australian Council for Educational Research   (Australia)

^d SMITHSONIAN INSTITUTION   (United States)

Author keywords

assessment; ecosystems; inquiry; integrated knowledge structures; online testing; science education

Indexed keywords

ECOSYSTEMS; KNOWLEDGE BASED SYSTEMS; TEACHING;

ASSESSMENT; INQUIRY; KNOWLEDGE STRUCTURES; ON-LINE TESTING; SCIENCE EDUCATION;

STUDENTS;

EID: 84896319756     PISSN: 00224308     EISSN: 10982736     Source Type: Journal    
DOI: 10.1002/tea.21145     Document Type: Article

References (74)

1. AAAS Project 2061. (n.d.). AAAS Project 2061 Science Assessment Website. Retrieved from http://assessment.aaas.org. NSF ESI-0352473; G.E. DeBoer, Principal Investigator.
2. Adams, W. K., Reid, S., Lemaster, R., McKagan, S. B., Perkins, K. K., Dubson, M., &, Wieman, C. E., (2008). A study of educational simulations part 1-Engagement and learning. Journal of Interactive Learning Research, 19 (3), 397-419.
3. American Association for the Advancement of Science. (1993). Benchmarks for science literacy. New York, NY: Oxford University Press.
4. Amsel, E., &, Brock, S., (1996). The development of evidence evaluation skills. Cognitive Development, 11, 523-550.
5. Austin, K. A., (2009). Multimedia learning: Cognitive individual differences and display design techniques predict transfer learning with multimedia learning modules. Computers & Education, 53, 1339-1354.
6. Baxter, G. P., (1995). Using computer simulations to assess hands-on science learning. Journal of Science Education and Technology, 4, 21-27.
7. Bennett, R. E., (2003). Online assessment and the comparability of score meaning. Princeton, NJ: ETS.
8. Bond, T. G., &, Fox, C. M., (2007). Applying the Rasch model: Fundamental measurement in the human sciences (Second ed.). Mahwah, NJ: Lawrence Erlbaum Associates.
9. Bridgeman, B., (2009). Experiences from large-scale computer-based testing in the USA. In, F. Scheurermann, &, J. Björnsson, (Eds.), The transition to computer-based assessment: New approaches to skills assessment and implications for large-scale testing (pp. 39-44). Luxembourg, LU: Office for Official Publications of the European Communities.
10. Brody, M., &, Koch, H., (1990). An assessment of 4th-, 8th-, and 11th-grade students' knowledge related to marine science and natural resource issues. Journal of Environmental Science, 22 (2), 16-26.
11. Buckley, B. C., Gobert, J., Horwitz, P., &, O'Dwyer, L., (2010). Looking inside the black box: Assessing model-based learning and inquiry in BioLogica. International Journal of Learning Technologies, 5 (2), 166-190.
12. Busco, V., (2009). Shifting from paper-and-pencil to computer-based testing: Requisites, challenges and consequences for testing outcomes-A Croatian perspective. In, F. Scheurermann, &, J. Björnsson, (Eds.), The transition to computer-based assessment: New approaches to skills assessment and implications for large-scale testing (pp. 117-119). Luxembourg, LU: Office for Official Publications of the European Communities.
13. Cagiltay, N., &, Yaman, S., (2013). How can we get benefits of computer-based testing in engineering education ? Computer Applications in Engineering Education 21 (2), 287-293.
14. Chang, H., &, Linn, M. C., (2013). Scaffolding learning from molecular visualizations. Journal of Research in Science Teaching, 50 (7), 858-886.
15. Costello, A. B., &, Osborne, J., (2005). Best practices in exploratory factor analysis: four recommendations for getting the most from your analysis. Practical Assessment Research & Evaluation, 10 (7), Available online: http://pareonline.net/getvn.asp?v=10&n=7
16. Csapõ, B., Molnár, G., &, Tõth, K. R., (2009). Comparing paper-and-pencil and online assessment of reasoning skills: A pilot study for introducing TAO in large-scale assessment in Hungary. In, F. Scheurermann, &, J. Björnsson, (Eds.), The transition to computer-based assessment: New approaches to skills assessment and implications for large-scale testing (pp. 120-126). Luxembourg, LU: Office for Official Publications of the European Communities.
17. DeBoer, G. E., Herrmann-Abell, C. F., Gogos, A., Michiels, A., Regan, T., &, Wilson, P., (2008). Assessment linked to science learning goals: Probing student thinking through assessment. In, J. Coffey, R. Douglas, &, C. Stearns, (Eds.), Assessing student learning: Perspectives from research and practice (pp. 231-252). Arlington, VA: NSTA Press.
18. DeBoer, G. E., Herrmann-Abell, C. F., Wertheim, J. A., &, Roseman, J. E., (2009, April). Assessment linked to middle school science learning goals: A report on field test results for four middle school science topics. Paper presented at the National Association for Research in Science Teaching (NARST) Annual Conference, Garden Grove, CA.
19. DeBoer, G. E., Lee, H. S., &, Husic, F., (2008). Assessing integrated understanding of science. In, Y. Kali, M. C. Linn, &, J. E. Roseman, (Eds.), Coherent science education: Implications for curriculum, instruction, and policy (pp. 153-182). New York, NY: Columbia University Teachers College Press. 153-182.
20. Driver, R., Squires, A., Rushworth, P., &, Wood-Robinson, V., (1994). Making sense of secondary science: Research into children's ideas. New York, NY: Routledge.
21. Gallegos, L., Jerezano, M., &, Flores, F., (1994). Preconceptions and relations used by children in the construction of food chains. Journal of Research in Science Teaching, 31 (3), 259-272.
22. Griffiths, A., &, Grant, B., (1985). High school students' understanding of food webs: Identification of a learning hierarchy and related misconceptions. Journal of Research in Science Teaching, 22 (5), 421-436.
23. Halldõrsson, A. M., McKelvie, P., &, Björnsson, J. K., (2009). Are Icelandic boys really better on computerized tests than conventional ones? Interaction between gender, test modality and test performance. In, F. Scheurermann, &, J. Björnsson, (Eds.), The transition to computer-based assessment: New approaches to skills assessment and implications for large-scale testing (pp. 178-193). Luxembourg, LU: Office for Official Publications of the European Communities.
24. Horwitz, P., Gobert, J. D., Buckley, B. C., &, O'Dwyer, L. M., (2010). Learning genetics with dragons: From computer-based manipulatives to hypermodels. In, M. J. Jacobson, &, P. Reimann, (Eds.), Designs for learning environments of the future: International perspectives from the learning sciences (pp. 61-87). New York, NY: Springer.
25. Kaiser, H. F., (1960). The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20, 141-151.
26. Kikis-Papadakis, K., &, Kollias, A., (2009). Reflections on paper-and-pencil tests to eAssessments: Narrow and broadband paths to 21st century challenges. In, F. Scheurermann, &, J. Björnsson, (Eds.), The transition to computer-based assessment: New approaches to skills assessment and implications for large-scale testing (pp. 99-103). Luxembourg, LU: Office for Official Publications of the European Communities.
27. King, K., (2011). Balanced, multilevel science assessment systems. Presented at the National Conference on Student Assessment, Orlando, FL.
28. Kingston, N.M., (2009). Comparability of computer- and paper-administered multiple-choice tests for K-12 populations: A synthesis. Applied Measurement in Education, 22, 22-37.
29. Klahr, D., Fay, A., &, Dunbar, K., (1993). Heuristics for scientific experimentation: A developmental study. Cognitive Psychology, 25, 111-146.
30. Koomen, M., (2006) The development and implementation of a computer-based assessment of science literacy in PISA 2006. Paper presented at the Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
31. Krajcik, J., Marx, R., Blumenfeld, P., Soloway, E., &, Fishman, B., (2000). Inquiry-based science supported by technology: Achievement and motivation among urban middle school students. Paper presented at the American Educational. Research Association, New Orleans, LA.
32. Kuhn, D., &, Phelps, E., (1982). The development of problem-solving strategies. In, H. Reese, (Ed.), Advances in child development and behavior (Vol. 17, pp. 1-44). New York, NY: Academic Press.
33. Kuhn, D., Amsel, E., &, O'Loughlin, M., (1988). The development of scientific thinking skills. San Diego, CA.
34. Kuo, C., &, Wu, H., (2013). Toward an integrated model for designing assessment systems: An analysis of the current status of computer-based assessments in science. Computers & Education, 68, 388-403.
35. Lee, M., (2009). CBAs in Korea: Experiences, results and challenges. In, F. Scheurermann, &, J. Björnsson, (Eds.), The transition to computer-based assessment: New approaches to skills assessment and implications for large-scale testing (pp. 194-200). Luxembourg, LU: Office for Official Publications of the European Communities.
36. Levin, D., &, Fletcher, G., (2013). Will schools be 'technology-ready' to administer nation's new assessments? In coming together to raise achievement: New assessments for the common core state standards (pp. 44-47). Princeton, NJ: ETS Center for K-12 Assessment Performance Management. Available at http://k12center.org/rsc/pdf/Coming-Together-June-2013.pdf
37. Linacre, J. M., (2012). Winsteps® (Version 3.75.0) [Computer Software]. Beaverton, Oregon: Winsteps.com. Retrieved January 1, 2012. Available from http://www.winsteps.com/
38. Liu, O. L., Lee, H., &, Linn, M. C., (2011). Measuring knowledge integration: Validation of four-year assessments. Journal of Research in Science Teaching, 48 (9), 1079-1107.
39. Martin, R., (2009). Utilising the potential of computer delivered surveys in assessing scientific literacy. In, F. Scheurermann, &, J. Björnsson, (Eds.), The transition to computer-based assessment: New approaches to skills assessment and implications for large-scale testing (pp. 172-177). Luxembourg, LU: Office for Official Publications of the European Communities.
40. Meijer, R., (2009). Transition to computer-based assessment: Motivations and considerations. In, F. Scheurermann, &, J. Björnsson, (Eds.), The transition to computer-based assessment: New approaches to skills assessment and implications for large-scale testing (pp. 104-107). Luxembourg, LU: Office for Official Publications of the European Communities.
41. Minnesota Department of Education, (2010). Draft test specifications for science. Available at: http://education.state.mn.us/MDE/Accountability-Programs/ Assessment-and-Testing/Assessments/MCA/TestSpecs/index.html
42. Munson, B., (1991). Relationships between an individual's conceptual ecology and the individual's conceptions of ecology. Unpublished doctoral thesis. The Graduate School of the University of Minnesota, Minneapolis.
43. National Assessment Governing Board (NAGB). (2008). Science framework for the 2009 national assessment of educational progress. Washington, DC: National Assessment Governing Board.
44. National Center for Education Statistics. (2012). The nation's report card. Science in action: Hands on and interactive computer tasks from the 2009 science assessment (NCES 2012-468). Washington, DC: Institute of Education Sciences, U.S. Department of Education.
45. National Research Council. (1996). National Science Education Standards. Washington, DC: National Academies Press.
46. Neumann, D. L., Hood, M., &, Neumann, M. M., (2012). An evaluation of computer-based interactive simulations in the assessment of statistical concepts. International Journal for Technology in Mathematics Education, 19, 17-23.
47. Neumann, D. L., (2010). Using interactive simulations in assessment: The use of computer-based simulations in the assessment of statistical concepts. The International Journal for Technology in Mathematics Education, 17, 43-51.
48. NGSS Lead States. (2013). Next generation science standards: For states, by states. Washington, DC: The National Academies Press.
49. Plass, J. L., Homer, B. D., &, Hayward, E. O., (2009). Design factors for educationally effective animations and simulations. Journal of Computing in Higher Education, 21 (1), 31-61.
50. Plass, J. L., Milne, C., Homer, B. D., Schwartz, R. N., Hayward, E. O., Jordan, T.,., Barrientos, J., (2012). Investigating the effectiveness of computer simulations for chemistry learning. Journal of Research in Science Teaching, 49 (3), 394-419.
51. Quellmalz, E. S., Timms, M. J., Silberglitt, M. D., &, Buckley, B. C., (2012). Science assessments for all: Integrating science simulations into balanced state science assessment systems. Journal of Research in Science Teaching, 49 (3), 363-393.
52. Quellmalz, E. S., Davenport, J. L., Timms, M. J., DeBoer, G. E., Jordan, K. A., Huang, C. W., &, Buckley, B. C., (2013). Next-generation environments for assessing and promoting complex science reasoning. Journal of Educational Psychology, 105 (4), 1100-1114.
53. Rebetez, C., Sangin, M., Betrancourt, M., &, Dillenbourgh, P., (2004) Effects of collaboration in the contenxt of learning from animations. Proceedngs of EARLI SIG meeting on Comprehension of Text and Graphics: Basic and applied issues, 9-11, September 2004, Balencia (Spain), 187-192.
54. Reiner, M., &, Eilam, B., (2001). Conceptual classroom environment-A system view of learning. International Journal of Science Education, 23 (6), 551-568.
55. Rieber, L. P., Tzeng, S., &, Tribble, K., (2004). Discovery learning, representation, and explanation within a computer-based simulation. Computers and Education, 27 (1), 45-58.
56. Ripley, M., (2009). Transformational computer-based testing. In, F. Scheurermann, &, J. Björnsson, (Eds.), The transition to computer-based assessment: New approaches to skills assessment and implications for large-scale testing (pp. 92-98). Luxembourg, LU: Office for Official Publications of the European Communities.
57. Scalise, K., Timms, M., Moorjani, A., Clark, L., Holtermann, K., &, Irvin, P. S., (2011). Student learning in science simulations: Design features that promote learning gains. Journal of Research in Science Teaching, 48 (9), 1050-1078.
58. Schauble, L., &, Glaser, R., (1990). Scientific thinking in children and adults. Contributions to Human Development, 21, 9-27.
59. Schauble, L., Klopfer, L., &, Raghavan, K., (1991). Students' transition from an engineering model to a science model of experimentation. Journal of Research in Science Teaching, 28, 859-882.
60. Scheurermann F., Björnsson J., (Eds.), (2009). The transition to computer-based assessment: New approaches to skills assessment and implications for large-scale testing. European Commission JRC Scientific and Technical Report. Luxembourg, LU: Office for Official Publications of the European Communities.
61. Schollum, B., (1983). Arrows in science diagrams: Help or hindrance for pupils ? Research in Science Education 13, 45-59.
62. Schwartz, D. L., &, Heiser, J., (2006). Spatial representations and imagery in learning. In, R. K. Sawyer, (Ed.), The Cambridge handbook of the learning sciences. New York, NY: Cambridge University Press.
63. Sørensen, H., &, Andersen, A. M., (2009). How did Danish students solve the PISA CBAS items? Right and wrong answers from a gender perspective. In, F. Scheurermann, &, J. Björnsson, (Eds.), The transition to computer-based assessment: New approaches to skills assessment and implications for large-scale testing (pp. 201-208). Luxembourg, LU: Office for Official Publications of the European Communities.
64. Tschirgi, J. E., (1980). Sensible reasoning: A hypothesis about hypotheses. Child Development, 51, 1-10.
65. van Lent, G., (2009). Risks and benefits of CBT versus PBT in high-stakes testing. In, F. Scheurermann, &, J. Björnsson, (Eds.), The transition to computer-based assessment: New approaches to skills assessment and implications for large-scale testing (pp. 83-91). Luxembourg, LU: Office for Official Publications of the European Communities.
66. Wilson, M., (2005). Constructing measures: An item response modeling approach. Mahwah, NJ: Lawrence Erlbaum Associates.
67. Wollman, W., (1977a). Controlling variables: Assessing levels of understanding. Science Education, 61, 371-383.
68. Wollman, W., (1977b). Controlling variables: A neo-Piagetian developmental sequence. Science Education, 61, 385-391.
69. Wright, B. D., &, Stone, M. H., (2004). Making measures. Chicago, IL: Phaneron Press.
70. Zacharia, Z. C., (2007). Comparing and combining real and virtual experimentation: An effort to enhance students' conceptual understanding of electric circuits. Journal of Computer Assisted Learning, 23 (2), 120-132.
71. Zhang, Z. H., &, Linn, M. C., (2011). Can generating representations enhance learning with dynamic visualizations ? Journal of Research in Science Teaching, 48 (10), 1177-1198.
72. Zimmerman, C., (2000). The development of scientific reasoning skills. Developmental Review, 20, 99-149.
73. Zimmerman, C., (2005) The Development of Scientific Reasoning Skills: What Psychologists Contribute to an Understanding of Elementary Science Learning, Report to the National Research Council, Committee on Science Learning Kindergarten through Eighth Grade.
74. Zimmerman, C., &, Glaser, R., (2001). Testing positive versus negative claims: A preliminary investigation of the role of cover story in the assessment of experimental design skills (Tech. Rep. No. 554). Los Angeles, CA: UCLA National Center for Research on Evaluation, Standards, and Student Testing (CRESST).