From Log Files to Assessment Metrics: Measuring Students' Science Inquiry Skills Using Educational Data Mining

Journal of the Learning Sciences

Volumn 22, Issue 4, 2013, Pages 521-563

  Gobert, Janice D ^a,b   Sao Pedro, Michael ^a,b   Raziuddin, Juelaila ^a   Baker, Ryan S ^c  

^a WORCESTER POLYTECHNIC INSTITUTE   (United States)

^b Apprendis LLC   (United States)

^c Howard Hughes Medical Institute   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84887969672     PISSN: 10508406     EISSN: None     Source Type: Journal    
DOI: 10.1080/10508406.2013.837391     Document Type: Article

References (137)

1. Anderson, J. 1993. Rules of the mind, Hillsdale, NJ: Erlbaum.
2. Anderson, J. R., Boyle, C. F., Corbett, A. T. and Lewis, M. W. 1990. Cognitive modeling and intelligent tutoring. Artificial Intelligence, 42: 7 - 49.
3. Baker, R. S., Corbett, A. T., Roll, I. and Koedinger, K. R. 2008. Developing a generalizable detector of when students game the system. User Modeling and User-Adapted Interaction, 18 (3): 287 - 314.
4. Baker, R., Corbett, A. and Wagner, A. 2006. " Human classification of low-fidelity replays of student actions ". In C. Heiner, R. Baker, & K. Yacef (Eds.), Proceedings of the Educational Data Mining Workshop at the 8th International Conference on Intelligent Tutoring Systems, 29 - 36. Jhongli, Taiwan: Springer.
5. Baker, R. and de Carvalho, A. 2008. " Labeling student behavior faster and more precisely with text replays ". In Proceedings of the 1st International Conference on Educational Data Mining Edited by: Baker, R. S., Barnes, T. and Beck, J. E. 38 - 47. Montreal, Canada.
6. Baker, R. S., Mitrovic, A. and Mathews, M. 2010. " Detecting gaming the system in constraint-based tutors ". In Proceedings of the 18th Annual Conference on User Modeling, Adaptation, and Personalization, UMAP 2010. LNCS 6075, Edited by: De Bra, P., Kobsa, P. and Chin, D. 267 - 278. New York, NY: Springer-Verlag.
7. Baker, R. and Yacef, K. 2009. The state of educational data mining in 2009: A review and future visions. Journal of Educational Data Mining, 1 (1): 3 - 17.
8. Beck, J. E. and Mostow, J. 2008. " How who should practice: Using learning decomposition to evaluate the efficacy of different types of practice for different types of students ". In B. Woolf, E. Aimeur, R. Nkambou, & S. Lajoie (Eds.), Proceedings of the 9th International Conference on Intelligent Tutoring Systems, 353 - 362. Berlin, Germany: Springer.
9. Behrens, J. T. (2013, April). Harnessing the currents of the digital ocean. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
10. Ben-David, A. 2008. About the relationship between ROC curves and Cohen's kappa. Engineering Applications of Artificial Intelligence, 21: 874 - 882.
11. Black, P. 1999. Testing: Friend or foe? Theory and practice of assessment and testing, New York, NY: Falmer Press.
12. Brusilovsky, P. and Peylo, C. 2003. Adaptive and intelligent Web-based educational systems. International Journal of Artificial Intelligence in Education, 13 (2-4): 159 - 172.
13. Buckley, B. C., Gobert, J. D. and Horwitz, P. 2006. " Using log files to track students' model-based inquiry ". In Proceedings of the 7th International Conference on Learning Sciences, ICLS, 57 - 63. Mahwah, NJ: Erlbaum.
14. Buckley, B., Gobert, J., Horwitz, P. and O'Dwyer, L. 2010. Looking inside the black box: Assessments and decision-making in biologica. International Journal of Learning Technology, 5 (2): 166 - 190.
15. Bybee, R., McCrae, B. and Laurie, R. 2009. PISA 2006: An assessment of scientific literacy. Journal of Research in Science Teaching, 46: 865 - 883.
16. Cen, H., Koedinger, K. R., & Junker, B. (2008). Comparing two IRT models for conjunctive skills. In B. Woolf, E. Aimeur, R. Nkambou, & S. Lajoie (Eds.), Proceedings of the 9th International Conference of Intelligent Tutoring Systems (pp. 796-798). Berlin, Germany: Springer.
17. Cetintas, S., Si, L., Xin, Y. and Hord, C. 2010. Automatic detection of off-task behaviors in intelligent tutoring systems with machine learning techniques. IEEE Transactions on Learning Technologies, 3 (3): 228 - 236.
18. Chen, Z. and Klahr, D. 1999. All other things being equal: Acquisition and transfer of the control of variables strategy. Child Development, 70: 1098 - 1120.
19. Chi, M., VanLehn, K. and Litman, D. 2010. " Do micro-level tutorial decisions matter: Applying reinforcement learning to induce pedagogical tutorial tactics ". In Intelligent tutoring systems: 10th International Conference, ITS 2010, Edited by: Aleven, V., Kay, J. and Mostow, J. 184 - 193. Heidelberg, Germany: Springer.
20. Clarke-Midura, J., Code, J., Zap, N. and Dede, C. 2012. " Assessing science inquiry in the classroom: A case study of the virtual assessment project ". In Cases on inquiry through instructional technology in math and science: Systemic approaches (pp. 138-164) Edited by: Lennex, L. and Nettleton, K. New York, NY: IGI.
21. Clarke-Midura, J., Dede, C. and Norton, J. 2011. The road ahead for state assessments, Cambridge, MA: Rennie Center for Education Research & Policy.
22. Corbett, A., & Anderson, J. (1995). Knowledge-tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 4, 253-278.
23. Corbett, A., Kaufmann, L., MacLaren, B., Wagner, A., & Jones, E. (2010). A cognitive tutor for genetics problem solving: Learning gains and student modeling. Journal of Educational Computing Research, 42, 219-239.
24. DeBoer, G. 1991. A history of ideas in science education: Implications for practice, New York, NY: Teachers College Press.
25. DeBoer, G., Abell, C., Gogos, A., Michiels, A., Regan, T., Wilson, P., Douglas, R. and Stearns, C. 2008. " Assessment linked to science learning goals: Probing student thinking through assessment. Project 2061. American Association for the Advancement of Science ". In Assessing science learning: Perspectives from research and practice, Edited by: Coffey, J. 231 - 252. Arlington, VA: NSTA Press.
26. de Jong, T. and van Joolingen, W. R. 1998. Scientific discovery learning with computer simulations of conceptual domains. Review of Educational Research, 68 (2): 179 - 201.
27. Desmarais, M. C., Meshkinfam, P. and Gagnon, M. 2006. Learned student models with item to item knowledge structures. User Modeling and User-Adapted Interaction, 16: 403 - 434.
28. Dewey, J. 1910. January 28). Science as subject-matter and as method. Science, 31: 121 - 127.
29. Dewey, J. (1916). Democracy and education: An introduction to the philosophy of education. New York, NY: MacMillan.
30. DiCerbo, K. E. and Behrens, J. T. 2012. " Implications of the digital ocean on current and future assessment ". In Computers and their impact on state assessment: Recent history and predictions for the future (pp. 273-306), Edited by: Lissitz, R. Charlotte, NC: Information Age.
31. Duschl, R. A. 2003. " Assessment of inquiry ". In Everyday assessment in the science classroom (pp. 41-59), Edited by: Atkin, J. M. and Coffey, J. E. Arlington, VA: NSTA Press.
32. Efron, B. and Gong, G. 1983. A leisurely look at the bootstrap, the jackknife, and cross-validation. The American Statistician, 37 (1): 36 - 48.
33. Ericsson, K. A. and Simon, H. 1980. Verbal reports as data. Psychological Review, 87: 215 - 251.
34. Fadel, C., Honey, M. and Pasnick, S. 2007. Assessment in the age of innovation. Education Week, 26 (38): 34 - 40.
35. Fan, J., Upadhye, S. and Worster, A. 2006. Understanding receiver operating characteristic (ROC) curves. Canadian Journal of Emergency Medicine, 8 (1): 19 - 20.
36. Ford, M. J. 2012. A dialogic account of sense-making in scientific argumentation and reasoning. Cognition and Instruction, 30 (3): 207 - 245.
37. Fu, A. C., Raizen, S. and Shavelson, R. J. 2009. " December 18). The nation's report card: A vision of large-scale science assessment ". In Science Vol. 326, 1637 - 1638.
38. Gertner, A. S., & VanLehn, K. (2000). Andes: A coached problem solving environment for physics. In G. Gauthier, C. Frasson, & K. VanLehn (Eds.), Proceedings of the 5th International Conference on Intelligent Tutoring Systems, LNCS 1839 (pp. 133-142). Montreal, Canada: Springer-Verlag.
39. Glaser, R., Schauble, L., Raghavan, K. and Zeitz, C. 1992. " Scientific reasoning across different domains ". In Computer-based learning environments and problem-solving, Edited by: DeCorte, E., Linn, M., Mandl, H. and Verschaffel, L. 345 - 371. Heidelberg, Germany: Springer-Verlag.
40. Gobert, J. 2005. " Leveraging technology and cognitive theory on visualization to promote students' science learning and literacy ". In Visualization in science education, Edited by: Gilbert, J. 73 - 90. Dordrecht: The Netherlands: Springer-Verlag.
41. Gobert, J. in press. " Microworlds ". In Encyclopedia of science education, Edited by: Gunstone, R. Heidelberg, Germany: Springer-Verlag.
42. Gobert, J. and Koedinger, K. September 2011. " Presentation at the Society for Research Effectiveness ". In Using model-tracing to conduct performance assessment of students' inquiry skills within a microworld September, Washington, DC.
43. Gobert, J., Sao Pedro, M., Baker, R. S., Toto, E. and Montalvo, O. 2012. Leveraging educational data mining for real time performance assessment of scientific inquiry skills within microworlds. Journal of Educational Data Mining, 4 (1): 153 - 185.
44. Gobert, J., Sao Pedro, M., Toto, E., Montalvo, O., & Baker, R. (2011, April). Science ASSISTments: Assessing and scaffolding students' inquiry skills in real time. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
45. Gobert, J. D., & Buckley, B. C. (2000). Introduction to model-based teaching and learning in science education. International Journal of Science Education, 22, 891-894.
46. Haertel, G., Lash, A., Javitz, H. and Quellmalz, E. April 2006. " Paper presented at the annual meeting of the American Educational Research Association ". In An instructional sensitivity study of science inquiry items from three large-scale science examinations April, San Francisco, CA.
47. Hambleton, R. K., & Jones, R. W. (1993). Comparison of classical test theory and item response theory and their applications to test development. Educational Measurement: Issues and Practice, 12(3), 38-47.
48. Hanley, J. and McNeil, B. 1982. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143: 29 - 36.
49. Harrison, A. M. and Schunn, C. D. 2004. " The transfer of logically general scientific reasoning skills ". In Proceedings of the 26th Annual Conference of the Cognitive Science Society, Edited by: Forbus, K., Gentner, D. and Regier, T. 541 - 546. Mahwah, NJ: Erlbaum.
50. Jacobson, M. J. and Wilensky, U. 2006. Complex systems in education: Scientific and educational importance and implications for the learning sciences. Journal of the Learning Sciences, 15 (1): 11 - 34.
51. Kasurinen, J., & Nikula, U. (2009). Estimating programming knowledge with Bayesian knowledge tracing. In Proceedings of the 14th Annual ACM SIGCSE Conference on Innovation and Technology in Computer Science Education (pp. 313-317). New York, NY: ACM Press.
52. Kent, J. T. 1983. Information gain and a general measure of correlation. Biometrika, 70 (1): 163 - 173.
53. Ketelhut, D. J. and Dede, C. April 2006. " Paper presented at the National Association for Research in Science Teaching ". In Assessing inquiry learning April, San Francisco, CA.
54. Ketelhut, D., Nelson, B., Sil, A. and Yates, A. April/May 2013. " Presentation at the annual meeting of the American Educational Research Association ". In Discovering what students know through data mining their problem-solving actions within the immersive virtual environment, SAVE Science April/May, San Francisco, CA.
55. Klahr, D. and Nigam, M. 2004. The equivalence of learning paths in early science instruction: Effects of direct instruction and discovery learning. Psychological Science, 15: 661 - 667.
56. Koedinger, K., & Corbett, A. (2006). Cognitive tutors: Technology bringing learning sciences to the classroom. In R. Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 61-77). New York, NY: Cambridge University Press.
57. Koedinger, K. R. and MacLaren, B. A. 2002. Developing a pedagogical domain theory of early algebra problem solving, CMU-HCII Tech. Rep. No. 02-100. Pittsburgh, PA: Carnegie Mellon University.
58. Koedinger, K. R., Suthers, D. D. and Forbus, K. D. 1999. Component-based construction of a science learning space. International Journal of Artificial Intelligence in Education, 10: 292 - 313.
59. Koomen, M. April 2006. " Paper presented at the annual meeting of the American Educational Research Association ". In The development and implementation of a computer-based assessment of science literacy in PISA 2006 April, San Francisco, CA.
60. Kuhn, D. 2005a. Education for thinking, Cambridge, MA: Harvard University Press.
61. Kuhn, D. 2005b. What needs to be mastered in mastery of scientific method?. Psychological Science, 16: 873 - 874.
62. Kuhn, D., Black, J., Keselman, A. and Kaplan, D. 2000. The development of cognitive skills to support inquiry learning. Cognition and Instruction, 18: 495 - 523.
63. Kuhn, D. and Pease, M. 2008. What needs to develop in the development of inquiry skills?. Cognition and Instruction, 46: 512 - 559.
64. Kuhn, D., Schauble, L. and Garcia-Mila, M. 1992. Cross-domain development of scientific reasoning. Cognition and Instruction, 9: 285 - 327.
65. Lee, D. M. C., Rodrigo, M. M. T., Baker, R. S. J. d., Sugay, J. O. and Coronel, A. 2011. " Exploring the relationship between novice programmer confusion and achievement ". In Proceedings of the 4th Bi-Annual International Conference on Affective Computing and Intelligent Interaction, Part 1 (pp. 175-184). Berlin, Germany: Springer In S. K. D'Mello, A. Graesser, B. Schuller, & J. Martin (Eds.),
66. Leeman-Munk, S., Wiebe, E., & Lester, J. (2013, April). Mining student science argumentation text to inform an intelligent tutoring system. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
67. Linn, R. L. (1994). Performance assessment: Policy promises and technical measurement standards. Educational Researcher, 23(9), 4-14.
68. Linn, R. L. (2000). Assessments and accountability. Educational Researcher, 29(2), 4-16.
69. Linn, R. L., Baker, E. L., & Dunbar, S. B. (1991). Complex, performance-based assessment: Expectations and validation criteria. Educational Researcher, 20(8), 15-21.
70. Lynch, C., Ashley, K., Pinkwart, N. and Aleven, V. 2009. Concepts, structures, and goals: Redefining ill-definedness. International Journal of Artificial Intelligence in Education, 19 (3): 253 - 266.
71. Massachusetts Department of Education. 2006. " science and technology/engineering curriculum framework ". In Massachusetts Malden, MA Author
72. McElhaney, K. and Linn, M. 2008. " Impacts of students' experimentation using a dynamic visualization on their understanding of motion ". In Proceedings of the 8th International Conference of the Learning Sciences, Vol. 2, 51 - 58. Utrecht, The Netherlands: International Society of the Learning Sciences. In P. A. Kirschner, J. J. G. van Merriënboer, & T. de Jong (Eds.)Vol.pp.
73. McElhaney, K. and Linn, M. 2010. " Helping students make controlled experiments more informative ". In Learning in the disciplines: Proceedings of the 9th International Conference of the Learning Sciences (ICLS 2010): Vol. 1. Full papers, Edited by: Gomez, K., Lyons, L. and Radinsky, J. 786 - 793. Chicago, IL: International Society of the Learning Sciences.
74. Mierswa, I., Wurst, M., Klinkenberg, R., Scholz, M. and Euler, T. 2006. " YALE: Rapid prototyping for complex data mining tasks ". In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 935 - 940. Philadelphia, PA: ACM Press.
75. Mislevy, R. J., Behrens, J. T., DiCerbo, K. and Levy, R. 2012. Design and discovery in educational assessment: Evidence centered design, psychometrics, and data mining. Journal of Educational Data Mining, 4: 11 - 48.
76. Mislevy, R., Hamel, L., Fried, R., G., Gaffney, T., Haertel, G., Hafter, A.,Wenk, A. (2003). Design patterns for assessing science inquiry (PADI Technical Report 1). Menlo Park, CA: SRI International.
77. Montalvo, O., Baker, R. S., Sao Pedro, M. A., Nakama, A. and Gobert, J. D. 2010. " Identifying students' inquiry planning using machine learning ". In Proceedings of the 3rd International Conference on Educational Data Mining 141 - 150. In R. Baker, A. Merceron, & P. Pavlik (Eds.)pp.Pittsburgh, PA: EDM.
78. National Assessment Governing Board. (2011). Science framework for the 2011 National Assessment of Educational Progress. Washington, DC: U.S. Department of Education.
79. National Research Council. 1996. National science education standards, Washington, DC: National Academies Press.
80. National Research Council. 2000. Inquiry and the national science education standards: A guide for teaching and learning, Washington, DC: National Academies Press.
81. National Research Council. 2011. A framework for K-12 science education: Practices, crosscutting concepts, and core ideas, Washington, DC: National Academies Press.
82. Njoo, M. and de Jong, T. 1993. Exploratory learning with a computer simulations for control theory: Learning processes and instructional support. Journal of Research in Science Teaching, 30: 821 - 844.
83. Organisation for Economic Co-operation and Development. (2010). PISA computer-based assessment of student skills in science. Retrieved from http://browse.oecdbookshop.org/oecd/pdfs/free/9810041E.pdf
84. Papert, S. (1980). Computer-based microworlds as incubators for powerful ideas. In R. Taylor (Ed.), The computer in the school: Tutor, tool, tutee (pp. 203-201). New York, NY: Teachers College Press.
85. Pardos, Z., Baker, R., Gowda, S. and Heffernan, N. 2011. The sum is greater than the parts: Ensembling models of student knowledge in educational software. SIGKDD Explorations, 13 (2): 37 - 44.
86. Pavlik, P., Cen, H. and Koedinger, J. 2009. " Performance factors analysis-A new alternative to knowledge tracing ". In Proceedings of the 14th International Conference on Artificial Intelligence in Education, 531 - 540. Brighton, England: IOS Press. In V. Dimitrova, R. Mizoguchi, B. du Boulay, & A. Graesser (Eds.),pp.
87. Pellegrino, J., Chudowsky, N. and Glaser, R. 2001. Knowing what students know: The science and design of educational assessment, Washington, DC: National Academies Press.
88. Quellmalz, E., DeBarger, A., Haertel, G. and Kreikemeier, P. 2005. Validities of science inquiry assessments: Final report, Menlo Park, CA: SRI International.
89. Quellmalz, E. S., DeBarger, A. H., Haertel, G., Schank, P., Buckley, B., Gobert, J. and Ayala, C. 2008. " Exploring the role of technology-based simulations in science assessment: The Calipers Project ". In Science assessment: Research and practical approaches, Washington, DC: NSTA.In
90. Quellmalz, E. and Haertel, G. 2004. Technology supports for state science assessment systems, Washington, DC: National Research Council.
91. Quellmalz, E., Kreikemeier, P., DeBarger, A. H. and Haertel, G. April 2007. " Paper presented at the annual meeting of the American Educational Research Association ". In A study of the alignment of the NAEP, TIMSS, and new standards science assessments with the inquiry abilities in the national science education standards April, Chicago, IL.
92. Quellmalz, E. S. and Pellegrino, J. W. 2009. January 2). Technology and testing. Science, 323: 75 - 79.
93. Quellmalz, E. S., Silberglitt, M. D., & Timms, M. J. (2011). How can simulations be components of balanced state science assessment systems (Policy Brief). San Francisco, CA: WestEd. Retrieved from http://www.wested.org/cs/we/view/rs/1198
94. Quellmalz, E. S., Timms, M. J., Buckley, B. C., Davenport, J., Loveland, M. and Silberglitt, M. D. 2012. " 21st century dynamic assessment ". In Technology-based assessments for 21st century skills: Theoretical and practical implications from modern research, Edited by: Mayrath, M., Clarke-Midura, J. and Robinson, D. H. Charlotte, NC: Information Age.
95. Quellmalz, E., Timms, M. and Schneider, S. 2009. Assessment of student learning in science simulations and games, Washington, DC: National Research Council.
96. Quellmalz, E. S., Timms, M. J., Silberglitt, M. D. and Buckley, B. C. 2012. Science assessments for all: Integrating science simulations into balanced state science assessment systems. Journal of Research in Science Teaching, 49: 363 - 393.
97. Quinlan, J. R. 1993. C4.5: Programs for machine learning, San Francisco, CA: Morgan Kaufmann.
98. Razzaq, L., Feng, M., Nuzzo-Jones, G., Heffernan, N., Koedinger, K. R., Junker, B., Rasmussen, K. (2005). The Assistment Project: Blending assessment and assisting. In G. M. C. K. Looi (Ed.), Proceedings of the 12th International Conference on Artificial Intelligence in Education (pp. 555-562). Amsterdam, The Netherlands: IOS Press.
99. Reimann, P. 1991. Detecting functional relations in a computerized discovery environment. Learning and Instruction, 1 (1): 45 - 65.
100. Romero, C. and Ventura, S. 2010. Educational data mining: A review of the state-of-the-art. IEEE Transactions on Systems, Man, and Cybernetics-Part C: Applications and Reviews, 40: 601 - 618.
101. Rowe, J., & Lester, J. (2010). Modeling user knowledge with dynamic Bayesian networks in interactive narrative environments. In C. G. Youngblood & V. Bulitko (Eds.), Proceedings of the 6th Annual AI and Interactive Digital Entertainment Conference (pp. 57-62). Palo Alto, CA: AAAI Press.
102. Rupp, A. A., Gushta, M., Mislevy, R. J.Shaffer, D. W.2010Evidence-centered design of epistemic games: Measurement principles for complex learning environments. Journal of Technology, Learning, and Assessment, 8(4). Retrieved from http://escholarship.bc.edu/jtla/vol8/4
103. Rutherford, F. J., & Ahlgren, A. (1989). Science for all Americans: A Project 2061 report on literacy goals in science, mathematics, and technology. Washington, DC: American Association for the Advancement of Science.
104. Sao Pedro, M. A. 2013. Real-time assessment, prediction, and scaffolding of middle school students' data collection skills within physical science simulations, Worcester: MA. Unpublished doctoral dissertationWorcester Polytechnic Institute
105. Sao Pedro, M., Baker, R., & Gobert, J. (2013a). Incorporating scaffolding and tutor context into Bayesian knowledge tracing to predict inquiry skill acquisition. In S. K. D'Mello, R. A. Calvo, & A. Olney (Eds.), Proceedings of the 6th International Conference on Educational Data Mining (pp. 185-192). Memphis, TN.
106. Sao Pedro, M., Baker, R. and Gobert, J. 2013b. " What different kinds of stratification can reveal about the generalizability of data-mined skill assessment models ". In Proceedings of the 3rd Conference on Learning Analytics and Knowledge (pp. 190-194) Leuven, Belgium: ACM Press.
107. Sao Pedro, M., Baker, R., Gobert, J., Montalvo, O. and Nakama, A. 2013. Leveraging machine-learned detectors of systematic inquiry behavior to estimate and predict transfer of inquiry skill. User Modeling and User-Adapted Interaction, 23: 1 - 39.
108. Sao Pedro, M. A., Baker, R. S., Montalvo, O., Nakama, A. and Gobert, J. D. 2010. " (pp. 181-190) ". In Proceedings of the 3rd International Conference on Educational Data Mining Edited by: Baker, R., Merceron, A. and Pavlik, P. Pittsburgh, PA. Using text replay tagging to produce detectors of systematic experimentation behavior patterns
109. Sao Pedro, M., Gobert, J. and Baker, R. April 2012. " Paper presented at the annual meeting of the American Educational Research Association ". In Assessing the learning and transfer of data collection inquiry skills using educational data mining on students' log files April, Vancouver, British Columbia, Canada.
110. Sao Pedro, M., Gobert, J., Heffernan, N. and Beck, J. 2009. " Comparing pedagogical approaches for teaching the control of variables strategy ". In Proceedings of the 31st Annual Meeting of the Cognitive Science Society, Eds, Edited by: Taatgen, N. A. and vanRijn, H. 1294 - 1299. Austin, TX: Cognitive Science Society.
111. Sao Pedro, M. A., Gobert, J. D. and Raziuddin, J. 2010. " Comparing pedagogical approaches for the acquisition and long-term robustness of the control of variables strategy ". In Learning in the disciplines: Proceedings of the 9th International Conference of the Learning Sciences, ICLS 2010: Vol. 1. Full papers, Edited by: Gomez, K., Lyons, L. and Radinsky, J. 1024 - 1031. Chicago, IL: International Society of the Learning Sciences.
112. Scalise, K., Timms, M., Clark, L. and Moorjani, A. April 2009. Student learning in science simulations: What makes a difference?, April, San Diego: CA. Paper presented at the annual meeting of the American Educational Research Association
113. Schauble, L., Glaser, R., Duschl, R., Schulze, S. and John, J. 1995. Students' understanding of the objectives and procedures of experimentation in the science classroom. Journal of the Learning Sciences, 4 (2): 131 - 166.
114. Schauble, L., Glaser, R., Raghavan, K. and Reiner, M. 1991. Causal models and experimentation strategies in scientific reasoning. Journal of the Learning Sciences, 1 (2): 201 - 238.
115. Schauble, L., Klopfer, L. E. and Raghavan, K. 1991. Students' transition from an engineering model to a science model of experimentation. Journal of Research in Science Teaching, 28: 859 - 882.
116. Schunn, C. D. and Anderson, J. R. 1998. " In J. R. Anderson, The atomic components of thought ". In Scientific discovery, 385 - 428. Mahwah, NJ: Erlbaum.
117. Schunn, C. D. and Anderson, J. R. 1999. The generality/specificity of expertise in scientific reasoning. Cognitive Science, 23 (3): 337 - 370.
118. Shavelson, R., Wiley, E. W. and Ruiz-Primo, M. 1999. Note on sources of sampling variability in science performance assessments. Journal of Educational Measurement, 36 (1): 61 - 71.
119. Shute, V. and Glaser, R. 1990. A large-scale evaluation of an intelligent discovery world: Smithtown. Interactive Learning Environments, 1: 55 - 71.
120. Siler, S., Klahr, D., Magaro, C., Willows, K. and Mowery, D. 2010. " Predictors of transfer of experimental design skills in elementary and middle school children ". In Proceedings of the 10th International Conference on Intelligent Tutoring Systems, Part II, LNCS 6095 Edited by: Aleven, V., Kay, J. and Mostow, J. 198 - 208. Pittsburgh, PA: Springer
121. Slotta, J. D., & Linn, M. C. (2009). WISE science: Web-based inquiry in the classroom. New York, NY: Teachers College Press.
122. Stecher, B. M., & Klein, S. P. (1997). The cost of science performance assessments in large-scale testing programs. Educational Evaluation and Policy Analysis, 19(1), 1-14.
123. Strand-Cary, M. and Klahr, D. 2008. Developing elementary science skills: Instructional effectiveness and path independence. Cognitive Development, 23: 488 - 511.
124. Timms, M., Clements, D. H., Gobert, J., Ketelhut, D. J., Lester, J., Reese, D. D.Wiebe, E.2012New measurement paradigms. Retrieved from http://works.bepress.com/michael_timms/36 (http://works.bepress.com/michael_timms/36)
125. Tsirgi, J. 1980. Sensible reasoning: A hypothesis about hypotheses. Child Development, 51: 1 - 10.
126. van Joolingen, W. and de Jong, T. 1991. Supporting hypothesis generation by learners exploring an interactive computer simulation. Instructional Science, 20: 389 - 404.
127. van Joolingen, W., & de Jong, T. (1993). Exploring a domain through a computer simulation: Traversing variable and relation space with the help of a hypothesis scratchpad. In D. Towne, T. de Jong, & H. Spada (Eds.), Simulation-based experiential learning (pp. 191-206). Berlin, Germany: Springer-Verlag.
128. Vapnik V. (1995). The nature of statistical learning theory. New York, NY: Springer-Verlag.
129. Voss, J. F. 2006. " Toulmin's model and the solving of ill-structured problems ". In Arguing on the Toulmin model: New essays in argument analysis and evaluation, Edited by: Hitchcock, D. and Verheij, B. 303 - 311. Berlin, Germany: Springer.
130. Walonoski, J. and Heffernan, N. 2006. " Detection and analysis of off-task gaming behavior in intelligent tutoring systems ". In Proceedings of the 8th International Conference on Intelligent Tutoring Systems, ITS 2006. LNCS 4053, Edited by: Ikeda, M., Ashlay, K. and Chan, T.-W. 382 - 391. Johngli, Taiwan: Springer-Verlag.
131. Williamson, D., Mislevy, R. and Bejar, I. 2006. Automated scoring of complex tasks in computer-based testing, Mahwah, NJ: Erlbaum.
132. Wilson, M., & Bertenthal, M. (Eds.). (2006). Systems for state science assessment. Washington, DC: National Academies Press.
133. Witten, I. H. and Frank, E. 2005. Data mining: Practical machine learning tools and techniques with Java implementations, San Francisco, CA: Morgan Kaufmann.
134. Yanamandra, K., Grudin, M. A., Casaite, V., Meskys, R., Forsgren, L. and Morozova-Roche, L. 2011. a-Synuclein reactive antibodies as diagnostic markers in blood sera of Parkinson's disease patients. PlosONE, 6(4), e18513. doi:10.1371/journal.pone.0018513
135. Yoon, S. 2008. Using memes and memetic processes to explain social and conceptual influences on student understanding about complex socio-scientific issues. Journal of Research in Science Teaching, 45: 900 - 921.
136. Zimmerman, C. (2007). The development of scientific thinking skills in elementary and middle school. Developmental Review, 27, 172-223.
137. Zohar, A. and David, B. A. 2008. Explicit teaching of meta-strategic knowledge in authentic classroom situations. Metacognition and Learning, 3 (1): 59 - 82.