Conducting causal effects studies in science education: Considering methodological trade-offs in the context of policies affecting research in schools

Journal of Research in Science Teaching

Volumn 50, Issue 9, 2013, Pages 1127-1141

  Taylor, Joseph ^a   Kowalski, Susan ^a   Wilson, Christopher ^a   Getty, Stephen ^a   Carlson, Janet ^a  

^a Biological Sciences Curriculum Study Colorado Springs   (United States)

Author keywords

accountability; policy; program evaluation; validity reliability

Indexed keywords

ACCOUNTABILITY; FEDERAL FUNDINGS; MEASUREMENT SENSITIVITY; PROGRAM EVALUATION; RANDOMIZED TRIAL; SCIENCE EDUCATION; STATISTICAL POWER; VALIDITY/RELIABILITY;

COMMERCE; PUBLIC POLICY; RESEARCH; SOCIETIES AND INSTITUTIONS;

ECONOMIC AND SOCIAL EFFECTS;

EID: 84885961145     PISSN: 00224308     EISSN: 10982736     Source Type: Journal    
DOI: 10.1002/tea.21110     Document Type: Article

References (40)

1. American Association for the Advancement of Science (AAAS). (1993). Benchmarks for science literacy. New York, NY: Oxford University Press.
2. Attali, Y., Powers, D., Freedman, M., Harrison, M., &, Obetz, S., (2008). Automated scoring of short-answer open-ended GRE subject test items, GRE Board Research Report No. 04-02 (ETS RR-08-20).
3. Bill and Melinda Gates Foundation. (2012). Primary sources: 2012 America's teachers on the teaching profession. Retrieved from http://www.scholastic.com/primarysources/pdfs/Gates2012-full.pdf
4. Boruch, R. F., DeMoya, D., &, Snyder, B., (2002). The importance of randomized field trials in education and related areas. In, F. Mosteller, &, R. Boruch, (Eds.), Evidence matters: Randomized field trials in education research (pp. 50-79). Washington, DC: Brookings Institution Press.
5. Bracey, G. W., (2005). No child left behind: Where does the money go? Education Policy Research Unit, Arizona State University. Retrieved from http://nepc.colorado.edu/files/EPSL-0506-114-EPRU-exec.pdf
6. Feuer, M. J., Towne, L., &, Shavelson, R. J., (2002). Scientific culture and educational research. Educational Researcher, 31 (8), 4-14.
7. Getty, S. R., Wilson, C. D., Taylor, J. A., &, Kowalski, S. M., (2011, April). Managing threats to validity in experimental tests of education interventions: Data and evidence from a large, cluster-randomized trial (CRT) of a high school science intervention. Paper presented at the National Association of Research in Science Teaching (NARST), Orlando, FL.
8. Hill, H. C., Charalambous, C. Y., &, Kraft, M. A., (2012). When rater reliability is not enough: Teacher observation systems and a case for the generalizability study. Educational Researcher, 41 (2), 56-64.
9. Institute for Education Sciences (IES). (2011a). What Works Clearinghouse procedures and standards handbook (v. 2.1). Retrieved from http://www.ies.ed. gov/ncee/wwc/documentsum.aspx?sid=19
10. Institute for Education Sciences (IES). (2011b). Education research grants request for application (CFDA Number: 84.305A). Retrieved from http://ies.ed.gov/funding/pdf/2012-84305A.pdf
11. Institute for Education Sciences (IES). (2013). Summer research training institute: Cluster-randomized trials. Retrieved from http://www.ies.ed.gov/ whatsnew/conferences/?id=1049
12. Johns Hopkins University Center for Data-Driven Reform in Education (CDDRE). (2011). Best evidence encyclopedia. Retrieved from www.bestevidence.org
13. Lee, O., &, Krajcik, J., (2012). Large-scale interventions in science education for diverse student groups in varied educational settings. Journal of Research in Science Teaching, 49 (3), 271-280.
14. Marx, R. W., (2012). Large-scale interventions in science education: The road to utopia ? Journal of Research in Science Teaching, 49 (3), 420-427.
15. McDonald, S.-K., Keesler, V. A., Kauffman, N. J., &, Schneider, B., (2006). Scaling-up exemplary interventions. Educational Researcher, 35 (3), 15-24.
16. National Research Council (NRC). (1996). National science education standards. Washington, DC: National Academy Press.
17. National Research Council (NRC). (2000). Inquiry and the national science education standards. Washington, DC: National Academy Press.
18. National Research Council (NRC). (2002). Scientific research in education. Committee on Scientific Principles for Education Research., R. J. Shavelson, &, L. Towne, (Eds.), Center for Education. Division of Behavioral and Social Sciences and Education. Washington, DC: National Academy Press.
19. National Research Council (NRC). (2012). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. Committee on a Conceptual Framework for New K-12 Science Education Standards. Board on Science Education, Division of Behavioral and Social Sciences. Washington, DC: National Academy Press.
20. National Science Foundation. (2011). Discovery research K-12 (DRK-12) program solicitation (Publication No. 11-588).
21. Nehm, R. H., Ha, M., &, Mayfield, E., (2012). Transforming biology assessment with machine learning: Automated scoring of written evolutionary explanations. Journal of Science Education and Technology, 21 (1), 193-196.
22. Nehm, R. H., &, Haertig, H., (2012). Human vs. computer diagnosis of students' natural selection knowledge: Testing the efficacy of text analytic software. Journal of Science Education and Technology, 21 (1), 56-73.
23. NGSS Lead States. (2013). Next Generation Science Standards: For states, by states. Washington, DC: The National Academies Press.
24. Pellegrino, J., Chudowsky, N., &, Glaser, R., (2001). Knowing what students know: The science and design of educational assessment. Washington, DC: National Academy Press.
25. Penuel, W. R., &, Fishman, B. J., (2012). Large-scale science education intervention research we can use. Journal of Research in Science Teaching, 49 (3), 281-304.
26. Raudenbush, S. W., &, Bryk, A. S., (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage Publications, Inc.
27. Ruiz-Primo, M. A., Li, M., Wills, K., Giamellaro, M., Lan, M.-C., Mason, H., &, Sands, D., (2012). Developing and evaluating instructionally sensitive assessments in science. Journal of Research in Science Teaching, 49 (6), 691-712.
28. Ruiz-Primo, M. A., Shavelson, R. J., Hamilton, L., &, Klein, S. P., (2002). On the evaluation of systematic science education reform: Search for instructional sensitivity. Journal of Research in Science Teaching, 39 (5), 369-393.
29. Sao Pedro, M. A., Gobert, J. D., Heffernan, N. T., &, Beck, J. E., (2009). Comparing pedagogical approaches for teaching the control of variables strategy. In N. A. Taatgen & H. van Rijn (Eds.), Proceedings of the 31st Annual Conference of the Cognitive Science Society. Austin, TX: Cognitive Science Society.
30. Schneider, B. H., Carnoy, M., Kilpatrick, J., Schmidt, W. H., &, Shavelson, R. J., (2007). Estimating causal effects using experimental and observational designs. Washington, DC: American Educational Research Association.
31. Schwandt, T. A., (2005). A diagnostic reading of scientifically based research for education. Educational Theory, 55 (3), 285-305.
32. Shadish, W. R., Cook, T. D., &, Campbell, D. T., (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin Company.
33. Slavin, R. E., (2008). What works? Issues in synthesizing educational program evaluations. Educational Researcher, 37 (1), 5-14.
34. Spybrook, J., Bloom, H., Congdon, R., Hill, C., Martinez, A., &, Raudenbush, S., (2011). Optimal design for longitudinal and multilevel research: Documentation for the Optimal Design Software Version 3.0. Retrieved from www.wtgrantfoundation.org.
35. The White House, Office of the Press Secretary. (2012). President Obama: Our children can't wait for congress to fix no child left behind, announces flexibility in exchange for reform for ten states [Press release]. Retrieved from http://www.whitehouse.gov/the-press-office2012/02/09/no-child-left-behind- announces-flexibility-exchange-reform-ten-states
36. U.S. Congress. (2001). No Child Left Behind Act of 2001 (Public Law 107-110). Washington, DC: Government Printing Office.
37. U.S. Congress. (2002). Education Sciences Reform Act of 2002. Washington, DC: Government Printing Office. Retrieved from http://www2.ed.gov/policy/ rschstat/leg/PL107-279.pdf
38. Wilson, C. D., Roth, K. J., Taylor, J. A., Landes, N., &, Stuhlsatz, M., (2012, March). In search of instructional sensitivity: The measurement problem in large scale studies of professional development programs. Paper presented at the annual conference of the National Association of Research in Science Teaching (NARST), Indianapolis, IN.
39. Wilson, C. D., Taylor, J. A., Kowalski, S. M., &, Carlson, J., (2010). The relative effects and equity of inquiry-based and commonplace science teaching on students' knowledge. Journal of Research in Science Teaching, 47 (3), 276-301.
40. Zellmer, M. B., Frontier, A., &, Pheifer, D., (2006). NCLB: Taking stock, looking forward. Educational Leadership, 64 (3), 43-46.