Using multilevel analyses to study the effectiveness of science curriculum materials

Evaluation and Research in Education

Volumn 21, Issue 1, 2008, Pages 18-42

  Rethinam, Vasuki ^a   Pyke, Curtis ^a,b   Lynch, Sharon ^a,b  

^a Montgomery County Public Schools   (United States)

^b GEORGE WASHINGTON UNIVERSITY   (United States)

Author keywords

Classroom context; Curriculum effectiveness; Gain scores; HLM; Multi level versus single level analysis; Science education

Indexed keywords

EID: 84856716518     PISSN: 09500790     EISSN: 17477514     Source Type: Journal    
DOI: 10.2167/eri418.0     Document Type: Article

References (44)

1. Aitkin, M., and Longford, N., 1986. Statistical modeling issues in school effectiveness studies. Journal of Royal Statistical Society A, 149: 1–43.
2. Aitkin, M., Anderson, D., and Hinde, J., 1981. Statistical modeling of data on teaching styles. Journal of the Royal Statistical Society A, 144 (4): 419–461.
3. American Association for the Advancement of Science. 1993. Benchmarks for Science Literacy, New York: Oxford University Press.
4. American Association for the Advancement of Science. 2001. AAAS Science Textbooks Conference [Computer software], Washington, DC: AAAS.
5. American Psychological Association. 2001. Publication Manual of the American Psychological Association, 5th edn, Washington, DC: American Psychological Association.
6. Anderman, E.M., Yoon, K.S., Roeser, R.W., and Blumenfeld, P., (1995) Learning to value mathematics: Individual differences and classroom effects. Paper presented at the annual meeting of the Society for Research in Child Development, March.
7. Bembry, K.L., and Schumacker, R.E., (2003) Investigating classroom effectiveness measure over time. Paper presented at the Annual meeting of the American Educational Research Association, San Diego, CA, April.
8. Cohen, J., 1988. Statistical Power Analysis for the Behavioral Sciences, 2nd edn, Hillsdale, NJ: Lawrence Erlbaum Associates.
9. Cook, T.D., 2002. Randomized experiments in educational policy research: A critical examination of the reasons the educational evaluation community has offered for not doing them. Educational Evaluation and Policy Analysis, 24 (3): 175–199.
10. Cronbach, L.J., and Furby, L., 1970. We should measure ‘change’–or should we?. Psychological Bulletin, 74 (1): 68–80.
11. Goldstein, H., 1997. Methods in school effectiveness research. School Effectiveness and School Improvement, 8: 369–395.
12. Harvard Smithsonian Center for Astrophysics. 2001. ARIES-Exploring Motion and Forces: Speed, Acceleration, and Friction, Watertown, MA: Charlesbridge Publishing.
13. Kirk, R.E., 1982. Experimental Design: Procedures for the Behavioral Sciences, 2nd edn, Belmont, CA: Brooks/Cole.
14. Kreft, I.G.G., (1996) Are multilevel techniques necessary? An overview, including simulation studies. Unpublished manuscript, California State University, Los Angeles, CA
15. Lawrence Hall of Science. 2000. Great Explorations in Math and Science (GEMS): The Real Reasons for Seasons: Sun-Earth Connections, Berkeley, CA: The Regents of the University of California at Berkeley.
16. Lederman, N.G., and Flick, L.B., 2005. Beware of the unit of analysis: It may be you!!. School Science and Mathematics, 105 (8): 381–383.
17. Lee, V.E., 2000. Using hierarchical linear modeling to study social contexts: The case of school effects. Educational Psychologist, 35 (2): 125–141.
18. Lee, V.E., and Loeb, S., 2000. School size in Chicago elementary schools: Effects on teachers’ attitudes and students’ achievement. American Educational Research Journal, 37 (1): 3–31.
19. Lee, V.E., Loeb, S., and Lubeck, S., 1998. Contextual effects of prekindergarten classrooms for disadvantaged children on cognitive development: The case of chapter 1. Child Development, 69 (2): 479–494.
20. Lynch, S., Kuipers, J., Pyke, C., and Szesze, M., 2005. Examining the effects of a highly rated science curriculum unit on diverse populations: Results from a planning grant. Journal of Research in Science Teaching, 42 (8): 912–946.
21. Lynch, S., Taymans, J., Watson, W., Ochsendorf, R., Pyke, C., and Szesze, M., 2007. Effectiveness of a highly-rated science curriculum unit for students with disabilities in general education classrooms. Exceptional Children, 73 (2): 202–223.
22. Maris, E., 1998. Covariance adjustment versus gain scores–revisited. Psychological Methods, 3 (3): 309–327.
23. Marks, H.M., 2000. Student engagement in instructional activity: Patterns in the elementary, middle, and high school years. American Educational Research Journal, 37 (1): 153–184.
24. McDonald, S-K., Keesler, V.A., Kauffman, N.J., and Schneider, B., 2006. Scaling-up exemplary interventions. Educational Researcher, 35 (3): 15–22.
25. Nye, B., Hedges, L.V., and Konstantopoulos, S., 1999. The long-term effects of small classes: A five-year follow-up of the Tennessee class size experiment. Educational Evaluation and Policy Analysis, 21 (2): 127–142.
26. Nye, B., Hedges, L.V., and Konstantopoulos, S., 2002. Do low-achieving students benefit more form small classes? Evidence from the Tennessee class size experiment. Educational Evaluation and Policy Analysis, 24 (3): 201–217.
27. Nye, B., Hedges, L.V., and Konstantopoulos, S., 2004. Do minorities experience larger benefits from small classes?. The Journal of Educational Research, 98 (2): 94–100.
28. Ochsendorf, R., Lynch, S., and Pyke, C., (2006) Evaluating a science curriculum unit: Learning through the process. Manuscript in preparation.
29. Odden, A., Borman, G., and Fermanich, M., 2004. Assessing teacher, classroom, and school effects, including fiscal effects. Peabody Journal of Education, 79 (4): 4–32.
30. O'Donnell, C., Watson, W., Pyke, C., and Lynch, S., (2006) Using the Project 2061 Curriculum Analysis to understand the results of a quasi-experimental curriculum unit evaluation of Seasons. Paper presented at the Annual Meeting of the National Association for Research in Science Teaching, San Francisco, CA, April.
31. Pyke, C., Lynch, S., Kuipers, J., Szesze, M., and Watson, W., 2004. Implementation Study of The Real Reasons for Seasons (2003–2004): SCALE-uP Report No. 4, Washington, DC: George Washington University.
32. Pyke, C., Lynch, S., Kuipers, J., Szesze, M., and Watson, W., 2004. Implementation Study of Exploring Motion and Forces (2003–2004): SCALE-uP Report No. 5, Washington, DC: George Washington University, SCALE-uP.
33. Pyke, C., and Ochsendorf, R., (2006) Concept assessment in curriculum unit evaluation. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA, April.
34. Raudenbush, S.W., and Bryk, A.S., 1988. Methodological advances in studying effects of classrooms and schools on student learning. Review of Research in Education, 15: 423–476.
35. Raudenbush, S.W., and Bryk, A.S., 2002. Hierarchical Linear Models: Applications and Data Analysis Methods, 2nd edn, Thousand Oaks, CA: Sage Publications.
36. Raudenbush, S.W., Bryk, A.S., Cheong, Y.F., and Congdon, R.T., 2004. HLM: Hierarchical Linear and Nonlinear Modeling, Chicago: Scientific Software International.
37. Rogosa, D.R., and Willett, J.B., 1983. Demonstrating the reliability of the difference score in the measurement of change. Journal of Educational Measurement, 20 (4): 335–343.
38. Rosenthal, R., and Rosnow, R.L., 1984. Essentials of Behavioral Research: Methods and Data Analysis, New York: McGraw Hill.
39. Rowe, K.J., and Hill, P.W., 1998. Modeling educational effectiveness in classrooms: The use of multi-level structural equations to model student's progress. Educational Research and Evaluation, 4 (4): 307–347.
40. Rowe, K.J., Hill, P.W., and Holmes-Smith, P., 1995. Methodological issues in educational performance and school effectiveness research: A discussion with worked examples. Australian Journal of Education, 39: 217–248.
41. State of Michigan (1993) Chemistry That Applies. Lansing, MI: State of Michigan.
42. Subedi, B.R., 2003. Factors influencing high school student achievement in Nepal. International Education Journal, 4 (2): 98–107.
43. Winer, B.J., 1971. Statistical Principles in Experimental Design, New York: McGraw Hill.
44. Wright, D.B., (2005) Comparing groups in a before-after design: When t test and ANCOVA produce different results. British Journal of Educational Psychology 1–13. http://www.sussex.ac.uk/Users/danw/pdf/draftlord.pdf. Accessed 05.03.2005.