Grades of evidence: Variability in quality of findings in effectiveness studies of complex field interventions

American Journal of Evaluation

Volumn 28, Issue 3, 2007, Pages 239-255

  Chatterji, Madhabi ^a  

^a Howard Hughes Medical Institute   (United States)

Author keywords

Evidence standards; Evidence based practices; Mixed method designs; Randomized experiments

Indexed keywords

EID: 34548180586     PISSN: 10982140     EISSN: 15570878     Source Type: Journal    
DOI: 10.1177/1098214007304884     Document Type: Review

References (32)

1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: Author.
2. Berliner, D.C. (2002). Educational research: The hardest science of all. Educational Researcher, 31 (8), 18-20.
3. Beveridge, W.I.B. (1957). The art of scientific investigation. New York: Norton.
4. Boissel, J. (1989). The impact of randomized clinical trials on medical practices. Controlled Clinical Trials, 10 (4), 120-134.
5. Campbell, D.T. (1981). Introduction: Getting ready for the experimenting society. In L. Saxe & M. Fine (Eds.), Social experiments: Methods for design and evaluation (pp. 13-18). Beverly Hills, CA: Sage.
6. Campbell, D.T., & Stanley, J. (1963). Experimental and quasi-experimental designs for research. Chicago: Rand McNally.
7. Chatterji, M. (2004). Evidence of "what works": An argument for extended-term mixed method (ETMM) evaluation designs. Educational Researcher, 33 (9), 1-13. (Reprinted: Chatterji, M. (2005). Evidence of "what works": An argument for extended-term mixed method (ETMM) evaluation designs. Educational Researcher, 34 (6), 13-24.
8. Chatterji, M., Kwon, Y.A., & Sng, C. (2006). Gathering evidence on an after-school supplemental instruction program: Design challenges, lessons, and early findings in light of NCLB. Educational Policy Analysis Archives, 14 (12), Retrieved on September 6, 2006, from http://epaa.asu.edu/epaa/v14n12/.
9. Cronbach, L.J., & Associates. (1980). Toward reform in program evaluation. San Francisco: Jossey-Bass.
10. Davis, C.E. (1994). Generalizing from clinical trials. Controlled Clinical Trials, 15 (1), 11-14.
11. Education Sciences Reform Act of 2002, H. R. 3801, 107th Cong. (2002).
12. Feuer, M.J., Towne, L., & Shavelson, R.J. (2002). Scientific culture and educational research. Educational Researcher, 31 (8), 4-14.
13. Fisher, R.A. (1960). The design of experiments. New York: Hafner Publishing.
14. Glass, G.V., & Hopkins, K.D. (1984). Statistical methods in education and psychology. Englewood Cliffs, NJ: Prentice Hall.
15. Hanley, J.A., & Lippman-Hand, A. (1983). If nothing goes wrong is everything all right? Interpreting zero numerators. Journal of the American Medical Association, 249, 1743-1745.
16. Holmes, C.T. (1989). Grade-level retention effects: A meta-analysis of research studies. In L. A. Shephard & M. L. Smith (Eds.), Flunking grades: Research and policies on retention (pp. 16-33). London: Falmer.
17. Klimt, C.R. (1981). The conduct and principles of randomized clinical trials. Controlled Clinical Trials, 1 (4), 283-293.
18. Hsieh, P., Levin, J.R., Acee, T., Chung, W., Hsieh, Y., Kim, H., et al. (2005). Is educational intervention research on the decline? Journal of Educational Psychology, 97 (4), 523-529.
19. Levin, J.R., & O'Donnell, A.M. (1999). What to do about educational psychology's credibility gaps? Issues in Education: Contributions from Educational Psychology, 5, 177-229.
20. Mathematica Policy Research, Inc., & Decision Information Resources, Inc. (2003). When schools stay open late: The national evaluation of the 21st century community learning centers program. Jessup, MD: U.S. Department of Education, ED Pubs.
21. Meinert, C.L. (1980). Toward more definitive clinical trials. Controlled Clinical Trials, 1 (3), 249-262.
22. No Child Left Behind Act of 2001, P.L. 107-110, 115 Stat. 1425 (2002).
23. Raudenbush, S. (2002, February 6). Scientifically-based research. Paper presented at the U.S. Department of Education "Use of Scientifically Based Research in Education" Working Group Conference, Washington, DC.
24. Reynolds, A.J. (2005). Confirmatory program evaluation: Applications to early childhood interventions. Teachers College Record, 107 (10), 2401-2425.
25. Saloman, G. (1991). Transcending the qualitative-quantitative debate: The analytic and systemic approaches to educational research. Educational Researcher, 20 (6), 10-18.
26. Schoenfield, A.H. (2006). What doesn't work: The challenge and failure of the what works clearinghouse to conduct meaningful reviews of studies of mathematics curricula. Educational Researcher, 35 (2), 13-21.
27. Sheiner, L.B. (1997). Learning versus confirming in clinical drug development. Clinical Pharmacology and Therapeutics, 61, 275-291.
28. Shepard, L.A., & Smith, M.L. (1989). Synthesis of research on grade retention. Educational Researcher, 47 (8), 84-88.
29. Urquhart, J. (2005). Some key points emerging from the COX-2 controversy. Pharmacoepidemiology and Drug Safety, 14 (3), 145-147.
30. U.S. Department of Education. (2003). Identifying and implementing educational practices supported by rigorous evidence: A user-friendly guide. Washington, DC: Institute for Education Sciences.
31. Wolff, N. (2000). Using randomized controlled trials to evaluate socially-complex services: Problems, challenges, and recommendations. Journal of Mental Health Policy and Economics, 3, 97-109.
32. Yusuf, S., Hawken S., Ounpuu, X., Duns, T., Avezum, A., Lanas, F., et al. (2004). Effect of potentially modifiable risk factors associated with myocardial infarction in 52 countries (the INTERHEART study): Case-control study. Lancet, 364, 937-952.