A Diagnostic reading of scientifically based research for education

Educational Theory

Volumn 55, Issue 3, 2005, Pages 285-305

  Schwandt, Thomas A ^a  

^a UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 33745676743     PISSN: 00132004     EISSN: 17415446     Source Type: Journal    
DOI: None     Document Type: Article

References (76)

1. Edward L. Thorndike, "The Contribution of Psychology to Education," Journal of Educational Psychology 1, no. 1 (1910): 8.
2. Of the twenty-nine people comprising the combined membership of the technical advisory group for the What Works Clearinghouse (see http://wwc.org/whoweare/memberlist.html) and the board of advisors of the Council for Excellence in Government's Coalition for Evidence-Based Policy (see http://www.excelgov.org/displayContent.asp?Keyword1/4prppcAdvisory), sixteen are from the disciplines of psychology and economics, or the fields of measurement and quantitative methods; one is from medicine;
3. one is from criminology;
4. four are senior fellows at Mathematica, the Brookings Institution, the Hoover Institute, and the Progressive Policy Institute;
5. and two are from Manpower Research Demonstration Corporation and Westat, Inc.
6. Jamie Chamberlin, "Federal Institute Will Emphasize Science in Education," APA Monitor on Psychology 34, no. 2 (2003): 20, http://www.apa.org/monitor/feb03/federal.html.
7. Bridget Murray, "Wanted: Politics-Free, Science-Based Education," APA Monitor on Psychology 33, no. 8 (2002): 52-53, http://www.apa.org/monitor/sep02/wanted.html.
8. Ibid.
9. See, for example, Thomas D. Cook, "Randomized Experiments in Educational Policy Research: A Critical Examination of the Reasons the Educational Evaluation Community Has Offered for Not Doing Them," Educational Evaluation and Policy Analysis 24, no. 3 (2002), 175-199;
10. Thomas D. Cook, "Why Have Educational Evaluators Chosen Not to Do Randomized Experiments?" Annals of the American Academy of Political and Social Science 589 (September 2003): 114-149;
11. Frederick Mosteller and Robert Boruch, eds., Evidence Matters: Randomized Trials in Education Research (Washington, D.C.: Brookings Institution Press, 2002);
12. and the exchange on the nature and merits of meta-analysis between Mark Lipsey and Thomas Schwandt in the American Journal of Evaluation 21, no. 2 (2000): 207-229.
13. William R. Shadish, Thomas D. Cook, and Donald T. Campbell, Experimental and Quasi- Experimental Designs for Generalized Causal Inference (Boston: Houghton Mifflin, 2002), quoted in Jeffrey C. Valentine and Harris Cooper, What Works Clearinghouse Study Design and Implementation Assessment Device, version 1.1 (Washington, D.C.: U.S. Department of Education, 2004), 1. Some of the criticisms of the WWC's restrictive view of legitimate science-based methods are summarized in Margaret Eisenhart and Lisa Towne, "Contestation and Change in National Policy on 'Scientifically Based' Educational Research," Educational Researcher 32, no. 7 (2003): 31-38
14. Coalition for Evidence-Based Policy, "Identifying and Implementing Educational Practices Supported by Rigorous Evidence: A User-Friendly Guide" (Washington, D.C.: U.S. Department of Education, 2003), http://www.ed.gov/rschstat/research/pubs/rigorousevid/guide.html. For other examples of this publicity effort, see Ron Beghetto, "Scientifically Based Research," ERIC Digest 167 (April 2003), http:// eric.uoregon.edu/publications/digests/digest167.html; Thomas D. Cook, "Sciencephobia," Education Next 1, no. 3 (2001), http://educationnext.org/20013/62.html;
15. Therese Mageau, "Determining 'What Works' - An Interview with Dr. Grover 'Russ' Whitehurst," T.H.E. Journal Online 31, no. 6 (2004);
16. and Robert E. Slavin, "A Reader's Guide to Scientifically Based Research," Educational Leadership 60, no. 5 (2003): 12-16.
17. Of course, this issue is not confined to the U.S. context. The resurgence of interest in science-based or evidence-based educational policies is manifest throughout the Anglophone world, wherever policymaking has taken up the ideologies and practices of the new managerialism, neoliberalism, and market globalization.
18. See, for example, Stephen J. Ball, "Labour, Learning and the Economy: A 'Policy Sociology' Perspective," Cambridge Journal of Education 29, no. 2 (1999): 195-206.
19. For information about the purpose and methodology of the OMB's Program Assessment Rating Tool, see http://www.whitehouse.gov/omb/part/.
20. Gert J.J. Biesta, "Bildung and Modernity: The Future of Bildung in a World of Difference," Studies in Philosophy and Education 21, no. 4-5 (2002): 346.
21. Richard J. Bernstein, The New Constellation: The Ethical-Political Horizons of Modernity/Postmodernity (Cambridge: MIT Press, 1991), 335-336.
22. See, for example, Brian Fay, Contemporary Philosophy of Social Science (Oxford: Blackwell, 1996).
23. Patti Lather, "This IS Your Father's Paradigm: Government Intrusion and the Case of Qualitative Research in Education," Qualitative Inquiry 10, no. 1 (2004): 25.
24. Ann Oakley, Experiments in Knowing (New York: The New Press, 2000), 303.
25. Ibid., 293.
26. Cook, "Sciencephobia," 9-10.
27. The central point here is that scientific research is an argument linking empirical content and conceptual work. For more on this, see Lisa Towne, Richard J. Shavelson, and Michael J. Feuer, eds., Science, Evidence, and Inference in Education: Report of a Workshop (Washington, D.C.: National Academies Press, 2001);
28. and Robert R. Alford, The Craft of Inquiry (Oxford: Oxford University Press, 1998
29. Efforts to delineate the general character of a scientific investigation of educational matters need not necessarily entail devaluing other forms of inquiry. The National Research Council committee charged with preparing Scientific Research in Education did not claim that other kinds of studies failed to qualify as important scholarship in education, nor did they argue that these other kinds of inquiries were meaningless.
30. See National Research Council, Scientific Research in Education, eds. Richard J. Shavelson and Lisa Towne (Washington, D.C.: National Academies Press, 2002).
31. I am indebted to my colleague Bill Trochim for suggesting this idea.
32. This characterization comes directly from Russ Whitehurst, "Evidence-Based Education (EBE)" (presentation delivered at the Student Achievement and School Accountability Conference, October 2002), http://www.ed.gov/nclb/methods/whatworks/eb/edlite-slide007.html.
33. See No Child Left Behind Act of 2001, Public Law 107-110 (HRl), http://www.ed.gov/nclb/ landing.jhtml?src=pb;
34. U.S. Department of Education, "New No Child Left Behind Flexibility: Highly Qualified Teachers," fact sheet, http://www.ed.gov/nclb/methods/ teachers/hqtflexibility.html; U.S. Department of Education Office of Postsecondary Education, Meeting the Highly Qualified Teachers Challenge: The Secretary's Third Annual Report on Teacher Quality (Washington, D.C.: U.S. Department of Education, 2004);
35. and The Teaching Commission, "Teaching at Risk: A Call to Action" (New York: The Teaching Commission, 2004), http://www. theteachingcommission.org. The Teaching Commission's report holds that teacher quality can be improved by linking teacher compensation to student performance, raising standards for entry into preservice teacher education programs, strengthening state licensing and certification requirements, and empowering school administrators to act as CEOs.
36. Russ Whitehurst, "Improving Teacher Quality," Spectrum: The Journal of State Government 75, no. 3 (2002): 15.
37. The following analysis draws on Joseph Dunne and Shirley Pendlebury, "Practical Reason," in The Blackwell Guide to Philosophy of Education, eds. Nigel Blake, Paul Smeyers, Richard Smith, and Paul Standish (Oxford: Blackwell, 2003), 197-198.
38. These are but four of the educational aims (what the WWC calls topic areas) that the WWC has chosen to emphasize in its systematic review of evidence to determine what works. See http://www. w-w-c.org/topics/current-topics.html.
39. Dunne and Pendlebury, "Practical Reason," 199.
40. Ian Sanderson, "Getting Evidence into Practice: Perspectives on Rationality," Evaluation 10, no. 3 (2004): 371.
41. Sarah McGough pointed out this possibility (personal communication with author, January 29, 2004).
42. Biesta, "Bildung and Modernity," 348.
43. See John Kaufman, "Education as Creative Conversation," Education Week, March 24, 2004;
44. and Robert J. Sternberg, "Good Intentions, Bad Results," Education Week, October 27, 2004.
45. David Wiggins, "Deliberation and Practical Reason," in Practical Reasoning, ed. Joseph Raz (Oxford: Oxford University Press, 1978), 150.
46. See, for example, Trisha Greenhalgh, "Narrative Based Medicine in an Evidence Based World," British Medical Journal 318 (January 1999): 324.
47. Stenhouse, quoted in John Elliott, "Making Evidence-Based Practice Educational," British Educational Research Journal 27, no. 5 (2001): 555-574.
48. David C. Berliner, "Toward a Future as Rich as Our Past" (Stanford, California: Carnegie Foundation for the Advancement of Teaching, 2003), http://www.carnegiefoundation.org/CID/essays/CID-Edu-Berliner.pdf.
49. This is a topic for another essay; however, it should be noted that there are several ways of modeling this kind of public science of educational research.
50. See, for example, Ben Agger, Public Sociology (Lanham, Maryland: Rowman and Littlefield, 2000);
51. William M. Sullivan, Reconstructing Public Philosophy (Berkeley: University of California Press, 1986);
52. and Bent Flyvbjerg, Making Social Science Matter (Cambridge, England: Cambridge University Press, 2001).
53. See the Education Science Reform Act of 2002, HR 3801, Public Law 107-279, http://www.ed.gov/ about/offices/list/ies/index.html, 126.
54. Michael J. Feuer, Lisa Towne, and Richard J. Shavelson, "Scientific Culture and Educational Research," Educational Researcher 31, no. 8 (2002): 4-14.
55. Michael J. Feuer, Lisa Towne, and Richard J. Shavelson, "Reply to Commentators on Scientific Culture and Educational Research," Educational Researcher 31, no. 8 (2002): 28
56. Frederick Erickson and Kris Gutierrez, "Culture, Rigor, and Science in Educational Research," Educational Researcher 31, no. 8 (2002): 22.
57. See also Department of Education, "Strategic Plan, 2002-2007" (Washington, D.C.: U.S. Department of Education, 2002), http://www.ed.gov/about/ reports/strat/ plan2002-07/index.html;
58. see esp. the section titled "Goal Four: Transform Education into an Evidence- Based Field."
59. Erickson and Gutierrez, "Culture, Rigor, and Science in Educational Research," 22.
60. Cook, "Randomized Experiments in Educational Policy Research," 195-196.
61. Cook, "Sciencephobia," 10.
62. See, for example, Ellen Lagemann and Lee Shulman, eds., Issues in Educational Research (San Francisco: Jossey-Bass, 1999).
63. This is the view promoted by the authors of the NRC report Scientific Research in Education.
64. See also Evelyn Jacob and C. Stephen White, eds., special issue on "Scientific Culture and Educational Research," Educational Researcher 31, no. 8 (2002), http://www.aera.net/publications/?id = 438.
65. See John Elliot, "The Paradox of Educational Reform in the Evaluatory State: Implications for Teacher Education," Prospects 32, no. 3 (2002): 273-287.
66. Just who are the consumers here is a question worthy of further exploration. There are both proximate and distant consumers, so to speak. The "nearby" consumers are the teachers and school administrators who will "buy" the educational interventions that work. However, they are making those decisions (at least ideally) on behalf of parents (acting on behalf of their children) who, desiring the best "outcomes" of education for their children, run the risk of being duped or defrauded by educational interventions that do not "work."
67. Robert Proctor, Value-Free Science? (Cambridge: Harvard University Press, 1991), 267.
68. Union of Concerned Scientists, Scientific Integrity in Policymaking: An Investigation into the Bush Administration's Misuse of Science (Cambridge, Massachusetts: Union of Concerned Scientists, 2004), http://www.ucsusa.org/global-environment/rsi/page.cfm?pagEID=1642.
69. Richard Lewontin, "Dishonesty in Science," New York Review of Books 51, no. 18 (November 18, 2004): 38.
70. General Accounting Office, "Additional Guidance Could Help Agencies Better Insure Independence and Balance" (Washington, D.C.: Government Printing Office, 2004), http://www.gao.gov/new.items/ d04328.pdf;
71. and National Academy of Science, "Science and Technology in the Public Interest: Ensuring the Best Presidential and Advisory Committee Appointments" (Washington, D.C.: National Academies Press, 2005).
72. Susan B. Neuman, "From Rhetoric to Reality: The Case for High-Quality Compensatory Prekindergarten Programs," Phi Delta Kappan 85, no. 4 (2003): 286-291.
73. See a summary and review of some of these studies in Richard Rothstein, "Must Schools Fail?" New York Review of Books 51, no. 19 (December 2, 2004): 29-32, 37.
74. Ibid.
75. At present, it is likely that more than 20,000 public schools will fail to meet the targets set for reading and math achievement by 2006. Neuman, "From Rhetoric to Reality," 287.
76. Jeffrey C. Goldfarb, The Cynical Society (Chicago: University of Chicago Press, 1991), 177.