Abusing research: The study of homework and other examples

Phi Delta Kappan

Volumn 88, Issue 1, 2006, Pages 9-22

  Kohn, Alfie ^a  

^a NONE

Author keywords

[No Author keywords available]

Indexed keywords

EID: 33748946050     PISSN: 00317217     EISSN: None     Source Type: Journal    
DOI: 10.1177/003172170608800105     Document Type: Review

References (78)

1. Frank Smith and Nel Noddings have also made this point.
2. Richard L. Allington, "Ideology k Still Trumping Evidence Phi Delta Kappan, February 2005, pp. 462-63.
3. Gary Natriello, "Failing Grades for Retention," School Administrator, August 1998, p. 15.
4. Similarly, "why is it that the stronger the research support for bilingual education" is, the "less support [we get] from policy makers?" asks James Crawford, the former executive director of the National Association for Bilingual Education (quoted in Mary Ann Zehr, "Advocates Note Need to Polish 'Bilingual' Pitch," Education Week, 1 February 2006, p. 12).
5. Douglas B. Reeves, "Galileo's Dilemma," Education Week, 8 May 2002, p. 33.
6. E. D. Hirsch, Jr., The Schools We Need: And Why We Don't Have Them (New York: Doubleday, 1996), pp. 181-82.
7. E. D. Hirsch, Jr., "Response to Prof. Feinberg," Educational Researcher, March 1998, pp. 38, 39; and
8. E. D. Hirsch, Jr., idem, The Schools We Need, p. 127.
9. Alfie Kohn, The Schools Our Children Deserve: Moving Beyond Traditional Classrooms and "Tougher Standards" (Boston: Houghton Mifflin, 1999), pp. 209-10. This was far from the only example in Hirsch's book in which the research failed to substantiate the claim for which it was cited, by the way. See also idem, pp. 294-97, n. 4.
10. See, for example, Richard L. Allington, "How to Improve High-Stakes Test Scores Without Really Improving," Issues in Education, vol. 6, 2000, pp. 115-24;
11. and Alfie Kohn, The Case Against Standardized Testing (Portsmouth, N.H.: Heinemann, 2000). The fact that no independent corroboration exists to show that testing, preceded by a steady diet of test preparation, has any real positive effect means that our children are serving as involuntary subjects in a huge high-stakes experiment.
12. Reeves, p. 44.
13. For more on this topic, see any of numerous writings by Richard Ailington, Gerald Coles, Elaine Garan, and Stephen Krashen, many of them published in this journal. See also Kohn, The Schools Our Children Deserve, pp. 159-71, 217-26.
14. Bill Jacob, "Implementing Standards: The California Mathematics Textbook Debacle," Phi Delta Kappan, November 2001, p. 266.
15. For details, see Daniel Tanner, "A Nation 'Truly' at Risk," Phi Delta Kappan, December 1993, pp. 288-97.
16. Zehr, Phi Delta Kappan, December 1993, pp. 288-97. op. cit.
17. Several years earlier, the Department of Education released a statement that boasted: "We will change education to make it an evidence-based field" (see www.ed.gov/about/reports/strat/plan2002-07/plan.pdf, p. 51). What actually seems to be taking place is an effort to change evidence to make it correspond to ideology. These examples dealing with education policies may be symptomatic of a much wider and deeper phenomenon;
18. see, for example, Chris Mooney, The Republican War on Science (New York: Basic, 2005);
19. and Union of Concerned Scientists, Scientific Integrity in Policy Making: Investigation of the Bush Administration's Abuse of Science, available at www.ucsusa.org/scientific_ integrity/interference/reports-scientific-integrity- in-policy-making.html.
20. See David J. Ferrero, "Does 'Research Based' Mean 'Value Neutral'?," Phi Delta Kappan, February 2005, pp. 425-32;
21. and Alfie Kohn, "Professors Who Profess," Kappa Delta Pi Record, Spring 2003, pp. 108-13, available at www.alfiekohn.org/teaching/ professing.htm.
22. Harry F. Harlow, "Fundamental Principles for Preparing Psychology Journal Articles," Journal of Comparative and Physiological Psychology, vol. 55, 1962, p. 895. Thanks to Gerald Bracey for calling this article to my attention.
23. See, for example, Elaine M. Garan, "Beyond the Smoke and Mirrors: A Critique of the National Reading Panel Report on Phonics," Phi Delta Kappan, March 2001, pp. 500-506.
24. The Singers played the phantom citation game, too. For example, they repeatedly asserted that children who are heavy TV viewers, or who watch any fast-paced program, cannot absorb information effectively. By way of proof, their later papers cited their early papers, but the early papers contained the same assertions in place of data. In one monograph, they claimed that certain types of programming may make children hyperactive, citing as proof three works by other researchers. When I tracked down these sources, two didn't even mention hyperactivity, and the third raised the claim only long enough to dismiss it as unsubstantiated. (Citations to the publications by the Singers are available on request. My own essay about television and children, which led me into this thicket, was eventually pub lished as "Television and Children: ReViewing the Evidence," in What to Look for in a Classroom . . . And Other Essays [San Francisco: Jossey-Bass, 1998].)
25. Alfie Kohn, The Homework Myth: Why Our Kids Get Too Much of a Bad Thing (Cambridge, Mass.: Da Capo Press, 2006). The first section of the book summarizes the evidence, the second section tries to explain why homework is so widely assigned and accepted despite what that evidence shows, and the third section draws from the practices of educators who have challenged the conventional wisdom in order to propose a different way of thinking about the subject.
26. The initial meta-analysis was published as Harris Cooper, Homework (White Plains, N.Y.: Longman, 1989);
27. it was then released in an abridged and slightly updated version as The Battle over Homework, 2nd ed. (Thousand Oaks, Calif.: Corwin, 2001).
28. The quotation appeared on p. 109 of the 1989 edition. The recent reanalysis is Harris Cooper, lorgianne Civey Robinson, and Erika A. Patall, "Does Homework Improve Academic Achievement? A Synthesis of Research, 1987-2003," Review of Educational Research, vol. 76, 2006, pp. 1-62.
29. Cooper, Homework, p. 100. The correlations were .02, 07, and .25, respectively. In the 2006 meta-analysis, Cooper and his colleagues grouped the results into grades K-6 and 7-12. The latter correlation was either .20 or .25, depending on the statistical technique being used; the former correlation was "not significantly different from zero" (Cooper, Robinson, and Patall, p. 43).
30. Valerie A. Cool and Timothy Z. Keith, "Testing a Model of School Learning: Direct and Indirect Effects on Academic Achievement," Contemporary Educational Psychology, vol. 16, 1991, pp. 28-44.
31. Cooper, Homework, p. 72. That difference shrank in the latest batch of studies (Cooper, Robinson, and Patall, 2006), but it still trended in the same direction.
32. Harris Cooper et al., "Relationships Among Attitudes About Home work, Amount of Homework Assigned and Completed, and Student Achievement," Journal of Educational Psychology, vol. 90, 1998, pp. 70-83.
33. See Kohn, The Schools Our Children Deserve and The Case Against Standardized Testing.
34. Cooper, Homework, p. 99.
35. See the table "Average Mathematics Scores by Students' Report on Time Spent Daily on Mathematics Homework at Grades 4, 8, and 12: 2000," available from the National Center for Education Statistics at http://nces. ed.gov/nationsreportcard/mathematics/results/ homework.asp.
36. As far as I can tell, no data on how 2004 NAEP math scores varied by homework completion have been published for 9- and 13-year-olds. Seventeen-year-olds were not asked to quantify the number of hours devoted to homework in 2004 but were asked whether they did homework "often," "sometimes," or "never" - and here more homework was correlated with higher scores (NAEP 2004 Trends in Academic Progress, 2005, p. 63, available at http://nces.ed.gov/nationsreportcard/pdf/main2005/2005464_3. pdf).
37. In 2000, fourth-graders who reported doing more than an hour of homework a night got exactly the same reading score as those whose teachers assigned no homework at all. Those in the middle, who said they did 30-60 minutes a night, got slightly higher scores (see http://nces.ed.gov/nationsreportcard/reading/ results/homework.asp).
38. In 2004, those who weren't assigned any homework did about as well as those who got either less than one hour or one to two hours; students who were assigned more than two hours a night did worse than any of the other three groups. For older students, more homework was correlated with higher reading scores (NAEP 2004 Trends in Academic Progress, p. 50).
39. Ina V. S. Mullis et al., Mathematics and Science Achievement in the Final Years of Secondary School: IEA's Third International Mathematics and Science Report (Boston: International Association for the Evaluation of Educational Achievement, Lynch School of Education, Boston College, 1998), p. 114.
40. Chuansheng Chen and Harold W. Stevenson, "Homework: A Cross-cultural Examination," Child Development, vol. 60, 1989, pp. 556-57.
41. David P. Baker and Gerald K. Letendre, National Differences, Global Similarities: World Culture and the Future of Schooling (Stanford, Calif.: Stanford University Press, 2005), pp. 127-28, 130. Emphasis in original.
42. Cooper's reviews confirm this conclusion (see below), as does my own literature search. And the entry on homework in the authoritative Encyclopedia of Educational Research includes the following summary statement: "Of all the research questions asked about homework, the paramount one has always focused on the relationship between homework and academic achievement." Whether homework has any effect on "objectives other than test marks and course grades -such as developing discipline and independence, extending understanding, or strengthening a positive attitude to learning - cannot be stated" (Suzanne Ziegler, "Homework," in Marvin C. Alkin, ed., Encyclopedia of Educational Research, 6th ed., vol. 2 [New York: Macmillan, 1992], p. 603).
43. Robert J. Marzano, Debra J. Pickering, and lane E. Pollock, Classroom Instruction That Works: Research-Based Strategies for Increasing Student Achievement (Alexandria, Va.: Association for Supervision and Curriculum Development, 2001).
44. Harris Cooper et al., "Relationships Between Five After-School Activities and Academic Achievement," Journal of Educational Psychology, vol. 91, 1999, pp. 369-78. The primary purpose of the study was to assess the impact of involvement in extracurricular activities, but a correlation was found between time spent on homework by older students and the grades given to them by teachers.
45. Cooper et al., "Relationships Among Attitudes."
46. Thomas L. Good, Douglas A. Grouws, and Howard Ebmeier, Active Mathematics Teaching (New York: Eongman, 1983). The first part of this book described a naturalistic study in which nine teachers whose students had high standardized math scores were compared to nine teachers whose students had lower scores. Not only was there no evidence of homework's effects relative to other variables being studied, but the authors cautioned that "correlational findings do not lead to direct statements about behaviors teachers should utilize in classrooms." In fact, far from endorsing the use of homework, or any of the other practices on display, they continued, "We were well aware of the possibility that many factors other than the behaviors we had observed in high-achievement classrooms might be responsible for the higher achievement of students" (p. 29). The second part of the book described an experimental study in which upper-elementary teachers were asked to do a number of things differently, including altering the content, context, and amount of the homework they gave. They were asked to limit it to 15 minutes a night and also to change when and how it was assigned, how it was scored, what explanations would precede it, and so on. Not only was homework only one of many simultaneous interventions, but the point was to change the homework experience, not to compare homework with its absence, so it would be impossible to infer any benefit from giving it.
47. Todd C. Gorges and Stephen N. Elliott, "Homework: Parent and Student Involvement and Their Effects on Academic Performance," Canadian Journal of School Psychology, vol. 11, 1995, p. 28. The study involved third-and fifth-graders in two suburban schools. More time spent on homework turned out not to be beneficial in three respects: there were no meaningful effects among the fifth-graders, to whom more homework had been assigned; the students who spent more time doing homework were the lower-achieving students; and the main impact of homework was on teachers' perceptions of children's competence, not on "actual subject-specific performance."
48. Michael S. Rosenberg, "The Effects of Daily Homework Assignments on the Acquisition of Basic Skills by Students with Learning Disabilities," Journal of Learning Disabilities, vol. 22, 1989, pp. 314-23. The problem, Rosenberg decided, was that the children didn't always do the homework, or do it correctly, or do it alone. He also observed that practice homework was of no value for children who hadn't already learned the material during class. In other words, his experiment too accurately matched the real world, where homework apparently provides little benefit. To remedy this, he set up a second experiment in which four children and their parents were pressed to follow his instructions to the letter. This time, drilling kids on spelling skills at home did improve quiz scores for three of the four students.
49. In another chapter, for example, Marzano and his colleagues (pp. 137-38) write: "Although the discovery approach has captured the fancy of many educators, there is not much research to indicate its superiority to other methods. Indeed, some researchers have made strong assertions about the lack of effectiveness of discovery learning, particularly as it relates to skills. For example, researchers McDaniel and Schlager (1990) note: 'In our view, discovery learning does not produce better skill' (p. 153)." That would indeed be a "strong assertion" - albeit from only one pair of researchers - if the sentence in question ended there, as Marzano and his colleagues imply that it did. But here's what McDaniel and Schlager actually wrote: "In our view, discovery learning does not produce better skill at applying the discovered strategy during transfer." On the other hand, they add a few sentences later, "The benefit of discovering a strategy seems to be that it encourages the development, practice, and/or refinement of procedures that aid the learner in generating searches for new strategies" (see Mark A. McDaniel and Mark S. Schlager, "Discovery Learning and Transfer of Problem-Solving Skills," Cognition and Instruction, vol. 7, 1990, p. 153). Marzano and his colleagues' decision to quote only the first part of the sentence in question (without acknowledging that fact) leads readers to conclude inaccurately that these researchers share their own dim view of discovery learning. As for the more general assertion that "there is not much research to indicate its superiority to other methods," everything depends on how one defines the "discovery approach." If this term is understood to mean learning that is inquiry-based, open-ended, process-oriented, or otherwise designed so that students play an active role in constructing meaning, then Marzano et al.'s statement is clearly untrue, and their omission of the numerous studies demonstrating the benefits of this approach is as misleading as their cropping of the comment by the only researchers they do cite.
50. Janine Bempechat, "The Motivational Benefits of Homework: A Social-Cognitive Perspective," Theory Into Practice, vol. 43, 2004, p. 193. These four citations are offered on the preceding page of her article (p. 192), as follows: "Those who have studied the effects of homework on academic achievement have discussed its non-academic benefits (Warton, 2001), its intermediary effects on motivation (Cooper et al., 1998), and its impact on the development of proximal student outcomes (Hoover-Dempsey et al., 2001) and general personal development (Epstein & Van Voorhis, 2001)." To be sure, all of these sources may have discussed these benefits, but none found that such benefits actually occur. The article by Hoover-Dempsey et al., for example, actually looked at the effects of parental involvement in homework, not at whether homework, per se, is beneficial. Later in her essay (p. 194) Bempechat makes another assertion: "As previous research has shown, homework is a critical means of communicating standards and expectations (Natriello & McDill, 1986)." But what those authors actually discussed was whether setting high standards and expectations led students to spend more time on their homework. Nothing in their study permits the conclusion that homework itself is useful - let alone "critical" - for communicating those standards. It's disturbing to imagine future writers citing Bempechat's article in support of the assertion that homework helps students to develop responsibility, study skills, self-discipline, and so on. (Incidentally, I've written to her twice to ask her about these discrepancies and have yet to receive a reply.)
51. Joyce L. Epstein and Frances L. Van Voorhis, "MoreThan Minutes: Teachers' Roles in Designing Homework," Educational Psychologist, vol. 36, 2001, p. 181.
52. Lyn Corno, "Homework Is a Complicated Thing," Educational Researcher, November 1996, p. 28.
53. Brian P. Gill and Steven L. Schlossman, "A Nation at Rest: The American Way of Homework," Educational Evaluation and Policy Analysis, vol. 25, 2003, p. 333.
54. Suzanne Ziegler, "Homework," ERIC ED 274 418, June 1986, p. 8.
55. The review itself: Rosanne A. Paschal, Thomas Weinstein, and Herbert J. Walberg, "The Effects of Homework on Learning: A Quantitative Synthesis," Journal of Educational Research, vol. 78, 1984, pp. 97-104.
56. The description of it: Herbert J. Walberg, Rosanne A. Paschal, and Thomas Weinstein, "Homework's Powerful Effects on Learning," Educational Leadership, April 1985, p. 79.
57. Bill Barber, "Homework Does Not Belong on the Agenda for Educational Reform," Educational Leadership, May 1986, p. 56. In that article, he also remarked that "if research tells us anything" about homework, it's that "even when achievement gains have been found, they have been minimal, especially in comparison to the amount of work expended by teachers and students" (p. 55).
58. Cooper, The Battle over Homework, p. xi;
59. and Harris Cooper and Jeffrey C. Valentine, "Using Research to Answer Practical Questions About Homework," Educational Psychologist, vol. 36, 2001, p. 144.
60. Cooper, The Battle over Homework, p. 64.
61. Laura Muhlenbruck, Harris Cooper, Barbara Nye, and James J. Lindsay, "Homework and Achievement: Explaining the Different Strengths of Relation at the Elementary and Secondary School Levels," Social Psychology of Education, vol. 3, 2000, p. 315.
62. Harris Cooper, "Synthesis of Research on Homework," Educational Leadership, November 1989, p. 90.
63. Harris Cooper, "Synthesis of Research on Homework," Educational Leadership, November 1989, p. 90. Ibid., p. 89.
64. Cooper, The Battle over Homework, p. 58.
65. Cooper and Valentine, p. 151.
66. The phrase "we also reviewed the research" apparently refers to an extended passage earlier in the essay that summarizes one of Cooper's previous articles - namely, Muhlenbruck, Cooper, Nye, and Lindsay - but that article contains no data to support these claims.
67. For example, "'Homework teaches children study and time-management skills,' [Cooper] said. . . . 'All kids should be doing homework'" (Jo Napolitano, "School's Lesson Plan: No More Homework," Chicago Tribune, 7 May 2005).
68. And a columnist for the American School Board Journal writes, "As you might expect, [Cooper] finds plenty of positive effects associated with homework, including improving students' study skills [and] developing their self-direction and responsibility" (Susan Black, "The Truth About Homework," American School Board Journal, October 1996, p. 49).
69. Cooper, Homework, p. 175.
70. Cooper says that he intends to draw not only from the data but from the "tacit knowledge" (the quotation marks are his) that he acquired from reading publications on the subject that don't include any data and also from "discussing homework issues with friends and colleagues" (Homework, p. 175). That seems reasonable, but only if one makes clear which of the resulting opinions aren't substantiated by actual research.
71. The Battle over Homework, p. 65.
72. Cooper also says that "general ranges for the frequency and duration of assignments" should be "influenced by community factors" (pp. 64-65). He doesn't explain what this means, but elsewhere he is quoted as suggesting that more homework might be given in a high-pressure suburban district - presumably because parents are demanding it, not because it is in any way justified (see Michael Winerip, "Homework Bound," Education Life
73. special supplement, New York Times, 3 January 1999, p. 40).
74. Cooper, The Battle over Homework, p. 28
75. summarizing Cooper et al., "Relationships Among Attitudes."
76. Example 1 The data reported by Cooper et al., "Relationships Among Attitudes," which I displayed above, offered a pretty compelling case that homework didn't do much for achievement regardless of how the results were carved up. But in the "Practical Implications" section of their conclusion (p. 82), the authors gave a very different impression. "First, by examining complex models and distinguishing between homework assigned and homework completed, we were able to show that, as early as the second and fourth grades, the frequency of completed homework assignments predicts grades." In fact, what they found was a "nonsignificant trend" toward a correlation between how much of the assigned homework the students said they did and what grades their teachers gave them - a finding that arguably would have no practical significance even if had been statistically significant. The authors continue: "Further, to the extent that homework helps young students develop effective study habits" - and of course they provide no evidence that this happens to any extent -"our results suggest that homework in early grades can have a long-term developmental effect that reveals itself as an even stronger relationship between completion rates and grades when the student moves into secondary school. Thus we suggest that the present study supports the assignment of homework in early grades, not necessarily for its immediate effects on achievement but rather for its potential long-term impact." This remarkable claim is based solely on the fact that the same correlation (between how much of the assigned homework kids claimed to do and what grades they ultimately received) was significant for older students. Given that teachers' grades generally reflect students'compliance with respect to a lot of things, it's amazing that there wasn't a strong correlation at all age levels. But there isn't a shred of evidence that the practice of assigning homework - which, remember, is what the authors are attempting to defend -has a beneficial "long-term impact" just because older kids get better grades for doing what they're told. Example 2: In Muhlenbruck et al., Cooper and his associates announce in their conclusion section that "homework appears to be assigned for different reasons in elementary school than in secondary school" (p. 315). This is evidently the outcome they were hoping to find in order to support the position that a lack of achievement effects for younger children shouldn't bother us because homework at that age is really just about teaching study skills and responsibility. But what the researchers actually investigated in this study was what teachers believe is beneficial to students of different ages, which, needless to say, doesn't prove that such benefits exist. Even those perceived differences, while statistically significant, were less than overwhelming. When asked whether they thought homework improved time-management skills, and when their responses ("very much," "some," or "not at all") were converted to a numeric scale, the average response of 28 elementary teachers worked out to 2.86, whereas the average response of 52 high school teachers was 2.6. (The high school teachers were also slightly less enthusiastic in endorsing the idea that homework helped students to learn [2.6 vs. 2.78], which pretty much undercuts the whole premise that elementary school homework is uniquely intended for nonacademic purposes.) Other conclusions in this study, concerning possible explanations for the fact that homework is of no academic benefit to elementary school students, are similarly constructed on the basis of dubious and marginal results; see p. 314 and compare what's said there to what had been reported earlier.
77. Ziegler, "Homework," 1992, p. 604.
78. Avram Goldstein, "Does Homework Help? A Review of Research," Elementary School Journal, vol. 60, 1960, p. 222.