To sample or not to sample? The 2000 census controversy

Journal of Interdisciplinary History

Volumn 30, Issue 1, 1999, Pages 1-36

  Anderson, Margo ^a   Fienberg, Stephen E ^b  

^a UNIVERSITY OF WISCONSIN MILWAUKEE   (United States)

^b CARNEGIE MELLON UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 33749309450     PISSN: 00221953     EISSN: None     Source Type: Journal    
DOI: 10.1162/002219599551895     Document Type: Article

References (46)

1. Memorandum for Plaintiff in Support of its Motion for Summary Judgment, U.S. House of Representatives et al. v. U.S. Department of Commerce et al., April 6, 1998, 1.
2. Memorandum of Law in Support of Plaintiffs' Motion for Summary Judgment, Glavin, Barr, et al. v. Clinton et al., April 6, 1998, 1.
3. Drew Days, Solicitor General of the United States, Statement from January 10, 1996, oral argument before the United States Supreme Court in Wisconsin v. New York, 517 U.S. 1 (1996), the 1990 Census adjustment lawsuit.
4. Memorandum for Plaintiff in Support of its Motion for Summary Judgment, U.S. House of Representatives et al. v. U.S. Department of Commerce et al., April 6, 1998, 1.
5. "Census Sampling Would Favor Democrats," Human Events, 22 Aug. 1997.
6. Rep. Dennis Hastert (R-IL), quoted in Associated Press report, July 14, 1997.
7. William Safire, "Sampling Is Not Enumerating," New York Times, 7 Dec. 1997.
8. Memorandum for Plaintiff in Support of its Motion for Summary Judgment, U.S. House of Representatives et al. v. U.S. Department of Commerce et al., April 6, 1998, 12.
9. "Will You Be Counted in 2000? Census Plans to 'Sample' May Lead to Big Errors," Investor's Business Daily, 8 July 1998, 1, 32;
10. David A. Freedman, "Adjusting the Census of 1990," Jurimetrics, XXXIV (1993), 102.
11. Memorandum for Plaintiff in Support of its Motion for Summary Judgment, U.S. House of Representatives et al. v. U.S. Department of Commerce et al, April 6, 1998, 13-14; "Delco, Bucks Join Lawsuit Over Census," Philadelphia Inquirer, 7 May 1998, 1.
12. See, for example, U.S. General Accounting Office, 1990 Census: Reported Net Undercount Obscured Magnitude of Error (Washington, D.C., 1991);
13. Eugene Ericksen and Teresa Defonso, "Beyond the Net Undercount: How to Measure Census Error," Chance, 6 (Fall 1993), 14, 38-43.
14. More precisely, as of a fixed date, the Bureau will send enumerators to a sample of all households that do not respond to the mail questionnaire. The rate of sampling will vary by tract to achieve a total response rate for each tract of at least 90 percent. A tract with a 50 percent mail-response rate would have an 80 percent sampling rate of the nonresponding households to get it up to 90 percent. Tracts with primary response rates in excess of 85 percent would be sampled at a 1 in 3 rate. Any tract with a primary response rate exceeding 95 percent would get a slightly reduced rate of sampling. The Bureau plans to make inferences from the sample to the residual nonresponders on a tract-by-tract basis. Every census tract will have a targeted completion rate of at least 90 percent of households. The sampling information would allow for inferences about those residual households for which no direct information was available.
15. The dual-systems method is a human-population version of the well-known capture-recapture method for estimating the size of a population. It is used in a variety of scientific contexts. For each block in the sample follow-up, the method combines the census results and the sample follow-up, thus directly adding those households and individuals that are found in the sample but not in the census. Then it adds an additional estimate of those found in neither the census nor the sample. The method uses the independence of the census and the sample to achieve this estimate. Though conceptually simple statistically, the rationale for the method is often difficult for nontechnical people to follow. Even Chief Justice William Rehnquist misdescribed it in his 1996 opinion on the 1990 census-adjustment controversy - Wisconsin v. New York, 517 U.S. 1 (1996). Two other complications have been the focus of much debate in the statistical community about the methodology. First, before applying the usual capture-recapture formulae, the Census Bureau does an adjustment to the block-by-block counts for erroneous enumerations in the census. Second, to improve the accuracy of the projection of the adjustments from the sample blocks to the rest of the nation, the Bureau used what was referred to as a "smoothing," or regression, model. An explication of the method and an exchange of views about whether it provides demonstrable improvements over the raw census counts can be found in a series of four articles in Jurimetrics, XXXIV (1993), 59-115, written by Fienberg, John Rolph, Freedman, and Kenneth Wachter, four of the statistical experts who participated in the trial that followed the 1991 decision not to adjust the 1990 census.
16. Anderson, The American Census: A Social History (New Haven, 1988).
17. Daniel O. Price, "A Check on Underenumeration in the 1940 Census," American Sociological Review, XII (1947), 44-49.
18. Ansley Coale, "The Population of the United States in 1950 by Age, Sex, and Color: A Revision of Census Figures," Journal of the American Statistical Association, L (1955), 16-54.
19. Leon Pritzker and Naomi D. Rothwell, "Procedural Difficulties in Taking Past Censuses in Predominantly Negro, Puerto Rican, and Mexican Areas," in David Heer (ed.), Social Statistics and the City (Cambridge, 1968), 55-79.
20. Heer, Social Statistics, 11.
21. Anderson and Fienberg, Who Counts? The Politics of Censustaking in Contemporary America (New York, forthcoming).
22. Constance Citro and Michael Cohen, The Bicentennial Census (Washington, D.C., 1985), 187.
23. New York v. United States Department of Commerce, 822 F. Supp. 906; 34 F. 3d 1114; Wisconsin v. City of New York, 517 U.S. 1 (1996).
24. Transcript of Mosbacher press conference, July, 15, 1991, Plaintiffs Exhibit, no. 678, City of New York v. United States Department of Commerce. 23 Members of the panel on Census Requirements for the Year 2000 and Beyond and of the panel to Evaluate Alternative Census Methods were appointed during the following months and began meeting in June 1992. Charles Schultze, former Office of Management and Budget official and economist, headed the Requirements panel. Norman Bradburn of the National Opinion Research Center headed the Methodology panel. Anderson and Fienberg were members of the Requirements panel.
25. Duane Steffey and Norman Bradburn (eds.), Counting People in the Information Age (Washington, D.C., 1994);
26. Barry Edmonston and Schultze (eds.), Modernizing the U.S. Census (Washington, D.C., 1995).
27. Steven Holmes, "Budget Cutting May Hit Census," New York Times, 23 Aug. 1995, 1.
28. Concerning the draft proposals for the "Re-engineered 2000 Census," see, for example, U.S. Bureau of the Census, DRAFT, "The Re-engineered 2000 Census," May 19, 1995.
29. U.S. Bureau of the Census, "The Plan for Census 2000, Revised and Reissued," February 28, 1996 (typescript).
30. Statement of the Honorable Herb Kohl, House Committee on Government Reform and Oversight, "Plans for the 2000 Census," February 29, 1996; Statement of Rep. Tom Petri to the Committee on Government Reform and Oversight, February 28, 1996.
31. Wachter and Freedman, Testimony on Plans for Census 2000, House Committee on Government Reform and Oversight, February 29, 1996.
32. See, for example, the report of the Committee on Government Reform and Oversight, U.S. House of Representatives, "Sampling and Statistical Adjustment in the Decennial Census: Fundamental Flaws" (Washington, D.C., 1996).
33. Integrating the SNRFU procedures with the ICM is new territory. The Bureau will be looking to the results of the dress rehearsal the better to understand the implications.
34. See, for example, the testimony before the Senate Committee on Governmental Affairs about "Census 2000," Senate Hearing 105-233, March 11 and April 16, 1997 (Washington, D.C., 1997), from Census Director Riche; Schultz, Chair of the National Academy of Sciences Panel on Census Requirements for the Year 2000 and Beyond; Commerce Secretary William Daley; and Under Secretary Everett Ehrlich. On the anti-sampling side, Wisconsin Attorney General James E. Doyle testified in support of an "actual head count." Former Assistant Attorney General Stuart Gerson cataloged the background surrounding the legality of sampling, noting that Section 195 of Title 13 governing the Census Bureau had not conclusively been found to bar sampling in the decennial count. Lawrence Brown, University of Pennsylvania statistician, testified that though he had some technical reservations about sampling for nonresponse follow-up, he believed "that the current plans [were] a very significant improvement over those in effect up to as recently as a month [earlier]. Nevertheless, if economic considerations [had permitted], in short, if Congress could [have found] the money, [he] would [have preferred] to see a full follow-up rather than the current sample response follow-up plan."
35. U.S.C. 141 note, as enacted in Section 210 of the Departments of Commerce, Justice, and State, the Judiciary, and Related Agencies Appropriations Act, 1998 (P.L. 105-119, 111 Stat. 2480, 2483-2487).
36. Brown et al., "Statistical Controversies in Census 2000." Technical Report 357, Department of Statistics, University of California, Berkeley (October 1998);
37. Cohen, Andrew A. White, and Keith Rust (eds.), Measuring a Changing Nation: Modern Methods for the 2000 Census (Washington, D.C., forthcoming).
38. Jerzy Neyman, "On the Two Different Aspects of the Representative Method: The Method of Stratified Sampling and the Method of Purposive Selection (with Discussion)," Journal of the Royal Statistical Society, XCVII (1934), 558-625;
39. Morris H. Hansen and William N. Hurwitz, "On the Theory of Sampling from Finite Populations," Annals of Mathematical Statistics, XIV (1943), 333-362;
40. Morris H. Hansen and William N. Hurwitz, idem and William G. Madow, Sample Survey Methods and Theory (New York, 1953), 2v. Neyman's groundbreaking 1934 article demonstrated, among other things, the virtues of probability sampling, as opposed to the purposive method; stratification; cluster sampling; and optimal allocation of sampling units to strata. A few years later at the Census Bureau, Hansen and Hurwitz helped to redesign the unemployment survey based on new ideas about multi-stage probability samples and cluster sampling. They expanded and applied their approach in various Bureau surveys, often in collaboration and interaction with others, and this effort culminated in their two-volume compendium, with Madow, in 1953.
41. Freedman and Wachter offered no detailed calculations in support of their allegation about the adjusted 1990 count. In his report about the analyses of the 1990 post-enumeration results, Howard Hogan, "The 1990 Post Enumeration Survey: Operations and Results," Journal of the American Statistical Association, LXXXVIII (1993), 1047-1060,
42. explained that the Census Bureau made several changes to the PES data and the adjusted census counts subsequent to the secretary's 1991 decision, as part of the Bureau's ongoing research program. Among them was a set of corrections for errors associated with the computer editing of erroneous enumerations and a change in the stratification scheme from the 1,392 post-strata chosen before the 1991 decision to 357 post-strata, in order to eliminate the smoothing component of the estimation process. Hogan presented the combined results of these steps on the state population counts. The use of these numbers for apportionment instead of the original adjusted counts released in 1991 leads to a conclusion similar to that of Freedman and Wachter (treating the new figures as if they were correct). However, because similar calculations based on the correction of the computer error alone fail to support their conclusion, we characterize their statement as part of another myth. This discussion illustrates the sensitivity of the official apportionment formula to very small changes in counts, a phenomenon widely understood among political scientists. Hence, it is important for the Bureau to announce its procedures - say, for post-stratification - in advance of data collection and analysis. Freedman and Wachter rely on Leo Breiman, "The 1991 Census Adjustment: Undercount or Bad Data," Statistical Science, IX (1994), 461,
43. for the evaluation of the level of error (55%) in the undercount estimate. Other evaluations of the PES do not accept Breiman's characterization of the 1990 census. See, for example, Hogan, "The 1990 Post Enumeration Survey;
44. Mary Mulry and Bruce Spencer, "Accuracy of the 1990 Census and Error Adjustments," Journal of the American Statistical Association, LXXX (1993), 1080-1091;
45. or the comments by Thomas Belin and John Rolph, and Ericksen, Fienberg and Kadane, in Statistical Science, IX (1994), 486-508, 511-515. Further, even were the statement a correct and proper interpretation of the known data, it would be effectively irrelevant to the usefulness of the DSE approach. The net national undercount, which is the difference between omissions and erroneous enumerations, is not the reason why the Census Bureau carried out the PES and did DSE. The closer the estimates for omissions and erroneous enumerations come to balancing out, the closer the net undercount estimate is to zero, and the larger the impact of "errors." In a zero-net estimated undercount, a single error would account for it all! Because omission and erroneous enumerations are not distributed in the same way across states and lower levels of geography, however, the effect of errors diminishes as one moves to comparisons of interest. Thus, Breiman's seemingly explosive statement may be of little importance when we come to understand the improved accuracy of PES-adjusted data. For a concrete example, see Appendix 1.
46. In 1850, Congress also created a board to deal with census problems. After the 1920 census, Congress failed to reapportion itself on account of a dispute that has considerable parallels with the current controversy. See Anderson, American Census.