Randomizing law

University of Pennsylvania Law Review

Volumn 159, Issue 4, 2011, Pages 929-1005

  Abramowicz, Michael ^a   Ayres, Ian ^b   Listokin, Yair ^b  

^a GEORGE WASHINGTON UNIVERSITY   (United States)

^b YALE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 79953806572     PISSN: 00419907     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (253)

1. See A. de G., The Scholar as Government Consultant: The Great Salt and Iron Debate in Ancient China, 8 AM. BEHAV. SCIENTIST 4, 4-6 (1965).
2. See, e.g., Orit Fischman Afori, Reconceptualizing Property in Designs, 25 CARDOZO ARTS & ENT. L.J. 1105, 1151 (2008) (referring to a statute providing intellectual property protection for vessel hulls as a "legal experiment");
3. Theodor Meron, Reflections on the Prosecution of War Crimes by International Tribunals, 100 AM. J. INT'L L. 551, 551 (2006) (referring to the Nuremberg and Tokyo war crimes tribunals as "a bold legal experiment").
4. See generally Alan Milner, Restatement: The Failure of a Legal Experiment, 20 U. PITT. L. REV. 795 (1959) (characterizing Restatements of the Law as a failed experiment). The most prominent academic account of experimental approaches to government also defines experimentation broadly, mentioning randomization as a possible ingredient of experimentation only once.
5. See Michael C. Dorf & Charles F. Sabel, A Constitution of Democratic Experimentalism, 98 COLUM. L. REV. 267, 348 (1998) (noting that systems for evaluating experiments "can themselves be benchmarked, and . . . can be combined with random-assignment experiments and other familiar methods of evaluation").
6. see Tar Timothy Chen, History of Statistical Thinking in Medicine, in ADVANCED MEDICAL STATISTICS 3, 11-14 (Ying Lu & Ji-Qian Fang eds., 2003).
7. See also R.A. Fisher, The Arrangement of Field Experiments, 33 J. MINISTRY AGRIC. GR. BRIT. 503, 506-07 (1926) (suggesting the use of random trials in agricultural field experimentation because "[o]ne way of making sure that a valid estimate of error will be obtained is to arrange the plots deliberately at random so that no distinction can creep in between pairs of plots treated alike and pairs treated differently").
8. A doctoral student, Heather Ross, developed the idea for an experiment on the effect of a negative income tax and then received governmental funding for her experiment. The experimental results are reported in three volumes. See generally DAVID KERSHAW & JERILYN FAIR, 1 THE NEW JERSEY INCOME-MAINTENANCE EXPERIMENT: OPERATIONS, SURVEYS, AND ADMINISTRATION (1976);
9. THE NEW JERSEY INCOMEMAINTENANCE EXPERIMENT: LABOR-SUPPLY RESPONSES (Harold W. Watts & Albert Rees eds., 1977);
10. THE NEW JERSEY INCOME-MAINTENANCE EXPERIMENT: EXPENDITURES, HEALTH, AND SOCIAL BEHAVIOR; AND THE QUALITY OF THE EVIDENCE (Harold W. Watts & Albert Rees eds., 1977). For useful summaries of the experiment
11. see DAVID GREENBERG ET AL., SOCIAL EXPERIMENTATION AND PUBLIC POLICYMAKING 111-64 (2003);
12. Frank P. Stafford, Income-Maintenance Policy and Work Effort: Learning from Experiments and Labor-Market Studies, in SOCIAL EXPERIMENTATION 95, 111 tbl.3.5 ( Jerry A. Hausman & David A. Wise eds., 1985).
13. See, e.g., Machaela M. Hoctor, Comment, Domestic Violence as a Crime Against the State: The Need for Mandatory Arrest in California, 85 CALIF. L. REV. 643, 655-57 (1997) (commenting on a Minneapolis experiment, in which police, assuming probable cause, assigned at random the arrest of alleged domestic violence perpetrators, and noting that suspects who were arrested had the lowest rate of recidivism). (Pubitemid 127426889)
14. See, e.g., Bernard E. Harcourt, Post-Modern Meditations on Punishment: On the Limits of Reason and the Virtues of Randomization (A Polemic and Manifesto for the Twenty-First Century), 74 SOC. RES. 307, 328-30 (2007) (proposing randomization in several areas, such as criminal justice, where it could be used to set the length of prison sentences);
15. Laurens Walker, Perfecting Federal Civil Rules: A Proposal for Restricted Field Experiments, LAW & CONTEMP. PROBS., Summer 1988, at 67, 72-77 (proposing randomized experiments on rules of civil procedure).
16. See GREENBERG ET AL., supra note 4, at 26 ("[M]ost social experiment test programs are targeted at persons or families who are somehow disadvantaged, particularly in terms of having low incomes.").
17. Medicare Prescription Drug, Improvement, and Modernization Act of 2003 § 721(b)(1), 42 U.S.C. § 1395b-8(b)(1) (2006) (requiring "development, testing, and evaluation of chronic care improvement programs using randomized controlled trials").
18. See NANCY MCCALL ET AL., RTI INT'L, EVALUATION OF PHASE 1 OF MEDICARE HEALTH SUPPORT (FORMERLY VOLUNTARY CHRONIC CARE IMPROVEMENT) PILOT PROGRAM UNDER TRADITIONAL FEE-FOR-SERVICE MEDICARE 1-5 (2007) (reporting preliminary results of the programs);
19. Reed Abelson, Medicare Finds How Hard It Is to Save Money, N.Y. TIMES, Apr. 7, 2008, at A1 (describing one such program).
20. See generally David P. Farrington & Brandon C. Welsh, A Half Century of Randomized Experiments on Crime and Justice, 34 CRIME & JUST. 55 (2006) (providing an overview of randomized criminal justice experiments from 1957 to 2004).
21. See Dennis J. Aigner, The Residential Electricity Time-of-Use Pricing Experiments: What Have We Learned? ("The purpose of the present paper is to consider the empirical results available so far from the DOE experiments in light of design and analysis concerns . . . ."), in SOCIAL EXPERIMENTATION, supra note 4, at 11, 12;
22. see also 1 RESEARCH TRIANGLE INST., ANALYTICAL MASTER PLAN FOR THE ANALYSIS OF THE DATA FROM THE ELECTRIC UTILITY RATE DEMONSTRATION PROJECTS 1 (1978) (analyzing projects designed to "evaluate experimentally the effects of time-of-use pricing of electricity for residential customers").
23. See R.A. FISHER, 3 STATISTICAL METHODS FOR RESEARCH WORKERS 203-09 (1925) (describing this experiment);
24. see also IAN AYRES, SUPER CRUNCHERS 46-80 (2007) (discussing the power of randomization as a tool for business and the government).
25. RONALD A. FISHER, THE DESIGN OF EXPERIMENTS 11 (Hafner Publ'g Co., 6th ed. 1951) (1935).
26. RONALD A. FISHER, THE DESIGN OF EXPERIMENTS 11, (1935), Id.
27. David P. Harrington, The Randomized Clinical Trial, 95 J. AM. STAT. ASS'N 312, 312 (2000).
28. David P. Harrington, The Randomized Clinical Trial, 95 J. AM. STAT. ASS'N 312, (2000), For more information on the logic of clinical trials, see generally id.
29. For more information about the trial and its methodology, see Streptomycin in Tuberculosis Trials Comm., Med. Research Council, Streptomycin Treatment of Pulmonary Tuberculosis, 2 BMJ 769 (1948).
30. Pub. L. No. 87-781, § 102(c), 76 Stat. 780, 781 (1962) (codified as amended at 21 U.S.C. § 355(d) (2006)).
31. See generally Karen Baswell, Note, Time for a Change: Why the FDA Should Require Greater Disclosure of Differences of Opinion on the Safety and Efficacy of Approved Drugs, 35 HOFSTRA L. REV. 1799, 1832 (2007) (arguing for "increased transparency and disclosure from the pharmaceutical and pharmaceutical research industries").
32. See Abigail Alliance for Better Access to Developmental Drugs v. Von Eschenbach, 495 F.3d 695, 697, 706 n.12 (D.C. Cir. 2007) (en banc) (considering whether "terminally ill patients [have] a right of access to experimental drugs that have passed limited safety trials but have not been proven safe and effective" and noting that "'[t]he history of the effectiveness requirement in drug regulation is inextricably linked to the advent of the randomized, controlled clinical trial as the cornerstone of medical research'"
33. (quoting Jennifer Kulynych, Will FDA Relinquish the "Gold Standard" for New Drug Approval? Redefining "Substantial Evidence" in the FDA Modernization Act of 1997, 54 FOOD & DRUG L.J. 127, 131 (1999)));
34. C.F.R. § 314.50(d)(5) (2010) (describing the required presentation of controlled and uncontrolled clinical trials in an application for FDA approval);
35. see also Charles J. Walsh & Alissa Pyrich, Rationalizing the Regulation of Prescription Drugs and Medical Devices: Perspectives on Private Certification and Tort Reform, 48 RUTGERS L. REV. 883, 889 (1996) (suggesting a plan to delegate some of the FDA's regulatory authority to private bodies in order to increase efficiency); (Pubitemid 126408211)
36. cf. 40 C.F.R. § 799.9420(d)(1)(iv)(D) (2009) (mandating randomized testing of toxic substances in rodents).
37. The National Environmental Policy Act of 1969, 42 U.S.C. § 4332(2)(C) (2006), requires an EIS for any "major Federal action significantly affecting the quality of the human environment." The purpose of the EIS is to improve agency decisionmaking by requiring "detailed information concerning significant environmental impacts."
38. Robertson v. Methow Valley Citizens Council, 490 U.S. 332, 349 (1989). Executive Order Number 12,866 states that "[i]n deciding whether and how to regulate, agencies should assess all costs and benefits of available regulatory alternatives, including the alternative of not regulating."
39. Exec. Order No. 12,866, 3 C.F.R. 638, 639 (1994), reprinted as amended
40. U.S.C. § 601 (2006). "The objectives of this Executive Order are to enhance planning and coordination . . . to reaffirm the primacy of Federal agencies in the regulatory decision-making process; to restore the integrity and legitimacy of regulatory review and oversight; and to make the process more accessible and open to the public."
41. See Bryan Keefer, Tsunami, COLUM. JOURNALISM REV., July-Aug. 2004, at 18, 18-23 (discussing reporters' reluctance to take sides on issues of public controversy during elections).
42. Cf. Scott Brewer, Scientific Expert Testimony and Intellectual Due Process, 107 YALE L.J. 1535, 1552-53 (1998) (describing a similar problem when judges and jurors try to assess scientific evidence beyond their competence).
43. For a useful overview of regression analysis, see generally WILLIAM MENDENHALL & TERRY SINCICH, A SECOND COURSE IN STATISTICS: REGRESSION ANALYSIS (6th ed. 2003). For a critical analysis of the use of empirical evidence in legal scholarship
44. see Lee Epstein & Gary King, The Rules of Inference, 69 U. CHI. L. REV. 1, 14-19 (2002).
45. For a study on the relationship between sexual activity and sex education, see Deborah Anne Dawson, The Effects of Sex Education on Adolescent Behavior, 18 FAMILY PLANNING PERSP. 162, 162 (1986).
46. Julie Berry Cullen et al., The Effect of School Choice on Student Outcomes: Evidence from Randomized Lotteries (Nat'l Bureau of Econ. Research, Working Paper No. 10113, 2003).
47. see also, e.g., Caroline M. Hoxby & Sonali Murarka, Methods of Assessing the Achievement of Students in Charter Schools 24 (Sept. 28, 2006) (unpublished manuscript), available at http://www.vanderbilt.edu/schoolchoice/ conference/papers/Hoxby-Murarka-2006-DRAFT.pdf (noting possible variables that increase self-selection bias in students who apply to charter schools, such as "prior achievement" or "parental motivation").
48. Paul Glewwe et al., Retrospective vs. Prospective Analyses of School Inputs: The Case of Flip Charts in Kenya (Nat'l Bureau of Econ. Research, Working Paper No. 8018, 2000).
49. See Steven D. Levitt, Using Electoral Cycles in Police Hiring to Estimate the Effect of Police on Crime, 87 AM. ECON. REV. 270 (1997). A regression simply using the change in the crime rate as a dependent variable and police presence as an independent variable would produce a biased coefficient, because the choice of how many police to hire is "endogen[ous]" and may depend in part on expectations of whether crime is likely to increase.
50. see Jonathan Klick & Alexander Tabarrok, Using Terror Alert Levels to Estimate the Effect of Police on Crime, 48 J.L. & ECON. 267 (2005), which uses terror-alert levels in Washington, D.C., as an instrument because the city deployed more police when the alert levels were raised, even though the terror-alert levels would not be correlated to ordinary crime rates. For an overview of regression-discontinuity studies
51. see Jinyong Hahn et al., Identification and Estimation of Treatment Effects with a Regression-Discontinuity Design, 69 ECONOMETRICA 201 (2001).
52. M. Keith Chen & Jesse M. Shapiro, Do Harsher Prison Conditions Reduce Recidivism? A Discontinuity-Based Approach, 9 AM. L. & ECON. REV. 1 (2007), which uses the variability in prison conditions as a function of offense levels to analyze recidivism rates.
53. See J.A. Hausman, Specification Tests in Econometrics, 46 ECONOMETRICA 1251, 1251 (1978) (proposing a "general form of specification test").
54. Saurabh Bhargava & Vikram Pathania, Driving Under the (Cellular) Influence: The Link Between Cell Phone Use and Vehicle Crashes 3 (AEI-Brookings Joint Ctr. for Regulatory Studies, Working Paper No. 07-15, 2007), available at http://ssrn.com/abstract=1089081.
55. Cf. Mike Stuckey, Hands-Free Phones Are Lifesavers, Study Says, MSNBC.COM, May 13, 2008, http://www.msnbc.msn.com/id/24580099 (discussing a study "predicting that banning the use of hand-held phones by U.S. drivers could save thousands of lives each year").
56. Some researchers have sought to counter publication bias by encouraging the publication of statistically insignificant results. See, e.g., Huai yong Cheng, Letter to the Editor, The Potential Value of Negative Studies, 6 J. AM. MED. DIRECTORS ASS'N 426 (2005). (Pubitemid 41755370)
57. See J. Bradford De Long & Kevin Lang, Are All Economic Hypotheses False?, 100 J. POL. ECON. 1257, 1258 (1992) (discussing "an approach that allows [its authors] to measure the fraction of unrejected null hypotheses that are, in fact, false"). The De Long and Lang statistical analysis rejects "the null hypothesis that more than about one-third of unrejected null hypotheses are true."
58. See, e.g., WILLIAM H. GREENE, ECONOMETRIC ANALYSIS 116-144 (5th ed. 2003) (discussing the "functional form of the regression model").
59. There may be flexibility both in determining the general coverage of the study (for example, what years or states to study), as well as in identifying outliers. Typically, when a researcher identifies an observation as an outlier, she will run a regression both with and without the outliers to determine whether the results are robust. There are also econometric techniques designed to produce estimates not overly susceptible to outliers. See, e.g., PETER J. ROUSSEEUW & ANNICK M. LEROY, ROBUST REGRESSION AND OUTLIER DETECTION 10 (2003) (describing regression techniques that account for the large influence of outliers).
60. See T. Dudley Wallace, Pretest Estimation in Regression: A Survey, 59 AM. J. AGRIC. ECON. 431, 431 (1977) ("Given the nature of economic data, pretesting has been and probably will continue to be widely used in spite of sharp criticism.").
61. The traditional t-statistic will be inaccurate when researchers test numerous regression specifications and then focus on only those whose t-statistics appear to indicate statistically significant results. See, e.g., Dmitry Danilov & Jan R. Magnus, Forecast Accuracy After Pretesting with an Application to the Stock Market, 23 J. FORECASTING 251, 258 (2004) (deciding which model to use depending on whether the t-statistic is significant). For a discussion of the origin and formation of the t-statistic
62. see ARTHUR M. GLENBERG & MATTHEW E. ANDRZEJEWSKI, LEARNING FROM DATA 243 (3d ed. 2008).
63. See Jeff Strnad, Should Legal Empiricists Go Bayesian?, 9 AM. L. & ECON. REV. 195, 197 (2007) (noting that, in law, "the researcher is dealing with observational data that cannot be extended by additional experimentation").
64. See John J. Donohue & Justin Wolfers, Uses and Abuses of Empirical Evidence in the Death Penalty Debate, 58 STAN. L. REV. 791, 840 & fig.10 (2005) (finding a "statistically significant correlation" between the estimated coefficients and standard errors reported within most studies).
65. See Hashem Dezhbakhsh & Joanna M. Shepherd, The Deterrent Effect of Capital Punishment: Evidence from a "Judicial Experiment" 13-15 (Emory L. & Econ. Paper Series, Working Paper No. 04-04 & Emory Univ. Econ. Working Paper No. 03-14, 2004), available at http://ssrn.com/abstract=432621 (analyzing the "effect of both suspending and reinstating the death penalty on state murder rates" and concluding that "states reinstating their death penalty experience a drop in murder rates").
66. See John J. Donohue III & Steven D. Levitt, The Impact of Legalized Abortion on Crime, 116 Q.J. ECON. 379, 414 (2001) (presenting evidence suggesting that "legalized abortion is a primary explanation for the large drops in murder, property, crime, and violent crime").
67. See JOHN R. LOTT, JR., MORE GUNS, LESS CRIME 159 (1998) ("Allowing citizens without criminal records or histories of significant mental illness to carry concealed handguns deters violent crimes . . . .").
68. See, e.g., Erick H. Turner et al., Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy, 358 NEW ENG. J. MED. 252, 253-56 (2008) (analyzing which reviews of antidepressant agents submitted to the FDA were subsequently published and observing a "bias toward the publication of positive results").
69. Esther Duflo, Scaling Up and Evaluation, in ANNUAL WORLD BANK CONFERENCE ON DEVELOPMENT ECONOMICS 2004: ACCELERATING DEVELOPMENT 341, 353 (François Bourguignon & Boris Pleskovic eds., 2004).
70. see James M. Wood & Roxanne M. Gariby, Hoarding Away Science: Towards a More Transparent View of Health and Online Registries for Independent Postmarket Drug Research, 60 FOOD & DRUG L.J. 547 (2005).
71. Justice Brandeis articulated the classic statement of this theory in his dissent in New State Ice Co. v. Liebmann, 285 U.S. 262 (1932). He wrote, "It is one of the happy incidents of the federal system that a single courageous State may, if its citizens choose, serve as a laboratory; and try novel social and economic experiments without risk to the rest of the country."
72. see Ann Althouse, Vanguard States, Laggard States: Federalism and Constitutional Rights, 152 U. PA. L. REV. 1745, 1750-76 (2004).
73. See Susan Rose-Ackerman, Risk Taking and Reelection: Does Federalism Promote Innovation?, 9 J. LEGAL STUD. 593, 615 (1980) (stating that federalism produces only "weak effects" in promoting innovation).
74. See Michael Abramowicz, Speeding up the Crawl to the Top, 20 YALE J. ON REG. 139, 143 (2003) (suggesting that "the existence of state competition is not enough to guarantee an optimal level of innovation").
75. But see Roberta Romano, The States as a Laboratory: Legal Innovation and State Competition for Corporate Charters, 23 YALE J. ON REG. 209, 246 (2006) (arguing that states are, in fact, "effective laborator[ies]" in the "corporate chartering context").
76. See Edward L. Rubin & Malcolm Feeley, Federalism: Some Notes on a National Neurosis, 41 UCLA L. REV. 903, 925 (1994) ("[I]ndividual states will have no incentive to invest in experiments that involve any substance or political risk . . . ."). (Pubitemid 24794307)
77. Cf. FERC v. Mississippi, 456 U.S. 742, 788 (1982) (O'Connor, J., concurring in the judgment in part and dissenting in part) ("[S]tate innovation is no judicial myth.").
78. Barry Friedman, Valuing Federalism, 82 MINN. L. REV. 317, 398-99 (1997) ("'Innovation' might have been a better word choice for Justice Brandeis than 'experimentation,' saving us all a lot of bother."). (Pubitemid 127423564)
79. See, e.g., Sara Rosenbaum, Mothers and Children Last: The Oregon Medicaid Experiment, 18 AM. J.L. & MED. 97, 126 (1992) (concluding that Oregon's series of Medicaid initiatives is an experiment that "this nation cannot afford to conduct" because it "fails the basic test of civility").
80. Esther Duflo, Field Experiments in Development Economics 23 (Jan. 2006) (unpublished manuscript), available at http://econ-www.mit.edu/files/800.
81. See RONALD A. MCQUEEN & CHRISTINA KNUSSEN, INTRODUCTION TO RESEARCH METHODS AND STATISTICS IN PSYCHOLOGY 42-44 (2006) (discussing how to control for a "researcher effect").
82. See AUSTIN FLINT, A TREATISE ON THE PRINCIPLES AND PRACTICES OF MEDICINE 1019-20 (4th ed. 1873) (discussing his treatment of cases of articular rheumatism with "palliative measures only"). The control group received what Flint called a "placebo," or "placeboic remedy," of a "very largely diluted" "tincture of quassia."
83. See generally HENRY A. LANDSBERGER, HAWTHORNE REVISITED 1-27 (1958) (describing the experiments).
84. See Richard Pearson Gillespie, Manufacturing Knowledge: A History of the Hawthorne Experiments 165 (1985) (unpublished Ph.D. dissertation, University of Pennsylvania) (on file with Van Pelt Library, University of Pennsylvania) ("[W]hile production in all three test groups showed an improvement, it could not be correlated with those periods in which lighting was higher.").
85. Compare WILLIAM H. WHYTE, JR., THE ORGANIZATION MAN 34 (1956) (noting that "output did shoot ahead where conditions were changed, but so did output shoot ahead where no changes had been made")
86. Stephen R.G. Jones, Was There a Hawthorne Effect?, 98 AM. J. SOC. 451, 467 (1992) (finding "essentially no evidence of Hawthorne effects, either unconditionally or with allowance for direct effects of the experimental variables themselves").
87. See Esther Duflo et al., Using Randomization in Development Economics Research: A Toolkit ("The comparison group may feel offended to be a comparison group and react by also altering their behavior (for example, teachers in the comparison group for an evaluation may 'compete' with the treatment teachers or, on the contrary, decide to slack off ) ."), in 4 HANDBOOK OF DEVELOPMENTAL ECONOMICS, 3895, 3951 (T. Paul Schultz & John Strauss eds., 2008);
88. see also Allen C. Barrett & Doris A. White, How John Henry Effects Confound the Measurement of Self Esteem in Primary Prevention Programs for Drug Abuse in Middle Schools, J. ALCOHOL & DRUG EDUC., Spring 1991, at 87, 99 (providing an alleged example of a John Henry effect).
89. But cf. David A. Braunholtz et al., Are Randomized Clinical Trials Good for Us (in the Short Term)? Evidence for a "Trial Effect," 54 J. CLINICAL EPIDEMIOLOGY 217, 219 (2001) (noting that once patients are "exposed to the informed consent process," the Hawthorne effect becomes less relevant).
90. James J. Heckman & Jeffrey A. Smith, Assessing the Case for Social Experiments, J. ECON. PERSP., Spring 1995, at 85, 99;
91. see also James J. Heckman & Richard Robb, Jr., Alternative Methods for Evaluating the Impact of Interventions (considering the "problem of estimating the impact of interventions in the presence of selection decisions by agents"), in LONGITUDINAL ANALYSIS OF LABOR MARKET DATA 156, 158 (James J. Heckman & Burton Singer eds., 1985);
92. James J. Heckman & Jeffrey Smith, Assessing the Case for Randomized Evaluation of Social Programs (describing the "selection problem" as the fact that "persons who participate in a program are different from persons who do not participate in the sense that the mean outcomes of participants in the non participation state would be different from those of non-participants"), in MEASURING LABOUR MARKET MEASURES 35, 45-46 (Karsten Jensen & Per Kongshoj Madsen eds., 1993);
93. James J. Heckman & V. Joseph Hotz, Choosing Among Alternative Nonexperimental Methods for Estimating the Impact of Social Programs: The Case of Manpower Training, 84 J. AM. STAT. ASSOC. 862, 874 (1989) ("[S]imple specification tests eliminate the most unreliable and misleading estimators that give rise to the sensitivity problem . . . .").
94. See generally James J. Heckman, Randomization and Social Program Evaluation (discussing the "benefits and limitations of randomized social experiments" (emphasis omitted)), in EVALUATING WELFARE AND TRAINING PROGRAMS 201 (Charles F. Manski & Irwin Garfinkel eds., 1992).
95. see James J. Heckman, Micro Data, Heterogeneity, and the Evaluation of Public Policy: Nobel Lecture, 109 J. POL. ECON. 673, 690-704 (2001).
96. See, e.g., Laurence A. Cole et al., Letter to the Editor, False-Positive hCG Assay Results Leading to Unnecessary Surgery and Chemotherapy and Needless Occurrences of Diabetes and Coma, 45 CLINICAL CHEMISTRY 313, 314 (1999) (noting that false positives can produce negative results, such as unnecessary chemotherapy, which put one patient into a coma).
97. See, e.g., Nicholas Wade, Speed Reading of DNA May Help Cancer Treatment, N.Y. TIMES, Mar. 9, 2010, at D4 ("Researchers at Johns Hopkins University have developed a way to monitor the progress of a patient's cancer treatment using a new technique for rapidly sequencing, or decoding, large amounts of DNA.").
98. See Emergency Unemployment Compensation Act of 1991, Pub. L. No. 102-164, § 201(c)(3)(A), 105 Stat. 1049, 1056 (providing for the "random selection of eligible individuals for participation in the program and for inclusion in a control group"). This program was in effect from November 17, 1991, to February 5, 1994. See I.R.C. § 3304 note;
99. see also PAUL T. DECKER ET AL., MATHEMATICAL POLICY RESEARCH, INC., ASSISTING UNEMPLOYMENT INSURANCE CLAIMANTS: THE LONG-TERM IMPACTS OF THE JOB SEARCH ASSISTANCE DEMONSTRATION 7 (2000), available at http://www. workforcesecurity.doleta.gov/dmstree/op/op2k/op-02-00.pdf (discussing the program);
100. Marcus Stanley et al., Developing Skills: What We Know About the Impacts of American Employment and Training Programs on Employment, Earnings, and Educational Outcomes 40-42 (Oct. 1998) (unpublished manuscript), available at http://www.economics.harvard.edu/faculty/katz/files/stanley-katz-krueger-98.pdf (explaining that various programs in Minnesota, Nevada, New Jersey, South Carolina, and Washington were "conducted as random assignment experiments, making their results particularly reliable").
101. See James J. Heckman, Detecting Discrimination, J. ECON. PERSPECTIVES, Spring 1998, at 101, 101-03 (discussing the difficulty in "[e]stimating the extent and degree of discrimination" in the labor market).
102. Cf. Robert Slonim & Alvin E. Roth, Learning in High Stakes Ultimatum Games: An Experiment in the Slovak Republic, 66 ECONOMETRICA 569, 570-71 (1998) (reporting on an experiment in the Slovak Republic that had varying stakes to "increase the power of the experiment to detect differences in behavior due to differences in stakes").
103. See, e.g., Dean Karlan & Jonathan Zinman, Expanding Credit Access: Using Randomized Supply Decisions to Estimate the Impacts 4 (June 25, 2007) (unpublished manuscript), available at www.hks.harvard.edu/inequality/Seminar/ Papers/Karlan-071.pdf (reporting on an experiment in which loan officers were "randomly encouraged . . . to approve some marginal applications").
104. See, e.g., Terry M. Moe, Beyond the Free Market: The Structure of School Choice, 2008 BYU L. REV. 557, 571 (noting that current voucher programs are "too small and too new" to determine "whether the voucher amounts are large enough to . . . stimulate a sufficient supply-side response by schools over time").
105. See, e.g., Abhijit V. Banerjee et al., Remedying Education: Evidence from Two Randomized Experiments in India, 122 Q.J. ECON. 1235, 1245 (2007) (noting that in an educational experiment using randomization, "[d]ifferential attrition between the treatment and comparison groups could potentially bias the results").
106. Abhijit V. Banerjee et al., Remedying Education: Evidence from Two Randomized Experiments in India, 122 Q.J. ECON. 1235, (2007), Id.
107. See, e.g., Richard B. Miller & David W. Wright, Detecting and Correcting Attrition Bias in Longitudinal Family Research, 57 J. MARRIAGE & FAM. 921, 922 (1995) (describing the standard method of responding to this problem by incorporating a variable representing the probability of dropping out directly into the study
108. (citing James J. Heckman, Sample Selection Bias, 47 ECONOMETRICA 153 (1979)
109. James J. Heckman, The Common Structure of Statistical Methods of Truncation, Sample Selection and Limited Dependent Variables and a Simple Estimator for Such Models, 5 ANNALS ECON. & SOC. MEASUREMENT 475 (1976)).
110. See Alan B. Krueger, Experimental Estimates of Education Production Functions, 114 Q.J. ECON. 497, 505 (1999) (reporting lower attrition rates of students in smaller classes, with some exceptions).
111. See, e.g., Guido W. Imbens & Joshua D. Angrist, Identification and Estimation of Local Average Treatment Effects, 62 ECONOMETRICA 467, 472 (1994) (discussing the intent-to-treat approach).
112. Edward Miguel & Michael Kremer, Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities, 72 ECONOMETRICA 159, 208 (2004).
113. Edward Miguel & Michael Kremer, Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities, 72 ECONOMETRICA 159, (2004), Id.
114. See generally Charles M. Tiebout, A Pure Theory of Local Expenditures, 64 J. POL. ECON. 416 (1956) (providing the seminal account of the effects of citizen mobility).
115. Ian Ayres and Matt Roche, President, Omniture ( June 14, 2007);
116. Paat Rusmevichientong & David Williamson, An Adaptive Algorithm for Selecting Profitable Keywords for Search-Based Advertising Services (same), in PROCEEDINGS OF THE 7TH ACM CONFERENCE ON ELECTRONIC COMMERCE 260, 260 ( Joan Feigenbaum et al. eds., 2006).
117. see generally Wikipedia, http://en.wikipedia.org/wiki/AdWords (last visited Jan. 15, 2011), and text accompanying infra note 192.
118. See Yair Listokin, Learning Through Policy Variation, 118 YALE L.J. 480, 513-14 (2008) (arguing that, in many cases, the expected effect of policy is less important than the variance of the expected effects, but other things being equal, higher-expected value policies are superior to lower-expected value policies).
119. See Alice M. Rivlin & P. Michael Timpane, Introduction and Summary to ETHICAL AND LEGAL ISSUES OF SOCIAL EXPERIMENTATION 1, 1 (Alice M. Rivlin & P. Michael Timpane eds., 1975) (introducing a collection of papers discussing the "new ethical and legal issues that need to be carefully examined" when testing new social policies).
120. See Kathryn A. Tuthill, Commentary, Human Experimentation: Protecting Patient Autonomy Through Informed Consent, 18 J. LEGAL MED. 221, 221 (1997) ("The doctrine of informed consent requires that a physician inform a patient or research subject of the benefits, risks, and alternatives to medical treatment or experimental procedures before such treatment is administered.").
121. 125 Schloendorff v. Soc'y of N.Y. Hosp., 105 N.E. 92, 93 (N.Y. 1914)
122. Bing v. Thunig, 143 N.E.2d 3 (N.Y. 1957).
123. See Peter G. Brown, Informed Consent in Social Experimentation: Some Cautionary Notes ("'[E]very man that hath any possession or enjoyment of any part of the dominions of any government doth thereby give his tacit consent . . . .'"
124. (quoting John Locke, An Essay Concerning the True Original, Extent and End of Civil Government, in THE ENGLISH PHILOSOPHERS FROM BACON TO MILL 403, 452 (Edwin A. Burtt ed., 1939)))
125. ETHICAL AND LEGAL ISSUES OF SOCIAL EXPERIMENTATION, supra note 122, at 79, 96.
126. See IMMANUEL KANT, GROUNDWORK OF THE METAPHYSIC OF MORALS 96 (H.J. Paton trans., Harper Torchbooks 1964) (1785) ("Act in such a way that you always treat humanity, whether in your own person or in the person of any other, never simply as a means, but always at the same time as an end." (emphasis omitted)).
127. See generally, e.g., Jeanne C. Fromer, Patent Disclosure, 94 IOWA L. REV. 539 (2009) (discussing the patent system's goal of encouraging disclosure of inventions).
128. See supra Part I.
129. Robert M. Veatch, Ethical Principles in Medical Experimentation, in ETHICAL AND LEGAL ISSUES OF SOCIAL EXPERIMENTATION, supra note 122, at 21, 37.
130. Eugene Volokh, Essay, Medical Self-Defense, Prohibited Experimental Therapies, and Payment for Organs, 120 HARV. L. REV. 1813, 1815 (2007);
131. see also Abigail Alliance for Better Access to Developmental Drugs v. Von Eschenbach, 495 F.3d 695, 713 (D.C. Cir. 2007) (en banc) (finding no constitutional right to have access to "investigational drugs").
132. See, e.g., Michael D. Greenberg, AIDS, Experimental Drug Approval, and the FDA New Drug Screening Process, 3 N.Y.U. J. LEGIS. & PUB. POL'Y 295, 305 (2000) (describing Phase II as "the first stage of testing at which drug efficacy becomes a formal consideration").
133. See, e.g., Denise C. Gottfredson & M. Lyn Exum, Research Note, The Baltimore City Drug Treatment Court: One-Year Results From a Randomized Study, 39 J. RES. CRIME & DELINQ. 337, 343 (2002) (detailing a program in which 235 eligible clients were randomly assigned either to drug court or to standard criminal processes).
134. See, e.g., Denise Irene Arnold, Lottery Prize Is Affordable Homes, N.Y. TIMES, Feb. 7, 1988, § 21 (Long Island Weekly ed.), at 12 (discussing a local housing lottery on Long Island).
135. See, e.g., 8 U.S.C. § 1153(e)(2) (2006) (providing for distribution of visas for diversity immigrants "strictly in a random order").
136. Maurice Rosenberg, The Impact of Procedure-Impact Studies in the Administration of Justice, LAW & CONTEMP. PROBS., Summer 1988, at 13, 16. 142 See supra Part II.
137. See, e.g., Thomas W. Hazlett & Robert J. Michaels, The Cost of Rent- Seeking: Evidence from Cellular Telephone License Lotteries, 59 S. ECON. J. 425, 425 (1993) (analyzing government lotteries for cellular-telephone licenses that led to over 320,000 applications).
138. Williamson v. Lee Optical Co., 348 U.S. 483 (1955).
139. Aguayo v. Richardson, 473 F.2d 1090, 1109-10 (2d Cir. 1973). One commentator has criticized the court for not indicating that its decision would be valid only for as long as the "value of the program remained uncertain."
140. Note, Reforming the One Step at a Time Justification in Equal Protection Cases, 90 YALE L.J. 1777, 1783 (1981).
141. See, e.g., Delaware v. Prouse, 440 U.S. 648, 663 (1979) (finding random stops of vehicles to check drivers' licenses and registrations inconsistent with the Fourth Amendment).
142. RONALD DWORKIN, LAW'S EMPIRE 178-84 (1986).
143. 408 U.S. 238 (1972) (per curiam).
144. See DAVID C. BALDUS ET AL., EQUAL JUSTICE AND THE DEATH PENALTY 133 (1990) (concluding from the data in their empirical study that "excessive sentences most likely result from suspect and illegitimate factors, such as the race and socioeconomic status of the defendant or the victim, or, perhaps, other idiosyncratic factors").
145. 523 U.S. 272, 289 (1998) (O'Connor, J., concurring).
146. See In re Friess, 457 N.Y.S.2d 33, 34-35 (1982) (denying severance of the charge against Judge Friess for coin-flipping from a charge that he resolved a different dispute "by submitting it to a show of hands by spectators in the courtroom").
147. See Judicial Inquiry & Review Comm'n v. Shull, 651 S.E.2d 648, 658 (Va. 2007) (removing Judge Shull for several instances of misconduct, including coin-flipping as well as ordering a victim of domestic abuse to show wounds on her thigh in court and making an "improper ex parte telephone call");
148. see also Gary Slapper, Weird Cases: Justice by Coin-Toss, TIMES ONLINE (London), Nov. 16, 2007, http://business.timesonline.co.uk/tol/business/law/ article2882090.ece (describing the Friess and Shull cases).
149. Shull, 651 S.E.2d at 652.
150. Adam Liptak, Lawyers Won't End Squabble, So Judge Turns to Child's Play, N.Y. TIMES, June 9, 2006, at A19. Liptak further reported that "[c]hildish lawyers are commonplace, but the use of children's games to resolve litigation disputes is apparently a new development."
151. Adam Liptak, Lawyers Won't End Squabble, So Judge Turns to Child's Play, N.Y. TIMES, 2006, Id.
152. see also Jeralyn E. Merritt, The "Rock, Paper, Scissors" Judge, TALKLEFT (June 9, 2006, 5:06:39 AM), http://www.talkleft.com/story/2006/06/09/ 305/45461 (defending Judge Presnell's acumen in the wake of the rock-paper-scissors story).
153. See Ian Ayres & Eric Talley, Solomonic Bargaining: Dividing a Legal Entitlement to Facilitate Coasean Trade, 104 YALE L.J. 1027, 1073-78 (1995) (demonstrating that awarding property rights probabilistically is efficient given that all parties have knowledge of the court's probability distribution).
154. see also Peter Cramton et al., Dissolving A Partnership Efficiently, 55 ECONOMETRICA 615, 626 (1987) (arguing that a "simple bidding game" can "achieve" an "ex post efficient allocation" of assets when dissolving a partnership).
155. See JON ELSTER, SOLOMONIC JUDGMENTS: STUDIES IN THE LIMITATION OF RATIONALITY 170-72 (1989) (discussing arguments for and against coin-flipping).
156. see also Adam M. Samaha, Randomization in Adjudication, 51 WM. & MARY L. REV. 1, 20-21, 29-30 (2009) (discussing Elster's argument in more detail).
157. See Steven Shavell, Economic Analysis of Public Law Enforcement and Criminal Law 2 (Nat'l Bureau of Econ. Research, Working Paper No. 9698, 2003), available at http://www.nber.org/papers/w9698.pdf (arguing that if parties are risk averse, then it may be "socially beneficial" to impose lesser sanctions than if the parties are risk neutral).
158. Cf. David Lewis, The Punishment That Leaves Something to Chance, 18 PHIL. & PUB. AFF. 53, 58-62 (1989) (defending punishments where the severity is randomized, en route to justifying harsher penalties for those who, by coincidence, cause more harm);
159. Florynce Kennedy, Letter to the Editor, For Whom the Coin Is Tossed, N.Y. TIMES, Feb. 13, 1982, at 24 (comparing Judge Friess's coin toss to other intrinsic uncertainties in the judicial system).
160. see John M. Darley, On the Unlikely Prospect of Reducing Crime Rates by Increasing the Severity of Prison Sentences, 13 J.L. & POL'Y 189 (2005).
161. Cf. Jeffrey R. Kling et al., Experimental Analysis of Neighborhood Effects, 75 ECONOMETRICA 83, 84 (2007) (analyzing a "randomized experiment in which some families living in high-poverty U.S. housing projects were offered . . . housing vouchers . . . while others were not").
162. See Gordon C.S. Smith & Jill P. Pell, Parachute Use to Prevent Death and Major Trauma Related to Gravitational Challenge: Systematic Review of Randomised Controlled Trials, 327 BMJ 1459, 1460 (2003) (noting ironically that "[t]he basis for parachute use is purely observational, and its apparent efficacy could potentially be explained by a 'healthy cohort' effect").
163. See Sarah J.L. Edwards et al., The Ethics of Randomised Controlled Trials from the Perspectives of Patients, the Public, and Health Care Professionals, 317 BMJ 1209, 1209 (1998) ("The scientific rationale for conducting a trial rests in collective equipoise, which means that the medical community as a whole is genuinely uncertain over which treatment is best.").
164. Martin L. Weitzman, Optimal Search for the Best Alternative, 47 ECONOMETRICA 641, 649 (1979) ("The purpose of the model formulated in this paper is to sharply characterize [the] optimal search among alternative sources with different characteristics.").
165. See Richard J. Lilford & Jennifer Jackson, Equipoise and the Ethics of Randomization, 88 J. ROYAL SOC'Y MED. 552, 554-55 (1995) (discussing the implication of partial information on informed consent).
166. But see Lester V. Manderscheid, Significance Levels-0.05, 0.01, or ?, 47 J. FARM ECON. 1381, 1381 (1965) (urging that the level of statistical significance employed in a particular experiment is "not arbitrary but rather is, or at least should be, a deliberate choice").
167. See, e.g., Sharona Hoffman, The Use of Placebos in Clinical Trials: Responsible Research or Unethical Practice?, 33 CONN. L. REV. 449, 459 (2001) (distinguishing these trials from true placebo trials).
168. See RICHARD H. THALER & CASS R. SUNSTEIN, NUDGE 160-62 (2008) (explaining the main features of Medicare Part D).
169. see Marianne Bertrand et al., How Much Should We Trust Differences-in-Differences Estimates?, 119 Q.J. ECON. 249, 249-61 (2004).
170. See Richard Craswell, Passing on the Costs of Legal Rules: Efficiency and Distribution in Buyer-Seller Relationships, 43 STAN. L. REV. 361, 372-85 (1991) (discussing the market impact of efficient and inefficient mandates);
171. see also Christine Jolls, Accommodation Mandates, 53 STAN. L. REV. 223, 230-72 (2000) (considering the efficiency of mandates intended to accommodate the special needs of population subgroups). (Pubitemid 33648727)
172. See Donald D. McNeil, Jr., Circumcision's Anti-AIDS Effect Found Greater than First Thought, N.Y. TIMES, Feb. 23, 2007, at A3 (noting that "two clinical trials were stopped . . . because the results were so clear").
173. See, e.g., Ying Kuen Cheng et al., Continuous Bayesian Adaptive Randomization Based on Event Times with Covariates, 25 STAT. MED. 55, 56 (2006) ("[O]utcome-adaptive randomization . . . uses the data from patients treated previously in the trial to unbalance the randomization probabilities in favour of the treatment . . . observed to have comparatively superior outcomes. [It] provides a compromise between ethical concerns and the scientific goal of obtaining unbiased treatment comparisons.").
174. See, e.g., 5 U.S.C. § 553 (2006) (setting forth procedural requirements for notice-and-comment rulemaking by administrative agencies).
175. See, e.g., Chevron U.S.A., Inc. v. Nat'l Res. Def. Council, Inc., 467 U.S. 837, 842- 45 (1984) (setting forth the contemporary standard of judicial review for evaluating agency interpretations of congressional mandates).
176. See, e.g., 5 U.S.C. § 706(2)(A) (requiring courts to "hold unlawful and set aside agency action . . . found to be . . . arbitrary, capricious, an abuse of discretion, or otherwise not in accordance with law").
177. 5 U.S.C., Id. § 553(b)
178. See, e.g., Portland Cement Ass'n v. Ruckelshaus, 486 F.2d 375, 402 (D.C. Cir. 1973) (remanding a case noting that the "record reveals a lack of an adequate opportunity . . . to comment on the proposed standards, due to the absence of disclosure of the detailed findings and procedures of the tests").
179. U.S.C. § 553(c) (requiring an agency to provide opportunity for comment by interested persons and to issue a "concise general statement of basis and purpose"). "Concise" and "general" are sometimes interpreted to mean "detailed" and "specific."
180. See, e.g., Auto. Parts & Accessories Ass'n v. Boyd, 407 F.2d 330, 338 (D.C. Cir. 1968) (warning against an "overly literal" interpretation of these words).
181. See, e.g., Greater Boston Television Corp. v. FCC, 444 F.2d 841, 851 (D.C. Cir. 1970) ("Its supervisory function calls on the court to intervene . . . if the court becomes aware . . . that the agency has not really taken a 'hard look' at the salient problems, and has not genuinely engaged in reasoned decision-making." (footnote omitted)).
182. See, e.g., Motor Vehicle Mfrs. Ass'n v. State Farm Mut. Auto. Ins. Co., 463 U.S. 29, 43 (1983) ("The scope of review under the 'arbitrary and capricious' standard is narrow and a court is not to substitute its judgment for that of the agency.").
183. See, e.g., Solicitation for Proposals for the Demonstration Project for Disease Management for Severely Chronically Ill Medicare Beneficiaries with Congestive Heart Failure, Diabetes, and Coronary Heart Disease, 67 Fed. Reg. 8267, 8269 (Feb. 22, 2002) (soliciting "applications for demonstration projects that use disease management, along with coverage of prescription drugs, to improve the quality of services furnished to specific beneficiaries and to manage expenditures under Parts A and B of the Medicare program").
184. 5 U.S.C. § 553(b)(B). But the courts have found that the temporary nature of a rule is not enough to escape notice-and-comment.
185. See, e.g., Tenn. Gas Pipeline Co. v. FERC, 969 F.2d 1141, 1145 (D.C. Cir. 1992) ("'[T]he limited nature of the rule cannot in itself justify a failure to follow notice and comment procedures.'"
186. (quoting Council of the S. Mountains, Inc. v. Donovan, 653 F.2d 573, 582 (D.C. Cir. 1981))).
187. But see Juan J. Lavilla, The Good Cause Exemption to Notice and Comment Rulemaking Requirements Under the Administrative Procedure Act, 3 ADMIN. L.J. 317, 377-78 (1989) (suggesting that courts treat a rule's temporary nature as a factor in determining whether it falls under the "good cause" exception in § 553(b)(B)).
188. Thomas O. McGarity, Some Thoughts on "Deossifying" the Rulemaking Process, 41 DUKE L.J. 1385, 1438 (1992);
189. see also Richard J. Pierce, Jr., Seven Ways to Deossify Agency Rulemaking, 47 ADMIN. L. REV. 59, 61 (1995) (observing that even though agencies should try to keep up with scientific and technological advances, "agencies rarely amend rules because the amendment process is as daunting as the process of promulgating a rule").
190. Richard L. Revesz, Environmental Regulation, Ideology, and the D.C. Circuit, 83 VA. L. REV. 1717, 1770 (1997). (Pubitemid 127445656)
191. See Exec. Order No. 12,866, 3 C.F.R. 638, 639 (1994)
192. 5 U.S.C. 601 (2006) (permitting regulation only where benefits exceed costs).
193. See Planned Parenthood of Se. Pa. v. Casey, 505 U.S. 833, 898 (1992) (striking down a Pennsylvania spousal-notification law). But a state might experiment with offering couples at the time of marriage the option of contracting for spousal notification.
194. See Andrew Blair-Stanek, Comment, Default and Choices in the Marriage Contract: How to Increase Autonomy, Encourage Discussion, and Circumvent Constitutional Constraints, 24 TOURO L. REV. 31, 49-51 (2008).
195. Cass R. Sunstein & Adrian Vermeule, Is Capital Punishment Morally Required? Acts, Omissions, and Life-Life Tradeoffs, 58 STAN. L. REV. 703, 705 (2005) (arguing that statistical evidence on the deterrent effect of the death penalty, if it exists, may make the death penalty a moral imperative);
196. Cass R. Sunstein & Justin Wolfers, Op-Ed., A Death Penalty Puzzle: The Murky Evidence for and Against Deterrence, WASH. POST, June 30, 2008, at A11 (calling for public debate on the death penalty to avoid distortions based on "misunderstanding . . . what the evidence actually shows").
197. One prominent clinical-trial company has run over "3,200 trials in some 90 countries" since 2000. QUINTILES TRANSNATIONAL CORP., THE VALUE OF CONTRACT RESEARCH ORGANIZATIONS 2, http://www.quintiles.com/elements/media/ white-papers/value-cros.pdf (last visited Jan. 15, 2011).
198. See supra notes 43-45 and accompanying text (discussing an example of policymakers misunderstanding certain data).
199. See, e.g., Michael Greenstone et al., Mandated Disclosure, Stock Returns, and the 1964 Securities Acts Amendments, 121 Q.J. ECON. 399, 400 (2006) (noting that "the 1964 [Securities Acts] Amendments provide a compelling setting for evaluating the consequences of mandatory disclosure regulations");
200. Paul G. Mahoney, The Political Economy of the Securities Act of 1933, 30 J. LEGAL STUD. 1, 1 (2001) (describing the 1933 Securities Act as "a severe testing ground for rent-seeking theories of economic regulation"); (Pubitemid 33647499)
201. Allen Ferrell, Mandated Disclosure and Stock Returns: Evidence from the Over-the-Counter Market 6-33 (Harvard John M. Olin Center for Law, Economics, and Business, Discussion Paper No. 453, 2003), available at http://ssrn.com/ abstract=500123 (using the 1964 Amendments to study the effect of imposing mandated disclosure on the informational efficiency of the OTC market).
202. See, e.g., Stephen E. Christophe et al., Short-Selling Prior to Earnings Announcements, 59 J. FIN. 1845, 1847 (2004) ("[R]egulators should consider requiring markets to make more extensive and timely disclosures of short-selling activity.");
203. Ian Ramsay, Short Selling: Further Issues, 21 SEC. REG. L.J. 214, 218 (1993) (arguing that a shortselling disclosure rule could "impede market efficiency" and may not operate quickly enough to allow information to "disseminate into the market").
204. Short Sales, 69 Fed. Reg. 48,008, 48,012 (Aug. 6, 2004);
205. see also Regulation of Short Sales, 17 C.F.R. § 242.200-.204 (2010);
206. OFFICE OF ECON. ANALYSIS, U.S. SEC. & EXCH. COMM'N, ECONOMIC ANALYSIS OF THE SHORT SALE PRICE RESTRICTIONS UNDER THE REGULATION SHO PILOT 3-4 (2007), available at http://www.sec.gov/news/studies/2007/regshopilot020607.pdf (describing the restrictions in place before Rule 202T was enacted and the purpose of the rule itself).
207. See OFFICE OF ECON. ANALYSIS, supra note 214, at 6-8 ("Finance theory predicts that under certain conditions, constraints on short selling may cause securities to be misvalued by the market, particularly when investors have highly divergent opinions about the stock.").
208. See Emilios Avgouleas, A New Framework for the Global Regulation of Short Sales: Why Prohibition Is Inefficient and Disclosure Insufficient, 115 STAN. J.L. BUS. & FIN. 376, 380-83 (2010) (describing and countering the market-abuse argument offered by the government for short-selling restrictions).
209. Evidence-Based Medicine, SA HEALTHINFO, http://www.sahealthinfo.org/ evidence/s.htm (last visited Jan. 15, 2011). If trading volume influences the effect of short-sale restrictions, then the pilot design insured that the exempt group of stocks and the control group were similar by performing separate selections for each group of three stocks with similar daily trading volumes.
210. See OFFICE OF ECON. ANALYSIS, supra note 214, at 4 ("The Pilot went into effect on May 2, 2005, and was scheduled to end on April 28, 2005, but [was] extended to August 6, 2007, to allow the Commission to consider potential rulemaking after evaluating the results of the Pilot." (footnote omitted)).
211. (quoting Order Suspending the Operation of Short Sale Price Provisions for Designated Securities and Time Periods, Exchange Act Release No. 50,104, 69 Fed. Reg. 48,032, 48,032 (Aug. 6, 2004)).
212. Sarbanes-Oxley Act of 2002, Pub. L. No. 107-204, 116 Stat. 745 (codified as amended in scattered sections of 11, 15, 18, 28 and 29 U.S.C.).
213. 15 U.S.C. § 7241 (2006).
214. 15 U.S.C. § 7262. The internal-controls requirements obligated companies to set up elaborate mechanisms for detecting malfeasance within the company or disclose the absence of such controls.
215. See, e.g., Peter C. Kostant, From Lapdog to Watchdog: Sarbanes-Oxley Section 307 and a New Role for Corporate Lawyers, 52 N.Y.L. SCH. L. REV. 535, 536 n.2 (2008) (collecting sources criticizing the Act).
216. See id. at 536-37 (defending, in particular, section 307).
217. See, e.g., Ellen Engel et al., The Sarbanes-Oxley Act and Firms' Going-Private Decisions, 44 J. ACCT. & ECON. 116, 143 (2007) ("[S]maller firms with high inside ownership experience higher going-private announcement returns in the post-SOX period compared to the pre-SOX period.");
218. Peter Iliev, The Effect of SOX Section 404: Costs, Earnings Quality, and Stock Prices, 65 J. FIN. 1163, 1193 (2010) (finding that for small firms the cost of Sarbanes-Oxley section 404 outweighs the benefits);
219. Roberta Romano, The Sarbanes-Oxley Act and the Making of Quack Corporate Governance, 114 YALE L.J. 1521, 1529 (2005) (arguing for the removal of mandatory corporate governance provisions from Sarbanes-Oxley);
220. Ivy Xiying Zhang, Economic Consequences of the Sarbanes-Oxley Act of 2002, 44 J. ACCT. & ECON. 74, 110 (2007) ("[T]he cumulative abnormal returns of U.S. firms and foreign firms complying with SOX around key SOX events are negative and statistically significant.").
221. See generally Christian Leuz, Was the Sarbanes-Oxley Act of 2002 Really this Costly? A Discussion of Evidence from Event Returns and Going-Private Decisions, 44 J. ACCT. & ECON. 146 (2007) (discussing and contributing to this academic debate).
222. See Romano, supra note 237, at 1529 (arguing that "the corporate governance provisions of SOX should be stripped of their mandatory force and rendered optional for registrants").
223. See Irwin H. Steinhorn & William M. Lewis, Corporate Compliance Under the Regulations Implementing Sarbanes-Oxley, 60 CONSUMER FIN. L.Q. REP. 30, 30 (2006) (noting that "'[t]he strength of the U.S. financial markets depends on investor confidence'"
224. (quoting Disclosure Required by Sections 404, 406, and 407 of the Sarbanes-Oxley Act of 2002, Securities Act Release No. 8177, 68 Fed. Reg. 5110, 5110 ( Jan. 31, 2003))).
225. see generally Lisa E. Bolton & Americus Reed II, Sticky Priors: The Perseverance of Identity Effects on Judgment, 41 J. MARKETING RES. 397 (2004).
226. See N. Gregory Mankiw, The Optimal Collection of Seigniorage: Theory and Evidence, 20 J. MONETARY ECON. 327, 332 (2004) (suggesting that an increase in tax revenue is normally associated with higher taxes).
227. See Paul Krugman, The Laffer Test (Somewhat Wonkish), THE CONSCIENCE OF A LIBERAL, N.Y.TIMES.COM BLOG (Aug. 10, 2010, 10:37 AM), http:// krugman.blogs.nytimes.com/2010/08/10/the-laffer-test-somewhat-wonkish (explaining that even very high marginal tax rates are not likely to reduce work much, though they do increase incentives to evade taxes).
228. See, e.g., Glenn Kessler, Now President Faces Tax Cut Test: Loss of Revenue Means Bush Needs to Slow Spending, WASH. POST, Feb. 11, 2001, at A5 (suggesting that a tax cut must be accompanied by a reduction in spending);
229. David E. Rosenbaum, Name That Tune About Tax Cuts, N.Y. TIMES, May 18, 2003, at BU4 (reporting Congress's Joint Committee on Taxation's finding that the "revenue feedback" from proposed tax cuts would be approximately three to twenty-three percent of the cuts over ten years).
230. See, e.g., Daniel J. Mitchell, Lowering Marginal Tax Rates: The Key to Pro-Growth Tax Relief, BACKGROUNDER (Heritage Foundation, Washington, D.C.), May 22, 2001, at 1, 5-7 (using the Kennedy tax cuts and the Reagan tax cuts to argue that lower taxes stimulate economic growth, leading to higher revenue).
231. See Basil Dalamagas, The Effects of Tax Rate Changes on Output and Government Deficits, 10 APPLIED ECON. LETTERS 97, 101 (2003) (concluding that the effects of lowering the tax rate vary based on "the kind of effective tax rates which fiscal authorities choose to decrease and the ratio of factor income tax rate to consumption tax rate").
232. See, e.g., David M. Herszenhorn, Bush and House in Accord for $150 Billion Stimulus, N.Y. TIMES, Jan. 25, 2008, at A1 (describing the 2008 tax rebate passed to boost the ailing economy).
233. See supra notes 248, 249, and accompanying text (discussing different theories on the relationship between tax revenues and tax rates).
234. see generally James Alm et al., Policy Watch: The Marriage Penalty, 13 J. ECON. PERSP. 193 (1999).
235. Cf. Alan B. Krueger & Jörn-Steffen Pischke, The Effect of Social Security on Labor Supply: A Cohort Analysis of the Notch Generation, 10 J. LABOR ECON. 412, 434 (1992) (concluding that previous estimates of the wealth effect of Social Security-i.e., of the extent to which the program's benefits motivated people to exit the labor force- were exaggerated).
236. See supra notes 240-41 and accompanying text (discussing why the experimental period needs to be relatively long).
237. See Statewide Employment Laws and Policies, HUMAN RIGHTS CAMPAIGN (July 26, 2010), http://www.hrc.org/documents/Employment-Laws-and-Policies.pdf (listing the states).
238. See, e.g., Employment Non-Discrimination Act of 2009, H.R. 3017, 111th Cong. (1st Sess. 2009).
239. 3685, 110th Cong. (1st Sess. 2007).
240. See THE GALLUP POLL: PUBLIC OPINION 2004 (Alec M. Gallup & Frank Newport eds., 2006) ("Americans overwhelmingly support 'equal rights in terms of job opportunities' for gay men and women.");
241. Elizabeth Mehren, Acceptance of Gays Rises Among New Generation, L.A. TIMES, Apr. 11, 2004, at A1 ("72% favor laws to protect homosexuals from job discrimination . . . .");
242. John Newsome, Employment Non-Discrimination Act (ENDA) Vote Tests Our Values: An Incrementalist Law Is a Blunder, S.F. CHRON., Nov. 7, 2007, at B11 ("[A] 2006 Gallup Poll reveals that 89 percent of respondents favor equal opportunities for gay people.");
243. GAY & LESBIAN ALLIANCE AGAINST DEFFAMATION, MEDIA REFERENCE GUIDE 29 (8th ed. 2010), available at http://www.glaad.org/document.doc?id=99 (noting that a 2005 Gallup poll showed 87% support for equal job opportunities for "homosexuals" but 90% support for equal job opportunities for-using different language-"gays and lesbians").
244. See, e.g., OFFICE OF MGMT. & BUDGET, EXEC. OFFICE OF THE PRESIDENT, STATEMENT OF ADMINISTRATION POLICY: H.R. 3685-THE EMPLOYMENT NON-DISCRIMINATION ACT (2007) (stating the policy position of former President George W. Bush, who opposed ENDA on "constitutional and policy grounds").
245. See Letter from Barry R. Bedrick, Assoc. Gen. Counsel, U.S. Gen. Accounting Office, to the Honorable James M. Jeffords, Chairman, Comm. on Health, Educ., Labor and Pensions 2 (Apr. 28, 2000), available at http://archive.gao.gov/f0502/575711.pdf (finding "no indication that these laws . . . generated a significant amount of litigation").
246. The General Accounting Office became the Government Accountability Office in 2004. Our Name, U.S. GOV'T ACCOUNTABILITY OFFICE, http://www.gao.gov/about/ namechange.html (last visited Jan. 15, 2011).
247. Ian Ayres & Jennifer Gerarda Brown, Mark(et)ing Nondiscrimination: Privatizing ENDA with a Certification Mark, 104 MICH. L. REV. 1639, 1670 (2006).
248. U.S. CENSUS BUREAU, NUMBER OF FIRMS, NUMBER OF ESTABLISHMENTS, EMPLOYMENT, ANNUAL PAYROLL, AND ESTIMATED RECEIPTS BY ENTERPRISE EMPLOYMENT SIZE FOR THE UNITED STATES AND STATES, TOTALS: 2007, available at http://www2.census.gov/econ/susb/data/2007/us-state-totals-2007.xls (last visited Jan. 15, 2011) (reporting 6,049,655 U.S. firms in 2007).
249. See generally Ian Ayres & Jennifer Gerarda Brown, Privatizing Employment Protections, 49 ARIZ. L. REV. 587 (2007). The fair employment license, however, falls short of ENDA protections on a few dimensions.
250. See Ayres & Brown, supra note 266, at 1655 (noting that the license would not be enforced by governmental agencies, and private suits could not be brought in federal court).
251. Ian Ayres, Menus Matter, 73 U. CHI. L. REV. 3, 8 (2006).
252. See Yair Listokin, What Do Corporate Default Rules and Menus Do? An Empirical Examination 40 (Yale Law Sch. John M. Olin Ctr. for Studies in Law, Econ., & Pub. Policy, Research Paper No. 335, 2006), available at http://ssrn.com/abstract=924578 ("The presence or absence of corporate menus leads to large differences in outcomes, as do differences in default rules.").
253. See James Heckman et al., Substitution and Dropout Bias in Social Experiments: A Study of an Influential Social Experiment, 115 Q. J. ECON. 651, 655 (2000) ("When good substitutes for an evaluated program are available, the effect of the program will be small even if the effect of training is large.").