The "hidden judiciary": An empirical examination of executive branch justice

Duke Law Journal

Volumn 58, Issue 7, 2009, Pages 1477-1530

  Guthrie, Chris ^a   Rachlinski, Jeffrey J ^b   Wistrich, Andrew J ^c  

^a VANDERBILT UNIVERSITY LAW SCHOOL   (United States)

^b MIT Sloan School of Management   (United States)

^c United States District Court for the Central District of California ^*  (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 68049112364     PISSN: 00127086     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (219)

1. See Thomas C. Mans, Selecting the 'Hidden Judiciary': How the Merit Process Works in Choosing Administrative Law Judges (Part I), 63 Judicature 60, 60 (1979) (coining the term "hidden judiciary").
2. But see Ronnie A. Yoder, The Role of the Administrative Law Judge, 22 J. Nat'l Ass'n Admin. L. Judges 321, 323 (2002) ("We [administrative law judges] used to be referred to as the hidden judiciary; but you do not see that phraseology much any more.").
3. U.S. Dep't of Labor, Occupational Employment and Wages, 2007, at 9 tbl.1 (Bureau of Labor Statistics, News Release No. 08-0620, rev. ed. 2009), available at http://www.bls.gov/news.release/pdf/ocwage.pdf (reporting that 14,100 "administrative law judges, adjudicators, and hearing officers" were employed in the United States in 2007).
4. Charles H. Koch, Jr., Policymaking by the Administrative Judiciary, 56 ALA. L. REV. 693, 693 (2005).
5. Daniel. L. Skoler, The Administrative Law Judiciary: Change, Challenge, and Choices, 462 Annals AM. Acad. POL. & SOC. SCI. 34, 36 (1982).
6. See Judith Resnik, Migrating, Morphing, and Vanishing: The Empirical and Normative Puzzles of Declining Trial Rates in Courts, 1 J. Empirical Legal Stud. 783, 799 (2004) (reporting that while federal judges conducted a total of about 85,000 trials (defined as "contested hearings at which evidence is presented") during 2001, just four federal administrative agencies (the SSA, the INS, the Board of Veterans Appeals, and the EEOC) conducted more than 720,000 trials that year).
7. Paul R. Verkuil, Reflections upon the Federal Administrative Judiciary, 39 UCLA L. REV. 1341, 1343 (1992) ("ALJs probably decide more 'cases' each year than do their federal judicial counterparts.").
8. Yoder, supra note 1, at 323.
9. See, e.g., Jeffrey J. Rachlinski, Chris Guthrie & Andrew J. Wistrich, Inside the Bankruptcy Judge's Mind, 86 B.U. L. REV. 1227, 1228 (2006) (reviewing the demand for and the trends toward judicial specialization).
10. See generally Chris Guthrie, Jeffrey J. Rachlinski & Andrew J. Wistrich, Blinking on the Bench: How Judges Decide Cases, 93 Cornell L. Rev. 1 (2007) [hereinafter Guthrie et al., Blinking] (proposing an intuitive-override model of decisionmaking and presenting relevant experimental evidence)
11. Chris Guthrie, Jeffrey J. Rachlinski & Andrew J. Wistrich, Inside the Judicial Mind, 86 Cornell L. Rev. 777 (2001) [hereinafter Guthrie et al., Judicial Mind] (reporting experimental evidence showing that judges are susceptible to misleading heuristics and biases when making judgments)
12. Andrew J. Wistrich, Chris Guthrie & Jeffrey J. Rachlinski, Can Judges Ignore Inadmissible Information? The Difficulty of Deliberately Disregarding, 153 U. Pa. L. Rev. 1251 (2005) (reporting experimental evidence showing that judges have difficulty deliberately disregarding relevant but inadmissible evidence when making merits decisions).
13. Amos Tversky & Daniel Kahneman, Judgment Under Uncertainty: Heuristics and Biases, 185 Science 1124, 1124 (1974) (introducing the concept of heuristics and biases)
14. See Guthrie et al., Blinking, supra note 8, at 6.
15. Keith E. Stanovich & Richard F. West, Individual Differences in Reasoning: Implications for the Rationality Debate?, in Heuristics and Biases: the Psychology of Intuitive Judgment 421, 436 (Thomas Gilovich, Dale Griffin & Daniel Kahneman eds., 2002) ("Although there are several two-process theories of reasoning that differ somewhat in their details, all agree on the general features of the two systems which, for simplicity, we label System 1 and System 2.").
16. See Stanovich & West, supra note 10, at 436.
17. Daniel Kahneman & Shane Frederick, Representativeness Revisited: Attribute Substitution in Intuitive Judgment, in Heuristics and Biases: The Psychology of Intuitive Judgment, supra note 10, at 49, 49
18. See Joseph C. Hutcheson, Jr., The Judgment Intuitive: The Function of the "Hunch" in Judicial Decision, 14 Cornell L.Q. 274, 278 (1929).
19. See Stanovich & West, supra note 10, at 436 ("System 2 conjoins the various characteristics associated with controlled processing.").
20. Shane Frederick, Cognitive Reflection and Decision Making, 19 J. ECON. PERSP. 25, 26 (2005).
21. Kahneman & Frederick, supra note 13, at 49.
22. Id. at 51.
23. Guthrie et al., Blinking, supra note 8, at 33.
24. Id. at 3.
25. Id. at 8-9 (identifying various ways in which intuition and deliberation can interact in judgment and decisionmaking).
26. See Mark Lubell et al., Institutional Design Capitalizing on the Intuitive Nature of Decision Making, in Better Than Conscious: Decision Making, the Human Mind, and Implications for Institutions 413, 414 (Christoph Engel & Wolf Singer eds., 2008) ("[T]he distinction between conscious and unconscious is a somewhat artificial construct. So, too, is the supposed 'system 1' versus 'system 2' distinction." (citation omitted)).
27. Most ALJs, like most generalist judges, are lawyers, see, e.g., Skoler, supra note 4, at 36 ("[Federal] ALJs must be lawyers."), but some state ALJs are not
28. Yoder, supra note 1, at 325 ("We come in all shapes and sizes[:] lawyers and non-lawyers, federal and state, central panels and agency employees.").
29. Butz v. Economou, 438 U.S. 478, 513 (1978) ("There can be little doubt that the role of the modern federal hearing examiner or administrative law judge within this framework is 'functionally comparable' to that of a judge. His powers are often, if not generally, comparable to those of a trial judge: He may issue subpoenas, rule on proffers of evidence, regulate the course of the hearing, and make or recommend decisions.").
30. James E. Moliterno, The Administrative Judiciary's Independence Myth, 41 WAKE Forest L. Rev. 1191, 1196-97 (2006) (observing that ALJs "possess, of course, the fundamental core of the judicial definition, which, while nowhere given authoritatively, is by wide approval known to consist in the impartial adjudication of cases").
31. Michael Asimow, The Administrative Judiciary: ALJ's in Historical Perspective, 20 J. Nat'l Ass'n Admin. L. Judges 157, 157 (2000)
32. see also R. Terrence Harders, Striking a Balance: Administrative Law Judge Independence and Accountability, J. Nat'l Ass'n Admin. L. Judges, Spring 1999, at 1, 9 ("[T]hose who come before ALJs recognize that the deciding of disputes, the prescribing of duties, and the recognition of entitlement affects them in much the same way that litigation plays out in courts of law.").
33. Although we focus on potential advantages that ALJs possess, we acknowledge that ALJs might face potential disadvantages as well. For example, ALJs frequently face crowded dockets and often have fewer resources, such as judicial clerks, than many of their generalist counterparts. This might make them more, not less, prone to making snap, routinized, bureaucratic judgments.
34. See Koch, supra note 3, at 693 ("Administrative agencies adjudicate massive numbers of individual disputes, far exceeding the number resolved by courts.").
35. We recognize that a meaningful fraction of judicial branch judges serve on specialized courts, but most judicial branch judges are generalists. See generally Cent. European & Eurasian Law Initiative, AM. Bar Ass'n, Specialized Courts: A Concept Paper (1996) (on file with the Duke Law Journal) (providing background on the composition of the U.S. specialized courts).
36. See L. Harold Levinson, The Status of the Administrative Judge, 38 AM. J. COMP. LAW 523, 535 (1990) (" Most ALJs specialize in adjudicating cases for only one agency...."). This is uniformly true in the federal government because there all federal ALJs are employed by a particular agency. In the states, the employment arrangements of ALJs are more complicated and less amenable to " scholarly generalizations."
37. Jim Rossi, Overcoming Parochialism: State Administrative Procedure and Institutional Design, 53 Admin. L. Rev. 551, 553 (2001)
38. see also Skoler, supra note 4, at 35 ("Writing about administrative law judges at both federal and state levels is a tricky business and therefore rarely done."). Historically, most state ALJs, like their federal counterparts, worked for specific agencies
39. See, e.g., James F. Flanagan, Redefining the Role of the State Administrative Law Judge: Central Panels and Their Impact on State ALJ Authority and Standards of Agency Review, 54 Admin. L. Rev. 1355, 1357 (2002) [hereinafter Flanagan, Redefining the Role] (observing that California created the first central panel in 1945 and that seven states had central panels by 1983). Beginning in the 1980s and 1990s, many more states-now about thirty of them-moved to adopt, in various forms, "central panels" or independent ALJ agencies
40. See, e.g., James F. Flanagan, An Update on Developments in Central Panels and ALJ Final Order Authority, 38 IND. L. Rev. 401, 403-04 (2005) [hereinafter Flanagan, Update on Developments] ("Twenty-five states, and three major cities, have established central panels thus far."); Rossi, supra, at 568 ("In more than thirty states, to one degree or another administrative law judges (ALJs) are housed in a central panel. .. ."); Nat'l Ass'n of Admin. Law Judiciary, Non-Central Panel States, http://www.naalj.org/nonpanel.html (last visited Mar. 23, 2009) (identifying 29 states with central panels). Professor James Flanagan describes these central panel arrangements as follows
41. A central panel of ALJs is a cadre of professional adjudicators who are administratively independent of the agencies whose cases they hear, and thus, they are removed from agency influence. The central panels are organized in several ways. In some states, the central panel is an independent agency within the executive branch. In others, the central panel is part of another agency for administrative support, but independent for all other purposes. A third variation puts the ALJs in a separate organization, but assigns each ALJ to a particular agency based upon expertise in the subject matter.
42. Flanagan, Redefining the Role, supra, at 1356; see also
43. William R. Andersen, Judicial Review of State Administrative Action-Designing the Statutory Framework, 44 Admin. L. Rev. 523, 554-57 (1992) (proposing three models of administrative adjudication).
44. But see Koch, supra note 3, at 733 ("[P]anel judges, since they serve many agencies, are generalists and thus do not provide the expertise and experience inherent in the traditional scheme."). At present, then, all federal ALJs and nearly half of the state ALJs work within one agency, and among those state ALJs who are employed by central panels, many of them also specialize in the law and policy of a particular agency.
45. See Flanagan, Redefining the Role, supra note 29, at 1406 ("States often require ALJs to have substantive knowledge or experience in the subject matter. .. .")
46. see also, e.g., N.J. Stat. Ann. § 52:14F-12(a) (West 2001) (requiring that judges hired into the environmental unit have "special expertise" in environmental law).
47. Tex. Gov't Code Ann. § 2003.049(d) (Vernon 2000) (requiring ALJs to have experience in utility law).
48. This is so not only for the obvious reason that ALJs are repeatedly exposed to similar matters, but also because the agencies that employ them often provide the ALJs with important guidance to keep them abreast of relevant legal and policy developments. Flanagan, Redefining the Role, supra note 29, at 1406.
49. see also, e.g., Tex. GOV'T Code Ann. § 2001.058(c) (providing a specific example requiring the agency to provide a statement on applicable rules and policies).
50. See Edward Cheng, The Myth of the Generalist Judge, 61 STAN. L. REV. 519, 540 (2008) (presenting evidence that generalist appellate judges tend to write opinions in various specialized areas).
51. See infra Part III.A.
52. See, e.g., Chris Guthrie, Panacea or Pandora's Box?: The Costs of Options in Negotiation, 88 IOWA L. REV. 601, 641-42 (2003) (finding that lawyers were less susceptible than nonlawyers to the pernicious effects of contrast or asymmetric dominance)
53. Chris Guthrie & Jeffrey J. Rachlinski, Insurers, Illusions of Judgment & Litigation, 59 Vand. L. Rev. 2017, 2047 (2006) ("This paper reports the results of several litigation problems-two anchoring problems, three framing problems, and one self-serving bias problem-involving nearly two hundred participants from the insurance industry. The results reported in the paper suggest that these experts, relative to others who have been studied, make decisions that more closely approximate rational choice.")
54. Russell Korobkin & Chris Guthrie, Psychology, Economics, and Settlement: A New Look at the Role of the Lawyer, 76 Tex. L. Rev. 77, 99-101 (1997) (finding that lawyers are less susceptible than nonlawyers to framing effects).
55. Scott Plous, the Psychology of Judgment and Decision Making 258 (1993) ("[S]everal studies have found that experts display either roughly the same biases as college students or the same biases at somewhat reduced levels.").
56. see also, e.g., John C. Anderson, D. Jordan Lowe & Philip M.J. Reckers, Evaluation of Auditor Decisions: Hindsight Bias Effects and the Expectation Gap, 14 J. Econ. Psychol. 711, 725-31 (1993) (indicating that auditors, like nonexperts, are influenced by the hindsight bias)
57. Hal R. Arkes et al., Eliminating the Hindsight Bias, 73 J. Applied Psychol. 305, 307 (1988) (demonstrating that psychologists, like nonexperts, are prone to the hindsight bias)
58. Linda Babcock et al., Forming Beliefs About Adjudicated Outcomes: Perceptions of Risk and Reservation Values, 15 Int'l REV. L. & Econ. 289, 296-97 (1995) (finding that framing effects similarly influenced lawyer and nonlawyer subjects)
59. Barbara J. McNeil et al., On the Elicitation of Preferences for Alternative Therapies, 306 New Eng. J. Med. 1259, 1261-62 (1982) (finding that physicians, like nonexperts, are susceptible to framing effects)
60. Gregory B. Northcraft & Margaret A. Neale, Experts, Amateurs, and Real Estate: An Anchoring-and-Adjustment Perspective on Property Pricing Decisions, 39 Organizational Behav. & Hum. Decision Processes 84, 94-96 (1987) (finding that real estate agents, like nonexperts, are influenced by anchoring effects when estimating real estate prices).
61. Rachlinski et al., supra note 7, at 1256 ("With regard to the two phenomena that we have tested in prior studies of generalist judges-anchoring and framing-the bankruptcy judges performed much like the generalist judges we have previously studied.")
62. see also Theodore Eisenberg, Differing Perceptions of Attorney Fees in Bankruptcy Cases, 72 Wash. U. L.Q. 979, 983-85 (1994) (reporting evidence suggesting that bankruptcy judges are susceptible to self-serving or egocentric biases when making judgments).
63. Rachlinski et al., supra note 7, at 1257.
64. The argument developed here bears some relation to Mark Seidenfeld's argument that accountability created by the prospect of judicial review might enhance agency decisionmaking. See Mark Seidenfeld, Cognitive Loafing, Social Conformity, and Judicial Review of Agency Rulemaking, 87 Cornell L. REV. 486, 509 (2002) ("Judicial review provides accountability, within the core psychological concept. .. .")
65. see also Mark Seidenfeld, The Psychology of Accountability and Political Review of Agency Rules, 51 Duke L.J. 1059, 1061-68 (2001) (reviewing the psychological research on accountability).
66. According to James Flanagan, "the agency's power to review the findings of the ALJ" is a "fundamental premise of administrative adjudication." Flanagan, Update on Developments, supra note 29, at 401.
67. see also James E. Moliterno, The Administrative Judiciary's Independence Myth, 41 Wake Forest L. Rev. 1191, 1224 (2006) ("[A]dministrative judges are crucially dependent, both in that their decisions often require executive affirmation and in that they are always subject to executive review. .. .").
68. This "fundamental premise" is embedded in the Administrative Procedure Act, which gave rise to the modern ALJ in federal agencies. See 5 U.S.C. § 557(b) (2006)
69. see also Flanagan, Redefining the Role, supra note 29, at 1364. Moreover, the Model State Administrative Procedure Act adopted the federal standard in 1981, as did many of the administrative procedure acts in the states. Id.
70. In recent years, some states-particularly those that have embraced a "central panel" system, see supra note 29, have moved to enhance the finality of ALJ decisions, see, e.g., Flanagan, Redefining the Role, supra note 29, at 1373 (observing that "[a]t the beginning of the 1990s, in almost all the states including those with central panels, the statutes permitted the agency to amend the ALJ's findings of fact and conclusions of law with relative ease," but noting that this began to change in the early 1990s).
71. Jim Rossi, ALJ Final Orders on Appeal: Balancing Independence with Accountability, J. Nat'l Ass'n Admin. L. Judges, Fall 1999, at 1, 2 ("[F]ollowing the proliferation of central panels, many states have increasingly given ALJ orders de jure or de facto finality by taking away an agency's opportunity to review ALJ decisions or requiring agencies to accept ALJ findings unless the agency overcomes a fairly rigorous evidentiary or reasoning burden.").
72. Despite this innovation in some states, ALJs, even those who preside in central panel states, are much less independent and much more accountable than their counterparts in the judicial branch, as Professor James Moliterno explains
73. [T]he central fact of review remains: central panel administrative judges' decisions are also reviewed directly by agencies, often case-party agencies. This is the key fact that largely deprives administrative judges of the judicial independence trait. In reality, nothing is fundamentally changed with the institution of the central panel concept. The administrative judiciary simply becomes a judiciary within the executive branch/administrative state. This feature may further serve to insulate the administrative judge in an impartiality sense, but the administrative judge remains a member of the executive branch, fully subject to override by the agency
74. Moliterno, supra, at 1233.
75. Moliterno, supra note 39, at 1225.
76. See, e.g., Jennifer S. Lerner & Philip E. Tetlock, Accounting for the Effects of Accountability, 125 Psychol. Bull. 255, 260-62 tbl.1 (1999) (documenting the extensive literature on accountability).
77. See, e.g., Lerner & Tetlock, supra note 41, at 256 ("When audience views are known prior to forming one's own opinion, conformity becomes the likely coping strategy. People can simply adopt positions likely to gain the favor of those to whom they are accountable, thereby allowing them to avoid the unnecessary cognitive work of analyzing the pros and cons of alternative courses of action, interpreting complex patterns of information, and making difficult trade-offs." (citations omitted))
78. Philip E. Tetlock, The Impact of Accountability on Judgment and Choice: Toward a Social Contingency Model, 25 Adv. Experimental Soc. Psychol. 331, 340-41 (1992) (observing that when people know the views of those to whom they are accountable, they frequently adopt those views).
79. Lerner & Tetlock, supra note 41, at 259, 260-62 tbl.1 (describing this phenomenon and identifying the decision heuristics and biases that are attenuated, as well as a small number that are actually amplified, by this form of accountability)
80. see also Tetlock, supra note 42, at 343 ("Predecisional accountability to unknown audiences frequently motivates vigilant, complex, and self-critical thinking."). For refinements, see Itamar Simonson & Peter Nye, The Effect of Accountability on Susceptibility to Decision Errors, 51 Organizational Behav. & Human Decision Processes 416, 437 (1992). "[A]ccountability is expected to reduce errors if decision makers can identify the response that would be regarded as more rational, but that response is different from the one that unaccountable decision makers tend to make." Id.
81. For the argument that any behavioral analysis of legal decisionmaking should consider the potential impact of accountability on the decisionmaker, see Gregory Mitchell, Why Law and Economics' Perfect Rationality Should Not Be Traded for Behavioral Law and Economics' Equal Incompetence, 91 GEO. L.J. 67, 110-14 (2002).
82. For example, ALJs in Oregon might face the same degree of scrutiny as the trial court: Ninety eight percent of the OAH's orders are final orders. These orders generally cannot be changed by agencies after ALJs issue them. They are appealable to a circuit court or the Court of Appeals (unemployment insurance decisions are appealable to the Employment Appeals Board). The remaining OAH orders are proposed orders. Proposed orders are decisions recommended to agencies by ALJs based on their review of the facts and law. Agencies are not required to accept the recommendations. However, if they do not, they are required to explain the reason for all "substantial" changes. If findings of fact are changed, they must show how most of the evidence at hearing supports their version of the facts; on appeal to the Court of Appeals, the Court can look at the entire record and determine independently (de novo) whether it agrees with the agency's version.
83. Office of Admin. Hearings, The OAH, http://www.oregon.gov/OAH/The_OAH2.shtml (last visited Mar. 1, 2009).
84. The fact that some ALJs make only recommended decisions might encourage reliance on intuitive processes. Knowing that a decision might be reviewed by an agency head could lead to a diffusion of responsibility so that ALJs worry less about the accuracy of their decisions. Although this is possible, we doubt it. We suspect that ALJs, like most judges, take their positions very seriously and use whatever resources and information that they have available to try to make sound judgments.
85. See, e.g., C.K. Rowland & Robert A. Carp, Politics and Judgment in Federal District Courts 8 (1996) ("[O]nly about 20 percent of all district court cases are appealed in any given year.").
86. Theodore Eisenberg, Appeal Rates and Outcomes in Tried and Nontried Cases: Further Exploration of Anti-Plaintiff Appellate Outcomes, 1 J. EMPIRICAL LEGAL STUD. 659, 685 (2004) ("About 20 percent of cases with definitive trial court judgments generate appeals, with tried cases appealed at about twice the rate of nontried cases.").
87. See, e.g., Rowland & CARP, supra note 47, at 9 (observing that interpretations of fact are free from review)
88. William M. Landes & Richard A. Posner, Harmless Error, 30 J. Legal stud. 161, 162 (2001)
89. Maurice Rosenberg, Standards of Review, in restructuring Justice: The Innovations of the Ninth Circuit and the Future of the Federal Courts 30, 31 (Arthur D. Hellman ed., 1990) (explaining bases of review).
90. See, e.g., Charles E. Daye, Powers of Administrative Law Judges, Agencies, and Courts: An Analytical and Empirical Assessment, 79 N.C. L. REV. 1571, 1616 (2001) (reporting, based on an empirical examination of ALJ decisionmaking in North Carolina, that agencies fully adopted 82 percent and partially adopted 6 percent of the decisions recommended by their ALJs).
91. We do not address the issue of political conflicts between the ALJs and the agency. An ALJ could interpret the law and facts perfectly well, but face an agency bent on moving the law in a new direction, or implementing a new policy. Although such conflicts would produce reversals of the ALJ, they are not really errors.
92. The Social Security Administration, for example, conducts these kinds of audits. U.S. GEN. Accounting Office, SSA Disability Decision Making: Additional Steps Needed to Ensure Accuracy and Fairness of Decisions at the Hearings Level 9 (2003), available at http://www.gao.gov/new.items/d0414.pdf (describing the SSA's Office of Quality Assurance and Performance Assessment's process for conducting quality assurance reviews of a random sample of ALJ decisions). This is similar to the peer review we recommended in a previous paper. Guthrie et al., Blinking, supra note 8, at 39.
93. See, e.g., Robin M. Hogarth, Educating Intuition 88-90 (2001) (distinguishing "kind" environments, which provide valuable feedback and in turn can improve intuitions, from "wicked" environments, which do not).
94. Adoption rates vary. In Oregon, adoption rates exceed 98 percent in many categories of cases. See Office of Admin. Hearings, supra note 45. Arizona reports that adoption rates of ALJ decisions are over 85 percent. Office of Admin. Hearings, Thirteenth Annual Report 1 (2008), available at http://www.azoah.com/13thAnnualReport.pdf (reporting that the Arizona agency acceptance of central panel ALJs' findings of fact and conclusions of law without modification was 92.6 percent, and that the overall rate of adoption of entire ALJ decisions (that is, including the remedy portion of the recommendation) without modification was 86.64 percent).
95. We recognize, of course, that some previous empirical studies of ALJs suggest that they are not ideal decisionmakers. Some ALJs have been shown to produce decisions that vary by the race of the litigants, see Stephen Labaton, Benefits Are Refused More Often to Disabled Blacks, Study Finds, N.Y. Times, May 11, 1992, at A1 (summarizing a 1992 study by the General Accounting Office and reporting that "[f]or 30 years, blacks with serious ailments have been much more likely than whites to be rejected for benefits under Social Security disability programs"), and others have been shown to be wildly erratic.
96. see Jaya Ramji-Nogales, Andrew I. Schoenholtz & Philip G. Schrag, Refugee Roulette: Disparities in Asylum Adjudication, 60 STAN. L. REV. 295, 296 (2007) (reporting "amazing disparities" in asylum grant rates among immigration judges).
97. See sources cited supra note 8.
98. See infra Part II.B.
99. Note that some of the judges identified more than one type of case in response to this question.
100. We do not have this data available for the city conference judges (although most are in the third category), so we do not categorize them in this way or include them in specialized analyses that distinguish the ALJs from one another based on the type of review to which their decisions are (or are not) subjected.
101. Copies of the original versions of these problems are included in Appendices A and B, along with a copy of the questions that elicited demographic information about the judges.
102. Frederick, supra note 16, at 26-28.
103. Id. at 27.
104. Id. at 35.
105. Id. at 27.
106. Researchers utilized the same three questions in Frederick's study. Id.
107. Id. at 26-27.
108. Id. at 27.
109. As we have noted elsewhere, see Guthrie et al., Blinking, supra note 8, at 11 n.57, this problem assumes that each machine produces widgets at the same rate, which seems a reasonable assumption given the structure of the problem and of the CRT as a whole.
110. Frederick, supra note 16, at 27.
111. Id. at 28, 29 tbl.1.
112. See id. at 27.
113. Id. at 28.
114. Id. In contrast to the bat-and-ball problem, the banana-and-bagel problem does not suggest an intuitive response. Accordingly, most people attempt to solve the problem deliberatively by explicitly or implicitly using algebra-i.e., x = bagel, y = banana, y = x + 13, so x + x + 13 = 37. Solving for x, x = 12 and y = 25.
115. Id. at 27.
116. Id. at 35.
117. Jorg Oechssler, Andreas Roider & Patrick W. Schmitz, Cognitive Abilities and Behavioral Biases 4-6 (Inst. for the Study of Labor, Discussion Paper No. 3481, 2008), available at http://ssrn.com/abstract=1294555.
118. Guthrie et al., Blinking, supra note 8, at 14.
119. Frederick, supra note 16, at 29 tbl.1.
120. Guthrie et al., Blinking, supra note 8, at 16.
121. Among these sixteen judges, five answered none of the questions; four answered both of the first two questions with the intuitive (but wrong) answer, and did not answer the third question; one answered the first question with the correct answer, the second question with the intuitive (but wrong) answer, and did not answer the third question; one answered the first question with the intuitive (but wrong) answer and did not respond to the second or third questions; one answered the first question with the intuitive answer and the second question with a wrong (but not the intuitive) answer, and did not respond to the third question; one answered the first question with the intuitive but incorrect answer, the second question with the correct answer, and did not respond to the third question; three answered the first question with the intuitive (but wrong) answer, skipped the second question, and answered the third question with the correct answer
122. An ordered logit regression revealed that the two groups did not differ significantly. z = 1.08, p = .28.
123. An ordered logit regression revealed that the ALJs did not perform significantly differently than the Florida trial judges. z = .80, p = .45.
124. All of the nonjudge data on this table were collected on college students, and comes from Frederick, supra note 16, at 29 tbl.1. The data from the Florida Judges come from Guthrie et al., Blinking, supra note 8, at 15 tbl.2. Note that the table in our previous article mistakenly reports the sample size as 192 rather than 252.
125. Previous research with other populations shows that men perform slightly better than women on the CRT. Frederick, supra note 16, at 37-38, 38 tbl.6 (showing male subjects with a mean sore of 1.47 and female subjects with a mean score of 1.08). We did not find this to be the case with the Florida trial judges in our previous work, but we did see differences among the ALJs. The fifty-eight male judges who both completed the CRT and identified their gender scored an average of 1.52, whereas the sixty-six women judges scored an average of 1.17. This difference was marginally significant, per an ordered logit regression, revealing z =1.80, p=.07.
126. The average CRT scores among the thirty-seven judges in the national conference sample who make recommendations and completed the CRT was 1.30, as compared to 1.20 among the forty-nine judges who are only subject to review; this was not a significant difference. The analysis was done with an ordered logit regression, revealing z =.37, p=.72. This last result is perhaps no surprise, but is inconsistent with the hypothesis that judges who make recommendations develop radically different habits of mind that lead them to engage in deliberative thinking across the board.
127. See Tversky & Kahneman, supra note 9, at 1128 ("[D]ifferent starting points yield different estimates, which are biased toward the initial values.").
128. Plous, supra note 35, at 146 (reporting the results of an unpublished study conducted by George Quattrone and his colleagues)
129. Tversky & Kahneman, supra note 9, at 1128 (introducing the phenomenon)
130. See Lerner & Tetlock, supra note 41, at 262-63 ("Moreover, two especially pervasive tendencies, (a) anchoring on an initial value and insufficiently adjusting a numerical estimate up or down from that anchor and (b) weighting sunk costs when considering future investments are also reduced by accountability." (citations omitted)).
131. See Guthrie et al., Judicial Mind, supra note 8, at 787-92.
132. Wistrich et al., supra note 8, at 1286-91.
133. Guthrie et al., supra note 8, at 791-92.
134. Wistrich et al., supra note 8, at 1288-91.
135. Although we were concerned that the request for an explanation might have induced some judges not to answer the question, this does not appear to have been the case. Twenty-one judges did not respond to this question: seven in the control, no-explanation condition; four in the control, explanation condition; four in the anchor, no-explanation condition; and six in the anchor and explanation condition. Among the forty-three judges who provided an award in the explanation conditions, four did not provide an explanation (one in the control condition and three in the anchor condition). These judges were nevertheless kept in the analysis.
136. The $1,000,000 award was in the no anchor condition. The five next highest awards were in the anchor condition.
137. F(1, 78) = 8.44, p < .005.
138. F(1, 78) = 1.65, p = .20, F(1, 78) = .01, p > .90, respectively for the effect of the accountability manipulation and the interaction of this variable with the anchoring condition. Further analysis of the judges' gender, years of experience, and whether they make recommendations showed that these variables did not affect the judges' awards. For gender and recommendation, this was done using a 2x2x2 ANOVA on anchor condition, explanation condition, and gender; all F's involving gender < 1.4, p's > .20; all F's involving recommendation < 1.5, p's > .20. Experience was tested with ANCOVA, using experience as a continuous variable; all F's involving experience and its interactions < 1.10, p's > .25. The judge's CRT score likewise had no effect. CRT score was tested with ANCOVA, using CRT score as a continuous variable; all F's involving CRT score and its interactions < .15, p's > .70. This last result was not entirely a surprise, as other researchers have found that scores on the CRT do not correlate with the anchoring effect. Oechssler et al., supra note 79, at 6 ("Furthermore, the effect of anchoring is not diminished by cognitive ability. In fact, if anything, the High CRT group seems to be more susceptible to anchoring, although this effect is not significant.").
139. See Daniel Kahneman & Amos Tversky, Choices, Values, and Frames, 39 AM. Psychologist 341, 342 (1984) ("[T]he psychophysical analysis of outcomes should be applied to gains and losses rather than to total assets.").
140. Daniel Kahneman, Jack L. Knetsch & Richard Thaler, Fairness as a Constraint on Profit Seeking: Entitlements in the Market, 76 AM. ECON. REV. 728, 731 (1986) ("These characteristics of evaluation make preferences vulnerable to framing effects, in which inconsequential variations in the presentation of a choice problem affect the decision.").
141. Id. (emphasis added).
142. Id. at 731-32.
143. See Guthrie et al., Judicial Mind, supra note 8, at 794-97 (describing how "categorization, or 'framing,' of decision options influences the way people evaluate options and affects their willingness to incur risk" in the context of litigation).
144. Ordered logit regression showed that the effect of framing was significant. z = 2.87, p < 005. Further analysis of the judges' gender and years of experience, also using ordered logit regressions, showed that neither the main effect of variables nor their interaction with frame affected the judges' ratings. For gender, z's < 1.62, p's > .10; for experience, z's < .65, p's > .50. The judges' CRT scores likewise did not affect the results. z's < 1.10, p's > .25. Although framing effects have been found to correlate with CRT scores in other contexts, the type of framing effect we studied here has not. Oechssler et al., supra note 79, at 4 ("For both items [coded as gains and losses], the High CRT group is more likely to choose the alternative that is compatible with risk neutrality."). That is, the framing of an option as a gain or a loss influences people's willingness to undertake risk and the framing influences how much importance people attach to gains and losses; the former, but not the latter, appears to be related to the cognitive abilities that the CRT is measuring.
145. Maya Bar-Hillel & Efrat Neter, How Alike Is It Versus How Likely Is It: A Disjunction Fallacy in Probability Judgments, 65 J. Personality & soc. Psychol. 1119, 1119 (1993).
146. see also Amos Tversky & Daniel Kahneman, Extensional Versus Intuitive Reasoning: The Conjunction Fallacy in Probability Judgment, 90 Psychol. rev. 293, 293-94 (1983) ("[P]robability theory does not determine the probabilities of uncertain events-it merely imposes constraints on the relations among them. For example, if A is more probable than B, then the complement of A must be less probable than the complement of B.").
147. Researchers have described this as "perhaps the simplest and most transparent rule of probability theory," Bar-Hillel & Neter, supra note 109, at 1130, which "even untrained and unsophisticated people accept and endorse," id.
148. Id. at 1119.
149. See Tversky & Kahneman, supra note 109, at 297.
150. Amos Tversky & Daniel Kahneman, Judgments of and by Representativeness, in Judgment Under Uncertainty: Heuristics and Biases 84, 92 (Daniel Kahneman, Paul Slovic & Amos Tversky eds., 1982).
151. Tversky & Kahneman, supra note 109, at 297.
152. Tversky & Kahneman, supra note 114, at 92.
153. Tversky & Kahneman, supra note 109, at 297.
154. Tversky & Kahneman, supra note 114, at 92.
155. Tversky & Kahneman, supra note 109, at 297.
156. Tversky & Kahneman, supra note 114, at 93.
157. The Linda problem can be criticized as methodologically flawed in that people might be assuming that the single feature is actually meant to be the conjunction of "bank teller" and "not active in the feminist movement." But Tversky and Kahneman have administered a version in which they avoid this problem by changing the second response to "Linda is a bank teller whether or not she is active in the feminist movement" and obtained similar results. Tversky & Kahneman, supra note 109, at 299.
158. Of these thirty-three judges, twenty ranked the fourth option as more likely than both options one and two, eight wrote in that they were all equally likely, three assigned the same ranking to options one and two as option four, and two simply put a check mark next to option four.
159. Of these thirty-six judges, one ranked the first and fourth options as equally likely.
160. Of these fifteen judges, seven ranked the first and fourth options as equally likely.
161. Although female judges were somewhat more likely to commit the conjunction error than male judges (89 percent, or fifty out of fifty-six, versus 79 percent, or thirty-one out of thirty-nine, respectively), this difference was not significant. Fisher's exact, p = .24. More experienced judges tended to be more likely to commit the error than their younger counterparts, but this trend was also not significant. Logistic regression of committing the error on years of experience yielded a negative, but not significant, coefficient of -.048, z = 1.35, p = .18. The error rate was also nearly identical among judges who make recommendations as opposed to the other judges (86 percent, or thirty-six out of forty-two, versus 84 percent, or forty-two out of fifty, respectively). This was not a significant difference. Fisher's exact, p = 1.00.
162. Previous work on the CRT has shown that people who score high on the CRT are less likely to commit the conjunction error. Oechssler et al., supra note 79, at 5 ("Of our subjects in the Low CRT group, 62.6% [committed the conjunction fallacy on the 'Linda' problem, but t]his percentage is much lower for the High CRT group at 38.3%...."). We found that among the judges who scored perfectly on the CRT and answered this question, 72 percent (thirteen out of eighteen) committed the fallacy, whereas 86 percent (sixty out of seventy) of the judges who got at least one of the CRT questions wrong committed the fallacy. This difference was not, however, significant. Fisher's exact, p = .29. The percentage who committed the conjunction error broken down by exact CRT score is: zero right on CRT, 84 percent (twenty-five out of thirty).
163. one right on CRT, 86 percent (nineteen out of twenty-two); two right on CRT, 89 percent (sixteen out of eighteen); three right on CRT, 72 percent (thirteen out of eighteen). We also ran a logistic regression of whether the judges committed the error based with CRT score as a predictor, which also showed no significant effect. z = .72, p = .47. That we only observed a trend might be due to a somewhat small sample size with which to identify the effect.
164. See Baruch Fischhoff, Hindsight ≠ Foresight: The Effect of Outcome Knowledge on Judgment Under Uncertainty, 1 J. Experimental Psychol.: Hum. Perception & Performance 288, 288 (1975) ("Reporting an outcome's occurrence increases its perceived probability of occurrence(.").
165. Jonathan Baron & John C. Hershey, Outcome Bias in Decision Evaluation, 54 J. Personality & Soc. Psychol. 569, 570 (1988) ("At issue here is whether there is an outcome bias(.").
166. Guthrie et al., Judicial Mind, supra note 8, at 799-803.
167. See Wistrich et al., supra note 8, at 1313-16 (finding no statistically significant difference in judges' conclusions based on foresight as compared to their conclusions based on hindsight).
168. Our study is similar to, and was inspired in part by, a study conducted by Professor Jonathan Casper and his colleagues. See Jonathan D. Casper, Kennette Benedict & Jo L. Perry, Juror Decision Making, Attitudes, and the Hindsight Bias, 13 LAW & HUM. BEHAV. 291 (1989).
169. See Jeffrey J. Rachlinski, A Positive Psychological Theory of Judging in Hindsight, 65 U. CHI. L. REV. 571, 584 (1998) ("[P]eople naturally integrate an outcome and the events that preceded it into a coherent story.").
170. Id. at 581 ("[A]cross a wide variety of tasks, materials, and populations, a sizeable and consistent bias clouds judgments made in hindsight." (footnote omitted)).
171. See Casper et al., supra note 128, at 300 (reporting that subjects assessing a potential civil rights violation displayed a "hindsight-bias-like process" in relying on outcome information).
172. Fisher's exact test, p = .05.
173. Further analysis of gender, years of experience, and CRT score revealed some interesting trends. Male judges expressed a stronger hindsight bias than their female counterparts. Among the male judges, 71 percent (ten out of fourteen) of those who read about the innocent kid concluded that the guard had discriminated, as compared to 13 percent (one out of eight) of those who read about the guilty kid. By contrast, among the female judges, 50 percent (four out of eight) who read about the innocent kid concluded that the guard had discriminated, as compared to 44 percent (four out of nine) of the judges who read about the guilty kid. The different reaction to the knowledge of the outcome between male and female judges was marginally significant. Ordered logit of finding of discrimination on the condition, gender, and interaction of condition and gender produced a marginal effect for the interaction. z = 1.69, p = .09.
174. The more experience the judges had, the less that they were influenced by the outcome, although this trend only approached significance. Ordered logit of finding of discrimination on the condition, experience, and interaction of condition and experience produced a coefficient of -.15, which was not significant. z = 1.55, p = .11. An effect of experience here is consistent with previous research on the hindsight bias, which demonstrates smaller effects among people with greater knowledge. See Jay J.J. Chistensen-Szalansky & Cynthia Fobian Williams, The Hindsight Bias: A Meta-Analysis, 48 Organizational Behav. & HUM. Decision Processes 147, 162 (1991) ("[Hindsight] effects are further reduced when the subject is familiar with the task, i.e., has expertise in or experience with the task.").
175. The CRT score did not appear to influence the judges' judgment on this task. Ordered logit of finding of discrimination on the condition, CRT score, and interaction of condition and CRT produced no significant interaction. z = .50, p = .62. Because there is no finding that CRT score influences the hindsight bias, this was not surprising.
176. Because eleven of the judges identified themselves as primarily responsible for employee discipline (nine judges) or discrimination (two judges) cases, we tried to assess whether these judges behaved differently than the other judges who did not have such experience. Among the judges with no experience in this area, 71 percent (ten out of fourteen) of those who read about the innocent kid concluded that the guard had discriminated, as compared to 31 percent (four out of thirteen) of those who read about the guilty kid. By contrast, among the judges with experience, 57 percent (four out of seven) who read about the innocent kid concluded that the guard had discriminated, as compared to 50 percent (one out of two) of the judges who read about the guilty kid. This suggests that experience might induce deliberative decisionmaking, but because only two experienced judges read about the guilty kid, the results cannot be interpreted as anything more than suggesting that experience aids judges' ability to avoid the hindsight and outcome bias.
177. See Jonathan M. Golding & Debra L. Long, There's More to Intentional Forgetting than Directed Forgetting: An Integrative Review, in Intentional Forgetting: Interdisciplinary Approaches 59, 93 (Jonathan M. Golding & Colin M. MacLeod eds., 1998) (reviewing theoretical accounts of this phenomenon)
178. Hollyn M. Johnson, Processes of Successful Intentional Forgetting, 116 PSYCHOL. BULL. 274, 274 (1994) ("[The] success of intentional forgetting depends on how one originally encoded the to-be-forgotten information(.").
179. Daniel M. Wegner et al., Paradoxical Effects of Thought Suppression, 53 J. Personality & soc. Psychol. 5, 6-7 (1987).
180. Id. at 7.
181. See Wistrich et al., supra note 8, at 1323-23 (concluding that although judges "do not disregard inadmissible information" in all cases, they are able to do so in some circumstances).
182. We pilot tested a version of this problem with ALJs at the city conference and made some adjustments to the context of the problem in light of recommendations from some of the judges in attendance
183. Fisher's exact test, p = .22.
184. Six judges did not provide complete responses: one judge in the condition without the evidence gave no award; one judge who read about the photo ruled it inadmissible, but gave no award; one judge (in the third condition) refused to rule or award; one judge who saw the photo ruled it admissible but gave no award; two judges who saw the photo ruled it inadmissible but gave no award
185. A tobit regression of the award on two dummy-coded variables to reflect the three conditions did not produce a significant effects for the regression (p < .33) or for either of the dummy codes (for exposure to the evidence versus the control condition, t =.22, p =.83, for exposure to the photo versus the other two conditions, t = 1.32, p = .19).
186. Among the nine judges who read about the photo of the rats and admitted the testimony, three chose the maximum fine, three chose fines between $0 and $2,000 (specifically, $400, $500, and $1500), and three chose no fine ($0); the average fine was $933. Among the four judges who saw the photo of the rats and admitted the testimony, two chose the maximum fine and two chose no fine ($0), yielding an average of $1,000.
187. See Michael Ross & Fiore Sicoly, Egocentric Biases in Availability and Attribution, 37 J. Personality & Soc. Psychol. 322, 333 (1979) ("[T]he egocentric biases in availability and attribution appear to be robust and pervasive.").
188. Neil D. Weinstein, Unrealistic Optimism About Future Life Events, 39 J. Personality & Soc. Psychol. 806, 809-11 (1980).
189. Lynn A. Baker & Robert E. Emery, When Every Relationship Is Above Average: Perceptions and Expectations of Divorce at the Time of Marriage, 17 LAW & HUM. Behav. 439, 441-43 (1993).
190. Ola Svenson, Are We All Less Risky and More Skillful than Our Fellow Drivers?, 47 Acta Psychologica 143, 145-46 (1981)
191. But see Don A. Moore & Deborah A. Small, Error and Bias in Comparative Judgment: On Being Both Better and Worse than We Think We Are, 92 J. Personality & Soc. Psychol. 972, 973-74 (2007) (arguing that self-serving responses are often reasonable ones given the subjects' ignorance about how others perform on such tasks).
192. Guthrie et al., Judicial Mind, supra note 8, at 813-14.
193. The distribution of these results departs significantly from what one would expect if there were no bias: Evaluating credibility, χ2(3) = 26.0, p < .001; facilitating settlement, χ2(3) = 24.2, p < .001; avoiding prejudice, χ2(3) = 32.2, p < .001.
194. Further analysis revealed that female judges made assessments that were somewhat more self-serving than male judges as to some of their abilities. Female judges made somewhat more self-serving assessments of their abilities to assess witness credibility than male judges (76 percent, or sixteen out of twenty-one male judges, versus 93 percent, or fourteen out of fifteen female judges), but an ordered logit regression of the rating against gender revealed that this tendency was not significant. z = 1.42, p = .16. Female judges, however, rated their abilities to facilitate settlement high; whereas 76 percent (sixteen out of twenty-one) of the male judges rated themselves as better than the median judge, all fifteen female judges did so. An ordered logit regression of the rating against gender revealed that this difference was a marginally significant effect. z = 1.71, p = .09. Male and female judges made roughly equally self-serving assessments of their ability to avoid bias (the only judge to rate himself below the median was a male judge). An ordered logit regression of the rating against gender revealed no significant effect. z = .46, p = .64.
195. CRT score also affected the self-serving bias. Judges who scored high on the CRT made somewhat more modest assessments of their abilities to evaluate witness credibility. An ordered logit regression of the rating against CRT score revealed that this tendency was a marginally significant effect. z =1.69, p = .09. CRT score did not correlate with the judges' assessments of their abilities to facilitate settlement. An ordered logit regression of the rating against gender revealed that this difference was a marginally significant effect. z = .46, p = .64. But judges who scored high on the CRT made more modest assessments of their abilities to avoid prejudice. An ordered logit regression of the rating against CRT score revealed that this tendency was a significant effect. z = 2.01, p < .05. These results are consistent with previous work showing that those who do well on the CRT also tend to avoid self-serving assessments. Oechssler et al., supra note 79, at 6 ("The fraction of subjects who judge their performance in the CRT correctly is dramatically higher for the High CRT group at 67.8% than for the Low group (9.7%).").
196. Years of experience did not correlate with the judges' self-serving tendencies on any of the three assessments. An ordered logit regression of the experience against each of the three evaluations revealed no significant effects: Evaluate witnesses, z = .45, p = .65; avoid prejudice, z = .48, p = .63; facilitate settlement, z = .23, p = .82.
197. To be sure, our exploration of the impact of specialization on ALJ decisionmaking was imperfect. We tested two different samples of judges who spend most of their time in a wide variety of highly specialized areas. Because we could not easily anticipate what these specialties would be in advance of collecting the data and did not anticipate the degree of variation, our scenarios only infrequently and sporadically present fact patterns that match the specialty of the judges. Our scenarios for the New York judges (our first sample) included housing and discrimination issues, but we only had two judges who oversee housing cases and nine judges who address discrimination. In preparing for the study of national judges, we hoped to capitalize on the number of judges who addressed discrimination by using two scenarios involving discrimination, but the national sample included only four such judges. We also anticipated that a number of judges would identify themselves as addressing health, safety, sanitation, or environmental issues. This was true for both samples, but in the case of the New York judges, our materials were unusable, and in the national sample, we did not observe any effect that might vary by experience.
198. See supra note 127 and accompanying text.
199. See supra note 148.
200. See supra note 122.
201. See supra note 29.
202. See supra note 39.
203. See, e.g., Christopher B. McNeil, Perceptions of Fairness in State Administrative Hearings, 92 Judicature 160, 162 (2009) (reporting that citizens find some central panel adjudications more fair than within-agency adjudications).
204. See supra text accompanying note 18.
205. Tversky & Kahneman, supra note 10, at 1124.