Understanding discrimination in the scored society

Washington Law Review

Volumn 89, Issue 4, 2014, Pages 1375-1412

  Zarsky, Tal Z ^a  

^a UNIVERSITY OF HAIFA   (Israel)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84922012959     PISSN: 00430617     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (156)

1. Danielle Keats Citron & Frank Pasquale, The Scored Society: Due Process for Automated Predictions, 89 Wash. L. Rev. 1 (2014).
2. For this Article, I rely on Citron and Pasquale’s description of the process, as follows: “Predictive algorithms mine personal information to make guesses about individuals’ likely actions and risks. A person’s on- and offline activities are turned into scores that rate them above or below others. Private and public entities rely on predictive algorithmic assessments to make important decisions about individuals.” Id. at 3.
3. Id. at 3–4 (noting the discussion in terms of “power”). The article also notes “mutual-scoring opportunities” and instances in which “individuals can score the scorers” but note that more often the situation is reversed. Id. at 3.
4. Evgeny Morozov, The Real Privacy Problem, 116 MIT Tech. Rev. 32-37 (2014), available at http://www.technologyreview.com/featuredstory/520426/the-real-privacy-problem/.
5. For a discussion of the rise of these dynamics almost a decade ago, see Tal Z. Zarsky, “Mine Your Own Business!”: Making the Case for the Implications of the Data Mining of Personal Information in the Forum of Public Opinion, 5 Yale J.L. & Tech. 1 (2003).
6. Citron & Pasquale, supra note 1, at 12–13.
7. Id. at 4.
8. Id.
9. Id.
10. Id. at 11.
11. Id. at 20 (noting that at times the systems take on a life of their own).
12. As explained throughout this Article, discrimination doctrine has severe shortcomings. It is also so often applied that we might be witnessing “discrimination fatigue,” and perhaps it is best we opt for broader theories and justifications such as human dignity. See Kenji Yoshino, The New Equal Protection, 124 Harv. L. Rev. 747-795 (2011). Yet given the salience of discrimination as a concept in the already existing discourse, it is important to engage in an analytical discussion as to its proper meaning.
13. In the academic context (beyond The Scored Society), see generally Solon Barocas & Andrew D. Selbst, Big Data’s Disparate Impact (forthcoming), available at http://ssrn.com/ abstract=2477899. In the policy realm, see Executive Office of the President, Big Data: Seizing Opportunities, Preserving Values 51 (2014), available at http://www.whitehouse.gov/sites/default/files/docs/big_data_privacy_report_may_1_2014.pdf. For a recent discussion in the New York Times which involves both academia and policy, see Seeta Peña Gangadharan et al., Room for Debate: Is Big Data Spreading Inequality?, N.Y. Times (Aug. 7, 2014, 12:22 PM), http://www.nytimes.com/roomfordebate/2014/08/06/is-big-data-spreading-inequality/the-dangers-of-high-tech-profiling-using-big-data.
14. Big Data and Consumer Privacy in the Internet Economy, 79 Fed. Reg. 32714 (June 6, 2014).
15. Discrimination indeed cannot be reduced to one theory. See Susan Sturm, Second Generation Employment Discrimination: A Structural Approach, 101 Colum. L. Rev. 458-473 (2001).
16. For discussions of instances where these were indeed the reasons for concerns regarding technology in general, see K.A. Taipale, Technology, Security and Privacy: The Fear of Frankenstein, The Mythology of Privacy and the Lessons of King Ludd, 7 Yale J.L. & Tech. 138 (2005).
17. “Protected groups” is a term of art broadly applied in the discrimination discourse. See infra text accompanying note 49; infra text accompanying notes 103–105.
18. Citron & Pasquale, supra note 1, at 13–15.
19. Similar solutions were also set forth by others in the broader “Big Data” context. See Viktor Mayer-Schönberger & Kenneth Cukier, Big Data 171–84 (2014).
20. Citron & Pasquale, supra note 1, at 21–22.
21. Id. at 28.
22. Id. at 25.
23. Id. at 28–29.
24. Id.
25. 25.OECD Privacy Principles, OECDprivacy.org (Aug. 9, 2010), http://oecdprivacy.org/ For a recent discussion of the formulation of Fair Information Practices (FIPs), see Robert Gellman, Fair Information Practices: A Basic History, Robert Gellman: Privacy & Info. Pol’y Consultant (Aug. 3, 2014), http://bobgellman.com/rg-docs/rg-FIPShistory.pdf.
26. See Marc Rotenberg, Fair Information Practices and the Architecture of Privacy (What Larry Doesn’t Get), 2001 Stan. Tech. L. Rev. 1-15 (2001).
27. OECD Privacy Principles, supra note 25.
28. Id.
29. Id.
30. Commission Proposal for a Regulation of the European Parliament and of the Council: On the Protection of Individuals with Regard to the Processing of Personal Data and on the Free Movement of Such Data (General Data Protection Regulation, COM (2012) 11 final (Jan. 25, 2012), available at http://ec.europa.eu/justice/data-protection/document/review2012/com_2012_11_en.pdf.
31. For a discussion as to the relation between antidiscrimination and data protection law, see Raphaël Gellert et al., A Comparative Analysis of Anti-Discrimination and Data Protection Legislations, in Discrimination and Privacy in the Information Society: Data Mining and Profiling in Large Databases 61, 61–89 (Bart Custers et al. eds., 2013).
32. Citron & Pasquale, supra note 1, at 5.
33. Id. at 13.
34. Id. at 7.
35. See Frederick Schauer, Profiles, Probabilities and Stereotypes 3 (2006).
36. Citron & Pasquale, supra note 1, at 24.
37. For a recent discussion of some of the doctrinal issues these practices bring about, see Barocas & Selbst, supra note 13, at 23–43.
38. Larry Alexander, What Makes Wrongful Discrimination Wrong?, 141 U. Pa. L. Rev. 149-151 (1992).
39. Id. at 154. In his important analysis of discrimination-related issues, Schauer also explains that rather than selecting a “moral standpoint at the outset” he chooses to explore a diverse collection of topics, while examining the intuitions and arguments which apply to them. Schauer, supra note 35, at 24.
40. See Citron & Pasquale, supra note 1, at 7, 13–15, 31, 33.
41. Id. at 5.
42. Indeed, intent is required in terms of applying the Equal Protection Clause. See Alexander, supra note 38, at 179.
43. This assumption is also noted in passing in The Scored Society. Citron & Pasquale, supra note 1, at 4.
44. Schauer, supra note 35, at 215.
45. See Zarsky, supra note 5, at 25 n.73 (referencing relevant work by Ian Ayres and Gary Becker).
46. See, e.g., Ian Ayres, Fair Driving: Gender and Race Discrimination in Retail Car Negotiations, 104 Harv. L. Rev. 817, 843–45 (1991). Ayers introduces several forms of statistical theories of discrimination, such as cost-based and revenue-based discrimination. The former refers to seller’s inferences that specific types of consumers impose additional costs on the firm and therefore should be avoided, while the latter refers to cases in which the sellers draw inferences regarding the revenue which could be derived from different customers given their specific traits and therefore charge higher prices. Id.
47. For instance, in the context of employment and Title VII antidiscrimination provisions, disparate impact is not prohibited if it can be explained by a business necessity. 42 U.S.C., § 2000e-2(k)(1)(A) (2012).
48. See Alexander, supra note 38, at 168 (discussing this distinction).
49. Achieving this analytical task is obviously easier said than done. See Barocas & Selbst, supra note 13, at 32 (noting the “four-fifths rule” applied in the employment context); Uniform Guidelines on Employment Selection Procedures, 29 C.F.R. § 1607.4(D) (2014).
50. See infra text accompanying notes 103–105.
51. Yoshino, supra note 12, at 756 (drawing out the five classifications the Supreme Court has formally accorded heightened scrutiny: “race, national origin, alienage, sex, and nonmarital parentage”). Yoshino argues there will be no more classes accepted by law. Id. at 757 (“this canon has closed”).
52. Id. at 757 n.73 (explaining that some states, such as California, have added “sexual orientation” as a protected class). In addition, age and disability have been recognized as classes worthy of protection under U.S. federal laws. For a discussion as to the groups recognized in the U.S. and other countries, see Sandra Fredman, European Comm’n, European Comm’n Directorate-General for Justice, Comparative Study of Anti-discrimination and Equality Laws of the US, Canada, South Africa and India 25 (2012), available at http://ec.europa.eu/justice/discrimination/files/comparative_study_ad_equality_laws_of_us_canada _sa_india_en.pdf.
53. Distinguishing between these notions of discrimination when structuring an analytical framework is a common analytical move among scholars discussing discrimination. For instance, Richard Ford explains that “[w]e have three distinct but related theories of discrimination: formal discrimination, discriminatory intent, and discriminatory effects,” thus mapping out a similar framework to the one noted in the text. Richard Ford, The Race Card 183 (2008) (emphasis in original).
54. For a discussion advocating some exceptions for using these factors when the consequences of neglecting them would prove “catastrophic” and their use will substantially increase the process’s effectiveness, see Schauer, supra note 35, at 186. This argument mostly pertains to the realm of national security and thus will be set aside in the context of this Article.
55. See Ford, supra note 53, at 180 (“Today, facial discrimination is rare. Because the law flatly forbids it with very few exceptions and because it is conspicuous when it happens, few people pass facially discriminatory laws or adopt facially discriminatory policies.”).
56. For more on this issue, see generally Richard Primus, The Future of Disparate Impact, 108 Mich. L. Rev. 1341 (2010); Ford, supra note 53, at 245–63.
57. See Schauer, supra note 35, at 203–04 (discussing the basic Aristotelian concept of equality which is a measure of fairness).
58. Alexander, supra note 38, at 159.
59. Barocas & Selbst, supra note 13, at 25. For a discussion on the degrading nature of racially discriminating laws and the importance to reduce this effect, see Ford, supra note 53, at 180–83 (explaining the insult caused to minorities when learning that others perceive them as inferior); Alexander, supra note 38, at 192 (discussing the insult biases entail); Jed Rubenfeld, Affirmative Action, 107 Yale L.J. 427-467 (1997).
60. One might note a possible caveat that this harm is reduced in the situation at hand, given its secretive nature. Yet this response is unacceptable. Not only might information regarding such conduct leak, the fact that those carrying out the scheme know of it generates substantial harm.
61. For an additional discussion of these elements, see infra notes 108–121 and accompanying text.
62. This form of discrimination is referred to as “first generation discrimination.” See Sturm, supra note 15, at 461.
63. See Ayres, supra note 46, at 843–45. Note that in some of these cases, minorities have chosen to discriminate against minorities as well for such reasons.
64. For a full description of this process, see Zarsky, supra note 5, at 9–15.
65. This, in fact, will require the scoring process to be deemed interpretable—understandable at least to the firm’s own analysts. For a broader explanation of the concept of interpretability and its implications, see Tal Z. Zarsky, Transparent Predictions, 2013 U. Ill. L. Rev. 1503-1519 (2013).
66. See supra text accompanying notes 17–22.
67. See supra text accompanying notes 17–22.
68. Citron & Pasquale, supra note 1, at 4.
69. Yoshino, supra note 12, at 768; Sturm, supra note 15, at 468; see also Kenj Yoshino, The New Equal Protection, Const. in 2020 (Dec. 1, 2004), http://constitutionin2020.blogspot.co.il/ 2004/12/new-equal-protection-post-by-kenji.html.
70. Note again that the following is a theoretical, not a doctrinal analysis. Legal doctrine has limited the meaning of “intent” in this context. See infra text accompanying note 84.
71. Barocas & Selbst, supra note 13, at 21–23.
72. Sturm, supra note 15, at 469 (“Exclusion increasingly results… as a byproduct of ongoing interactions shaped by the structures of day-to-day decision-making and workplace relationships.”).
73. See Ronen Perry & Tal Z. Zarsky, Queues in Law, 99 Iowa L. Rev. 1596, 1608–14 (2014) (explaining the analytical foundations of negative and positive equality and pointing to other relevant sources).
74. For a discussion in the context of employment, see Sturm, supra note 15, at 473.
75. See supra text accompanying note 58.
76. Citron & Pasquale, supra note 1, at 28.
77. Id. at 25.
78. See Jerry Kang & Kristin Lane, Seeing Through Colorblindness: Implicit Bias and the Law, 58 UCLA L. Rev. 465, 484–85 (2010) (noting studies that indicate implicit racial bias and discrimination in employment); Amy L. Wax, Discrimination as Accident, 74 Ind. L.J. 1129, 1138 (1999) (discussing the role unconscious bias plays in various contexts and how it might amount to disparate treatment). But see Gregory Mitchell & Philip E. Tetlock, Antidiscrimination Law and the Perils of Mindreading, 67 Ohio St. L.J. 1023, 1108–10 (2006) (calling for caution when applying the findings indicating biases in the lab to actual policy decisions which pertain to and impact actual interactions in the field).
79. For a famous discussion of such bias against married couples in a somewhat different context, see Batya Friedman & Helen Nissenbaum, Bias in Computer Systems, in Human Values and the Design of Computer Technology 21-30 (Batya Friedman ed., 1997) (one can further speculate that given the preference the system gave to the choices of the couple’s predefined “leading member” this bias actually adversely impacted women).
80. Zarsky, supra note 65, at 1518–19; see also Steve Lohr, For Big-Data Scientists, ‘Janitor Work’ Is Key Hurdle to Insights, N.Y. Times (Aug. 17, 2014, 4:55 PM), http://www.nytimes.com/2014/08/18/technology/for-big-data-scientists-hurdle-to-insights-is-janitor-work.html (explaining that the analysis of big data first requires a great deal of manual work—fifty to eighty percent of the overall time of analysis—collecting and preparing data for the subsequent process. These often ignored preparatory steps allow for a variety of instances in which the biases discussed could impact the scoring process.).
81. See Alexander, supra note 38, at 179–83, for an interesting discussion as to when this form of discrimination should be considered a conscious one.
82. See id. at 181.
83. See supra text accompanying note 60.
84. Kang & Lane, supra note 78, at 500 (demonstrating how de-biasing could be achieved via juror education to battle bias in the courtroom).
85. This measure is counter-intuitive, as one would assume that enhanced automation would generate a greater, rather than a lesser, amount of this form of discrimination. Automation is often considered to invoke inherent opacity, which in turn will allow for various forms of implicit discrimination such as “masking” to unfold.
86. For an analysis of the standard for “recklessness” in the context of discrimination in employment, see generally Joseph A. Seiner, Punitive Damages, Due Process, and Employment Discrimination, 97 Iowa L. Rev. 473 (2012).
87. A possible difficulty might arise in establishing whether those relying on the datasets are indeed reckless, or merely negligent. Applying the “reckless” paradigm to this juncture allows for easily including this discussion within the broader topic of intentional discrimination. For a discussion of discrimination via negligence in a somewhat different context, see David Benjamin Oppenheimer, Negligent Discrimination, 141 U. Pa. L. Rev. 899 (1993) (focusing on workplace discrimination).
88. See Barocas & Selbst, supra note 13, at 13. For a different form of misrepresentations of minorities, see Kate Crawford, The Hidden Biases in Big Data, Harv. Bus. Rev. Blog Network (April 1, 2013, 2:00 PM), http://blogs.hbr.org/2013/04/the-hidden-biases-in-big-data/.
89. See, e.g., Bernard E. Harcourt, Against Prediction: Profiling, Policing, and Punishing in an Actuarial Age 29–30 (2007).
90. Current law only sees “intent” as conscious choices. See Linda H. Krieger, The Content of Our Categories: A Cognitive Bias Approach to Discrimination and Equal Employment Opportunity, 47 Stan. L. Rev. 1161-1231 (1995).
91. Analyzing the normative shortcomings of those whose actions contributed to the creation of the tainted dataset calls for a distinct discussion which is unnecessary at this juncture.
92. For a discussion of duties to act in a broad variety of settings related to discrimination law, see Oppenheimer, supra note 87, at 936.
93. It is possible that the disclosure need not include the actual datasets but merely indicate the data sources used. Thereafter, experts can inform the scoring entity that such sources are historically tainted.
94. See Andrea Romei & Salvatore Ruggieri, A Multidisciplinary Survey on Discrimination Analysis, Knowledge Eng’g Rev., 2013, at 39–40; infra text accompanying notes 126–127.
95. See Citron & Pasquale, supra note 1, at 5, 17 (noting certain credit companies have refused to divulge their algorithms on the basis of trade secret protections); id. at 26-30 (discounting concerns that divulging the logic of predictive algorithms would result in individuals gaming those systems).
96. Barocas & Selbst, supra note 13, at 19.
97. Dothard v. Rawlinson, 433 U.S. 321, 323–24 (1977).
98. Additional policy work is required to set parameters as to how to distinguish between an unacceptably structured blatant proxy, and a simple and less accurate one that would be allowed.
99. Adverse outcomes towards minorities might follow as well, yet this section focuses on harms related to intent. The extent to which outcome-based concerns are sufficient to generate a regulatory response is discussed infra, Part II.C.
100. This description is not entirely hypothetical, but reflects recent lawsuits against various entities such as lenders and insurance companies. For one example of such a lawsuit, see Citron & Pasquale, supra note 1, at 15.
101. In many instances, these issues are not resolved by courts, but settled outside of them. See, e.g., Charlie Savage, Wells Fargo Will Settle Mortgage Bias Charges, N.Y. Times, July 13, 2012, at B3.
102. See Civil Rights Act of 1964, Pub. L. No. 88-352, 78 Stat. 253 (codified as amended at 42 U.S.C. §§ 2000e to 2000e-17 (2012); see generally Primus, supra note 56, for a discussion of the proper way to understand disparate impact provisions in view of a recent Supreme Court precedent; Richard A. Primus, Equal Protection and Disparate Impact: Round Three, 117 Harv. L. Rev. 493-515 (2003) (drawing out the legislative motives behind disparate impact doctrine).
103. 42 U.S.C. § 2000e-2(k)(1)(A)(i).
104. See Barocas & Selbst, supra note 13, at 32 (“[T]he definitions of … business necessity have never been clear….”).
105. See Washington v. Davis, 426 U.S. 229-242 (1976); Stacy E. Seicshnaydre, Is the Road to Disparate Impact Paved with Good Intentions?: Stuck on State of Mind in Antidiscrimination Law, 42 Wake Forest L. Rev. 1141, 1154 n.65 (2007). Note, however, that other countries have broader recognition and protection for disparate impact. See Fredman, supra note 52, at 50–55.
106. See Primus, supra note 56, at 1376 (referring to this rationale as an “evidentiary dragnet”).He also sees this rationale as weak, noting that the rule does not include a “good faith” exception. Id.
107. Alexander and Cole make a similar point regarding the logic of applying an overall ban on discrimination—be it rational or irrational, see Larry Alexander & Kevin Cole, Discrimination by Proxy, 14 Const. Comment. 453-456 (1997) (“[W]e would be…. wise to pay the cost of forbidding some rational racial discrimination in order to ensnare nonrational and irrational racial discrimination that might otherwise sneak through masquerading as rational discrimination.”).
108. Prominent antidiscrimination scholars have noted the difficulty with unwrapping the theoretical components of disparate impact. Richard Ford, for example, has noted that the theory of disparate impact remains “the most controversial doctrine in contemporary civil rights.” Ford, supra note 53, at 217.
109. Cf. Schauer, supra note 35, at 189 (explaining how the inclusion of race or ethnicity in an algorithm may produce racial or ethnic separation, and that this might be undesirable morally and socially); see also Primus, supra note 56, at 1376 (explaining that one of the justifications for prohibiting disparate impact is to “redress self-perpetuating racial hierarchies inherited from the past”).
110. Cf. Barocas & Selbst, supra note 13, at 54 (noting anti-subordination theory as a principle that “undergirded anti-discrimination law”).
111. Citron & Pasquale, supra note 1, at 18.
112. These are dynamics that reinforce themselves, possibly replicating a “caste system.” J.M. Balkin, The Constitution of Status, 106 Yale L.J. 2313-2353 (1997).
113. See Citron & Pasqual, supra note 1, at 7; cf. Schauer, supra note 35, at 189.
114. See generally Alexander & Cole, supra note 107, at 457 (“Members of groups discriminated against might suffer psychically even if not personally affected by the law.”). Note the difference between this argument and the one pertaining to the insult derived from intentional discrimination. Here, the stigma attaches as a result of the discriminatory outcome, regardless of the perceived or actual intent of those carrying out the scoring process.
115. Alexander, supra note 38, at 162.
116. Cf. Lea Shepard, Toward a Stronger Financial History Antidiscrimination Norm, 53 B.C. L. Rev. 1695-1765 (2012) (noting the effect of low credit scores on racial equality and social mobility).
117. For protection with respect to creditors, see id. at 1729. With respect to employment, see Title VII of The Civil Rights Act of 1964, 42 U.S.C. §§ 2000e to 2000e-17 (2012).
118. For an example of such a setting, see Latanya Sweeney, Discrimination in Online Ad Delivery, 56(5) Comm. of the ACM 44 (2013), available at http://privacytools.seas.harvard.edu/files/privacytools/files/p44-sweeney.pdf.
119. See Zarsky, supra note 80, at 1562 (presenting a similar argument).
120. Alexander and Cole, supra note 107, at 457, make a similar argument regarding covert users of classifications.
121. James Grimmelmann, Regulation by Software, 114 Yale. L.J. 1719-1736 (2005) (explaining that when dealing with software on a case-by-case basis, it might be very difficult for those interacting with it to recognize the existence of systematic discrimination).
122. On the importance of explaining the proper meaning of statistical inferences and how they could be misunderstood in the age of data mining and data analysis, see Zarsky, supra note 65, at 1565.
123. See supra text accompanying notes 21–22.
124. See supra text accompanying notes 19–23.
125. See Romei & Ruggieri, supra note 94, at 39–40.
126. In their bibliographical review of these issues, Andrea Romei and Salvatore Ruggieri sum up the current strategies examined in the computer science realm as follows (while referring to a variety of sources noting these studies): “We categorize three non mutually-exclusive strategies toward discrimination prevention: (i) a controlled distortion of the training set (a pre-processing approach); (ii) a modification of the classification learning algorithm (an in-processing approach), by integrating anti-discrimination criteria within it; (iii) a post-processing of the classification model, once it has been extracted, to correct its decision criteria.” Andrea Romei & Salvatore Ruggieri, Discrimination Data Analysis: A Multi-disciplinary Bibliography, in Discrimination and Privacy in the Information Society: Data Mining and Profiling in Large Databases 109-122 (Bart Custers et al. eds., 2013) (citations omitted).
127. The first steps of doing so are carried out by Barocas and Selbst, supra note 13, at 51.
128. For instance, it is possible that such actions must be carried out by courts, but not private entities on their own. See Primus, supra note 56, at 1369. Another factor in the legality of these processes is whether there is a visible victim of discrimination when recalibrating these parameters. See id. at 1369. It is indeed possible that when tinkering with the scoring mechanisms no such visible victim exists, and therefore such actions are permissible.
129. See id. at 1362 (explaining how a recent Supreme Court decision might be understood to state that “any operation of the disparate impact standard is an equal protection problem,” thus meaning that it constitutes discrimination against the majority group).
130. In the context of gender and race, see Schauer, supra note 35, at 153 n.28. This argument is rejected by Alexander in a similar context. See Alexander, supra note 38, at 187–89 (noting that past wrongdoings do not “appear to bear on the morality of acting on present ordinary preferences,” beyond the notion that those subjected to past wrongs should receive reparations). But see Primus, supra note 102, at 523–32 (noting “integrating the workplace” as a possible justification for the disparate impact rules under Title VII).
131. See Alexander, supra note 38, at 189.
132. For a discussion of this issue, see Schauer, supra note 35, at 153.
133. Cynthia Dwork & Deirdre K. Mulligan, It’s Not Privacy and It’s Not Fair, 66 Stan L. Rev. Online 35-37 (2013), http://www.stanfordlawreview.org/sites/default/files/online/topics/ DworkMullliganSLR.pdf (“Protecting against this sort of discriminatory impact is advanced by data about legally protected statuses, since the ability to both build systems to avoid it and detect systems that encode it turns on statistics.”).
134. See Reva Siegel, Equality Talk: Antisubordination and Anticlassification Values in Constitutional Struggles Over Brown, 117 Harv. L. Rev. 1470, 1471–72 (2004) (noting that propositions for limiting the collection of such personal racial information have been struck down and further noting that courts have not struck down rules requiring the collection of such data). For an additional discussion of this proposition, see Ford, supra note 53, at 334.
135. For a recent case, where the lack of a proper methodology for measuring disparate impact led to the failure of this claim, see EEOC v. Kaplan Higher Educ. Corp., 748 F.3d 749 (6th Cir. 2014).
136. For a critical discussion of this term, see Ford, supra note 53, at 338.
137. Citron & Pasqual, supra note 1, at 32–33.
138. Id. at 24.
139. Id. at 32–33.
140. Id.
141. Alexander, supra note 38, at 185.
142. Id.
143. For a similar argument in the context of “general” discrimination theory, see Balkin, supra note 112, at 2359–60.
144. For a discussion as to the problems and limited utility of requiring that causation theories are to be established prior to including them in a predictive model, see Zarsky, Transparent Predicitions, supra note 65, at 1562-1567; Schonberger & Cukier, supra note 19, at 61–72 (2014) (discussing the limited role causation is destined to play in the age of “big data”).
145. Alexander, supra note 38, at 198.
146. See Lea Shepard, Seeking Solutions to Financial History Discrimination, 46 Conn. L. Rev. 993, 1003–04 (2014).
147. Id. at 1011.
148. Citron & Pasquale, supra note 1, at 24.
149. Edith Ramirez, FTC Chairwoman, Keynote Address at the Tech. Policy Inst. Aspen Forum, The Privacy Challenges of Big Data: A View from the Lifeguard’s Chair 7–8 (Aug. 19, 2013), available at http://www.ftc.gov/sites/default/files/documents/publicstatements/privacy-challenges-big-data-view-lifeguard%E2%80%99s-chair/130819bigdataaspen.pdf.
150. Indeed, the morality of the process is compromised when the errors are not distributed equally and a specific social segment, such as the poor, is subject to a higher rate of errors. See, e.g., Kate Crawford, Think Again: Big Data, Foreign Policy 4 (May 9, 2013, 3:49 PM), http://www.foreignpolicy.com/articles/2013/05/09/think_again_big_data (discussing Boston’s Street Bump App).
151. Ramirez, supra note 150, at 8.
152. For a discussion of the importance of considering alternatives of automated selection processes so to properly balance the understanding of the issues at hand, see Tal Z. Zarsky, Governmental Data Mining and Its Alternatives, 116 Penn. St. L. Rev. 285-310 (2011).
153. Schauer, supra note 35, at 290–91.
154. Id.
155. Zarsky, supra note 5, at 27–29. But cf. Citron & Pasquale, supra note 1, at 4 (disagreeing with the argument voiced in the text).
156. Others have provided additional reasons why racial discrimination concerns might diminish in the age of big data—most notably Lior Strahilevitz. See generally Lior J. Strahilevitz, Privacy Versus Antidiscrimination, 75 U. Chi. L. Rev. 363-364 (2008) (explaining that increasing the availability of negative information regarding individuals will reduce decision makers’ reliance on information regarding groups, which in turn often constitutes racial statistical discrimination). For a review of this strand in the literature, see Scott R. Peppet, Regulating the Internet of Things: First Steps Toward Managing Discrimination, Privacy, Security & Consent, Tex. L. Rev. 34 n.191 (forthcoming 2014), available at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2409074.