Intermediaries and hate speech: Fostering digital citizenship for our information age

Boston University Law Review

Volumn 91, Issue 4, 2011, Pages 1435-1484

  Citron, Danielle Keats ^a   Norton, Helen ^b  

^a UNIVERSITY OF MARYLAND SCHOOL OF LAW   (United States)

^b UNIVERSITY OF COLORADO   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 80051868311     PISSN: 00068047     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (269)

1. Yaakov Lappin, 'Kill a Jew' Page on Facebook Sparks Furor, JERUSALEM POST, July 5, 2010, at 5.
2. Jesse Solomon, Hate Speech Infiltrates Social-Networking Sites, Report Says, CNN (Mar. 15, 2010, 4:37 PM), http://www.cnn. com/2010/TECH/03/15/hate. speech.social. networks/index.html.
3. Maria Seminerio, "Hate Filter" Tackles Racist Sites, ZDNet (Nov. 12, 1998, 3:29 PM), http://www.zdnet.co.uk/news/networking/1998/11/12/us- report-andquothatefilterand quot-tackles-racist-sites-2069870/.
4. Mark MacAskill & Marcello Mega, YouTube Cuts Murder Race-Hate Clips, Sunday Times (London) (Sept. 28, 2008), http://www.timesonline.co.uk/tol/news/ uk/scotland/article4837923.ece.
5. Theresa Howard, Online Hate Speech: It's Difficult to Police, USA Today, Oct. 2, 2009, at 4D.
6. See generally Simon Wiesenthal Center, Facebook, Youtube + How Social Media Outlets Impact Digital Terrorism and Hate (2009)
7. (providing screenshots of social media websites promoting hate). Hate groups recruit new members on popular social network sites like YouTube and Facebook. Social Networks Are New Sites for Hate Speech, REUTERS, May 13, 2009, http://www.pcmag.com/article2/0, 2817, 2347004, 00.asp.
8. aborn88, How to Kill a Beaner, YouTube (June 1, 2008), http://www.youtube.com/watch?v=Dq-tUPOGp8w.
9. Liz Nordlinger, Cartman Started It, St. PETERSBURG TIMES (Fla.), Feb. 25, 2010, at 8;
10. Matthew Moore, Facebook 'Kick a Ginger' Campaign Prompts Attacks on Redheads, Telegraph (U. K.) (Nov. 22, 2008, 12:47 AM), http://www.telegraph.co. uk/news/world news/northamerica/canada/3498766/Facebook-Kick-a-Ginger-campaign- prornpts-attackson-redheads.html.
11. Phil Bradley, Facebook Group: Hitting Women, Phil Bradley's Weblog (Feb. 18, 2010), http://philbradley.typepad.com/phil-bradleys-weblog/2010/02/facebook- group-hit ting-women. html (reporting that as of February 10, 2010 the Facebook page Hitting Women remained on Facebook);
12. Julie Ross Godar, Facebook and Hate Speech: Are You a Fan of Hitting Women?, BlogHer (Feb. 18, 2010, 5:39 PM). As of December 20, 2010, the Facebook group Hitting Women was no longer available.
13. Corilyn Shropshire, Facebook Wrestles with Anti-Semitism, Hous. Chron., May 15, 2009, at 6.
14. David Badash, Facebook or Hate Book? Facebook Shuts Down Anti-Gay Hate Groups!, The New Civil Rights Movement (Mar. 9, 2010), http://thenewcivilrights movement.com/facebook-or-hate-book-facebook-shuts-down-anti-gay-hategroups/ successes/2010/03/09/8828.
15. See The Murder of Alan Berg: 25 Years Later, Denver Post, June 18, 2009, at A-01.
16. The Simon Wiesenthal Center has documented the extraordinary increase in online hate over the past ten years. Simon Wiesenthal Center, supra note 9, at 1 (discussing its growth from one website in 1995 to 10, 000 today and the 25% increase in sites devoted to hate in the past year alone). This growth mirrors the escalating power and range of the technologies that facilitate the distribution of expression generally, including but not limited to digital hate.
17. See Nathan Myhrvold, Moore's Law Corollary: Pixel Power, N. Y. TIMES, June 7, 2006, at G3 (explaining that the speed and breath of computing power doubles every eighteen months). As Microsoft founder Bill Gates explains of the information age, "we're always in a time of utter change, maybe even accelerating change."
18. John Markoff, Gates's Lieutenants Look Ahead, Hoping to Avoid Other Companies' Mistakes, N. Y. TIMES, June 17, 2006, at C1.
19. For various proposals to modify the First Amendment standards to be applied to governmental regulation of online hate speech, see Jennifer L. Brenner, True Threats - A More Appropriate Standard for Analyzing First Amendment Protection and Free Speech When Violence Is Perpetrated over the Internet, 78 N. D. L. Rev. 753, 783 (2002);
20. John P. Cronan, The Next Challenge for the First Amendment: The Framework for an Internet Incitement Standard, 51 Cath. U. L. Rev. 425, 428 (2002);
21. Nancy S. Kim, Web Site Proprietorship and Online Harassment, 2009 Utah L. Rev. 993, 997 (urging courts to impose tort liability upon website sponsors "for creating unreasonable business models" by failing to adopt "reasonable measures" to prevent foreseeable harm of online harassment).
22. See, e.g., R. A. V. v. City of St. Paul, 505 U. S. 377, 391 (1992) (holding that a city ordinance that prohibited expression that "arouses anger, alarm or resentment in others... on the basis of race, color, creed, religion, or gender" impermissibly discriminated on the basis of viewpoint in violation of the First Amendment (internal quotation marks omitted)).
23. Although this Article focuses only on private intermediaries' voluntary responses to cyber hate, we do not discount the possibility that government might have a role to play regarding the perpetrators of digital hate in at least some circumstances. See, e.g., Danielle Keats Citron, Cyber Civil Rights, 89 B. U. L. Rev. 61, 86-95 (2009) [hereinafter Citron, Cyber Civil Rights] (exploring law's coercive role in deterring and remedying cyber harassment);
24. Danielle Keats Citron, Law's Expressive Value in Combating Cyber Gender Harassment, 108 MICH. L. Rev. 373, 404-14 (2009) [hereinafter Citron, Law's Expressive Value] (documenting the expressive value of a cyber civil rights agenda in addressing cyber gender harassment).
25. See David S. Ardia, Free Speech Savior or Shield for Scoundrels: An Empirical Study of Intermediary Immunity Under Section 230 of the Communications Decency Act, 43 LOY. L. A. L. Rev. 373, 386 (2010) (suggesting that intermediaries fall into three general categories: communication conduits, content hosts, and search/application providers). This Article focuses on voluntary measures available to a specific subset of internet intermediaries - content hosts and search/application providers - given their unique role in hosting online communities and in linking individuals to them. We leave to the side intermediaries that primarily serve as communication conduits (such as internet service and broadband providers) that we see as more akin to the phone company or the postal service in that they carry, but do not typically mediate, expressive content.
26. This Article refers to sites that enable the production and sharing of digital content in mediated social settings as "social media." See Danielle Keats Citron, Fulfilling Government 2.0's Promise with Robust Privacy Protections, 78 Geo. Wash. L. Rev. 822, 824 n. 12 (2010) (explaining that social media include social-network sites, video-sharing sites, photo-sharing sites, and the like).
27. This is not necessarily true outside of the United States, as many countries' laws require intermediaries to moderate content and to ensure its compliance with substantive restrictions. See, e.g., Wendy Seltzer, Remarks, The Politics of the Internet, 102 Am. Soc'Y Int'l L. Proc. 45, 45-47 (2008) (describing how some countries require internet service providers, search engines, and other intermediaries to prevent in-country users from reaching certain sites).
28. For arguments that intermediaries can also play a central role in responding to defamation and other reputational harms, see David S. Ardia, Reputation in a Networked World: Revisiting the Social Foundations of Defamation Law, 45 Harv. C. R.-C. L. L. Rev. 261, 264 (2010);
29. Daniel H. Kahn, Social Intermediaries: Creating a More Responsible Web Through Portable Identity, Cross-Web Reputation, and Code-Backed Norms, 11 Colum. Sci. & TECH. L. Rev. 176, 195-96 (2010).
30. See Jennifer Gordon & R. A. Lenhardt, Rethinking Work and Citizenship, 55 UCLA L. Rev. 1161, 1185 (2008) (arguing that citizenship requires a person's ability to participate in society in a meaningful manner).
31. In Lawrence Lessig's estimation, social norms may often regulate behavior as effectively as law. Lawrence Lessig, The Law of the Horse: What Cyberlaw Might Teach, 113 Harv. L. Rev. 501, 507 (1999);
32. Lawrence Lessig, The New Chicago School, 27 J. Legal Stud. 661, 669-70 (1998) [hereinafter Lessig, The New Chicago School] (explaining that institutions can shape behavior through the development of social norms, as well as through law, markets, and architecture). Nancy Kim similarly characterizes cyber harassment primarily as a failure of website operators' business norms, suggesting that tort law should encourage operators to engage in a range of preventive behaviors to deter online harassment. (Pubitemid 128429248)
33. Mark A. Lemley, The Law and Economics of Internet Norms, 73 Chi.-Kent L. Rev. 1257, 1260-61 (1998) (questioning the claim that reliance on norms is more effective than regulation in achieving cyberspace governance)
34. and Neil Weinstock Netanel, Cyberspace Self-Governance: A Skeptical View from Liberal Democratic Theory, 88 Calif. L. Rev. 395, 401-02 (2000) (concluding that selective governmental regulation of cyberspace will better realize liberal democratic ideals than cyberspace self-governance). In evaluating comparative costs and benefits, that debate largely assumes the freedom to choose between law and norms in a particular context. This Article, in contrast, focuses on a context where such a choice is often unavailable because law - i.e., government regulation of online hate speech - is constrained by the First Amendment.
35. Olmstead v. United States, 277 U. S. 438, 485 (1928) (Brandeis, J., dissenting).
36. Mary Ann Glendon, Rights Talk: The Impoverishment of Political Discourse 101-05 (1991) (exploring how silence can provide misleading lessons about social responsibility ethos).
37. Joanne Scott & Susan Sturm, Courts as Catalysts: Re-thinking the Judicial Role in New Governance, 13 Colum. J. Eur. L. 565, 566 (2006) ("New governance moves away from the idea of specific rights elaborated by formal legal bodies and enforced by judicially imposed sanctions. It locates responsibility for law-making in deliberative processes which are to be continually revised by participants in light of experience, and provides for accountability through transparency and peer review.").
38. See Orly Lobel, The Renew Deal: The Fall of Regulation and the Rise of Governance in Contemporary Legal Thought, 89 MINN. L. Rev. 342, 388, 389, 391 (2004) ("The governance model aims to create a flexible and fluid policy environment that fosters 'softer' processes that either replace or complement the traditional 'hard' ordering of the regulatory model[, such as]... social labeling, voluntary corporate codes of conduct, private accreditation, and certification by nongovernmental actors.... Flexibility implies variation in the communications of intention to control and discipline deviance. Less coercive sanctions can promote flexibility in implementation and compliance.").
39. Robert D. Putnam, Bowling Alone: The Collapse and Revival of American Community 137-47 (2000).
40. John Stuart Mill, Considerations on Representative Government, reprinted in Three Essays 143, 196-98 (Oxford Univ. Press 1975).
41. Alexis de Tocqueville, Democracy in America 501 (Harvey C. Mansfield & Delba Wintrop eds. and trans., 2000) (1835).
42. Richard Dagger, Civic Virtues: Rights, Citizenship, and Republican Liberalism 104 (1997).
43. Indeed, public participation and civic engagement are often viewed as essential for members of a democracy to form a citizenry. Jürgen Habermas, Between Facts and Norms: Contributions to a Discourse Theory of Law and Democracy 307-08 (William Rehg trans., The MIT Press 1996) (1992);
44. John Dewey, The Public and Its Problems 148-50 (1927).
45. John Dewey, The School as Social Centre, 3 The Elementary Sch. Tchr. 73, 76 (1902).
46. Sheldon S. Wolin, Politics and Vision: Continuity and Innovation in Western Political Thought 434 (1960).
47. Amitai Etzioni, On Virtual, Democratic Communities, in Community in THE Digital Age: Philosophy and Practice 225, 228-29 (Andrew Feenberg & Darin Barney eds., 2004) (explaining that the town of Blacksburg, Virginia has an online community called Blacksburg Electronic Village where various groups and neighborhoods post meetings, share information, and interact). Examples abound of online political engagement, including the use of social media to raise campaign funds and organize voters during the 2008 Presidential election.
48. Miki Caul Kittilson & Russell J. Dalton, Virtual Civil Society: The New Frontier of Social Capital?, Pol. Behav. (Oct. 7, 2010) (forthcoming), available at http://www.springerlink.com/content/740r3560j640080t/;
49. see also Nathaniel J. Gleicher, Money Bombs and Democratic Participation: Regulating Internet Fundraising, 70 Md. L. Rev. (forthcoming 2011) (manuscript at 5-6), available at http://ssm.com/abstract=16955 52.
50. Jacob Christensen, Managing Mondays: Facebook, a Viable Workplace Tool?, LINKED 2 Leadership Blog (Apr. 5, 2010), http://linked21eadership. com/2010/04/05/mm-facebooka-workplace-tool/;
51. Two on Facebook... FNN Video and Employee Groups, One Degree (Jan. 11, 2008), http://www.onedegree.ca/2008/01/two-on-facebook.html.
52. For a discussion of the relationship between workplace relationships and civic engagement, see Cynthia Estlund, Working Together: How Workplace Bonds Strengthen a Diverse Democracy. Estlund suggests, however, that the rise of internet technology in the workplace may weaken rather than strengthen these bonds.
53. Popular social network sites like Facebook and MySpace were originally organized around existing institutions like schools, universities, and towns to enhance existing social connections. Felicia Wu Song, Virtual Communities: Bowling Alone, Online Together 22 (Digital Formations No. 54 2009).
54. Benjamin R. Barber, Strong Democracy: Participatory Politics for a New Age 267 (1984).
55. Michael J. Sandel, Democracy's Discontent: America in Search of a Public Philosophy 314-15 (1996). Some groups pursue distinctly illiberal aims, as this Article explores.
56. Charlotte Garden, Labor Values Are First Amendment Values: Why Union Comprehensive Campaigns Are Protected Speech, 79 FORDHAM L. Rev. 2617, 2656-58 (2011).
57. see Harry C. Boyte, A Different Kind of Politics: John Dewey and the Meaning of Citizenship in the 21st Century. 12 GOOD SOC'Y, no. 2, 2003 at 1, 7. Dewey enlisted schools in the battle against bigotry: intolerance would lose force if exposed to "the ideas of others."
58. For an insightful discussion of schools as crucial speech-facilitating institutions, see Joseph Blocher, Institutions in the Marketplace of Ideas, 57 DUKE L. J. 821, 856-59 (2008).
59. Nicole B. Ellison et al., The Benefits of Facebook "Friends:" Social Capital and College Students' Use of Online Social Network Sites, 12 J. COMPUTER-MEDIATED Comm. 1143, 1161-62 (2007);
60. see also Sebastian Valenzuela et al., Lessons from Facebook: The Effect of Social Network Sites on College Students' Social Capital 33 (Apr. 5, 2008) (unpublished manuscript), available at http://online.journalism.utexas.edu/2008/ papers/Valenzuela.pdf (finding that Facebook users come from diverse backgrounds, contrary to the popular myth that they are typically female, upper-middle class students). Intermediaries of course also support bonding ties - those involving groups of similar backgrounds.
61. Anupam Chander, Whose Republic?, 69 U. Chi. L. Rev. 1479, 1481 (2002)
62. (reviewing Cass Sunstein, Republiccom (2001)). For an innovative conception of transnational cultural citizenship
63. see Sonia K. Katyal, The Dissident Citizen, 57 UCLA L. Rev. 1415, 1467-75 (2010).
64. Timothy Fort defines "mediating institutions" to mean "communities which socialize their members", and "require individuals to grasp their responsibilities to others, at least within their group, so that a person's very identity is developed." Timothy L. Fort, The Corporation as Mediating Institution: An Efficacious Synthesis of Stakeholder Theory and Corporate Constituency Statutes, 73 Notre Dame L. Rev. 173, 174-75 (1997);
65. see also Andrew Crane et al., Stakeholders as Citizens? Rethinking Rights, Participation, and Democracy, 53 J. Bus. Ethics 107, 108 (2004) (describing various conceptions of corporate citizenship as including "corporations as citizens; corporations as administrators of citizenship; and stakeholders as citizens").
66. The term "digital citizenship" can mean different things depending upon the community and audience. For instance, some political scientists have used the term to refer broadly to the "ability to participate in society online", arguing that disadvantaged groups cannot fully participate as citizens due to their limited access to the internet. Karen MOSSBERGER, CAROLINE J. TOLBERT & RAMONA S. MCNEAL, DIGITAL CITIZENSHIP: THE Internet, Society, and Participation 1 (2008). Intermediaries, too, have invoked the concept of digital citizenship as an aspiration for civil online behavior. Indeed, we first encountered this term in conversations with those in the industry who had already identified the facilitation of "digital citizenship" as a goal. Interview with Hemanshu Nigam, former Chief Safety Officer, MySpace (June 22, 2010).
67. Evgeny Morozov's work explores the related, though distinct, question of how democratic freedoms can be threatened by governmental abuse of networked technologies. See generally Evgeny Morozov, The Net Delusion: The Dark Side of Internet Freedom (2011).
68. Anupam Chander has also explored this question with great insight in Googling Freedom, 99 Calif. L. Rev. 1 (2011).
69. Timothy Zick, Speech Out of Doors: Preserving First Amendment Liberties in Public Places 304 (2009). Professor Zick explains that new technologies contribute to the phenomenon of "absent presence" where people occupy personal "technology bubbles" and disconnect from others who physically surround them.
70. See Kyu Ho Youm, First Amendment Law: Hate Speech, Equality, and Freedom of Expression, 51 J. Comm. 406, 406 (2001) (book review) (describing reports by Don Black - the "godfather of the Internet racist movement" - that the internet dramatically increased his ability to disseminate his views compared to his previous reliance on traditional print media).
71. See Adam G. Klein, A Space for Hate: The White Power Movement's Adaptation into Cyberspace 55 (2009) (describing "information laundering" to mean "the legitimizing factor of an interconnected information superhighway of web directories, research engines, news outlets, and social networks that collectively funnel into and out of today's hate websites"). Klein continues: For information-seekers, the result of this funneling process is a wider array of perspectives, and thus, a broader understanding of any given topic. However, for propaganda-providers like the white power movement, the same process inadvertently lends the credibility and reputation of authentic websites to those illegitimate few to which they are nonetheless connected. Such is the case with many of today's leading search engines like Google, that unwittingly filter directly into hate websites, or public networks like YouTube, which host their venomous content everyday.
72. See Mari J. Matsuda, Public Response to Racist Speech: Considering the Victim's Story, 87 Mich. L. Rev. 2320, 2352 n. 166 (1989) (describing history of escalating racist violence that accompanies racist speech);
73. Alexander Tsesis, Dignity and Speech: The Regulation of Hate Speech in a Democracy, 44 Wake Forest L. Rev. 499, 509-15 (2009) [hereinafter Tsesis, Dignity and Speech] (describing history of anti-Semitic and racist speech that incited or escalated violent acts);
74. Alexander Tsesis, The Empirical Shortcomings of First Amendment Jurisprudence: A Historical Perspective on the Power of Hate Speech, 40 Santa Clara L. Rev. 729, 740-55 (2000) (detailing the relationship between hate speech and acts of violence against Jews, Native Americans and African-Americans).
75. See, e.g., David Kretzmer, Freedom of Speech and Racism, 8 Cardozo L. Rev. 445, 463 (1987) (describing social science demonstrating the importance of speech as a precondition to acts of racial violence or scapegoating).
76. Hate speech that takes the form of "fighting words" may sometimes provoke violent responses from its targets in addition to inciting violence against them. See Charles R. Lawrence III, If He Hollers, Let Him Go: Regulating Racist Speech on Campus, 1990 DUKE L. J. 431, 452;
77. Ronald Turner, Regulating Hate Speech and the First Amendment: The Attractions of, and Objections to, an Explicit Harms-Based Analysis, 29 Ind. L. Rev. 257, 298-300 (1995) (describing violence provoked by use of the n-word or other face-to-face uses of particular racial or religious epithets).
78. Christopher Wolf, Racists, Bigots and the Law on the Internet: Internet Hate Speech and the Law, Anti-Defamation League, http://www.adl.org/Internet/ Internet-law3.asp (last visited Apr. 5, 2011). Smith killed former Northwestern University basketball coach Ricky Byrdsong and Indiana University student Won Joon Yoon and wounded six Orthodox Jews and three African-Americans.
79. Elizabeth Brackett, The Hate Crimes Question, PBS Online News Hour (Aug. 11, 1999), http://www.pbs.org/newshour/bb/law/julydec99/hate-8-ll.html. The internet helped make the WCOTC one of the fastest growing hate groups in the United States.
80. The Consequences of Right-Wing Extremism on the Internet: Inspiring Extremist Crimes, Anti-Defamation League, http://www.adl.org/internet/extremism- rw/inspiring. asp (last visited Apr. 5, 2011). WCOTC's website operator at the time of the rampage confirmed that Smith sent him "about five" email messages "congratulating" him on the group's websites and indicating that he regularly read them.
81. See Richard Delgado & David Yun, The Neoconservative Case Against Hate-Speech Regulation - Lively, D 'Souza, Gates, Carter, and the Toughlove Crowd, 47 Vand. L. Rev. 1807, 1822-23 (1994) (explaining how our culture has developed a host of narratives about overcoming hurt feelings while ignoring hurtful words that undermine victims' ability to respond and to mobilize effectively against hate);
82. Steven H. Shiffrin, Racist Speech, Outsider Jurisprudence, and the Meaning of America, 80 CORNELL L. Rev. 43, 86 (1994) ("[R]acial vilification can create a repressive environment in which the speech of people of color is chilled or not heard.");
83. Mike Adams, Facebook Devolves into Dark Web of Anonymous Hate Speech, NATURALNEWS (Aug. 26, 2010), https://www.natural news.com/029572-Facebook-hate- speech.html (stating that hate speech on Facebook has caused individuals who would otherwise be participating in the public discourse to close their accounts).
84. Kim Zetter, Court: Cyberbullying Threats Are Not Protected Speech, WIRED BLOG: Threat Level (Mar. 18, 2010, 3:23 PM), http://www.wired.com/threatlevel/ 2010/03/cyber bullying-not-protected/.
85. D. C. v. R. R., 106 Cal. Rptr. 3d 399, 405 (Ct. App. 2010).
86. Dahlia Lithwick, Fear of Blogging: Why Women Shouldn 't Apologize for Being Afraid of Threats on the Web, SLATE (May 4, 2007, 7:20 PM), http://www.slate.com/id/2165654/.
87. Greg Sandoval, Blogger Cancels Conference Appearance After Death Threats, CNET News Blog (Mar. 26, 2007), http://news.cnet.com/8301-10784-3-6170683-7. html.
88. Jessica Valenti, Women: How the Web Became a Sexists' Paradise, GUARDIAN (London), Apr. 6, 2007, at 16;
89. Blog Death Threats Spark Debate, BBC NEWS (Mar. 27, 2007), http://news.bbc.co.uk/go/pr/fr/-/2/hi/technology/6499095.stm.
90. Kathy Sierra, Comment to CCR Symposium: A Behavioral Argument for Stronger Protections, Concurring Opinions (Apr. 18, 2009, 2:25 PM), http://www.concurring opinions.com/archives/2009/04/ccr-symposium-a-l. htmlcomments.
91. Ryan Wilson, HUDALJ 03-98-0692-8 ¶¶ 2-3, 6 (July 19, 2000). For an excellent description and analysis of the case
92. see Catherine E. Smith, Intentional Infliction of Emotional Distress: An Old Arrow Targets the New Hate Hydra, 80 Denv. U. L. Rev. 1, 35-48 (2002).
93. Wilson, HUDALJ 03-98-0692-8, at ¶¶ 9-11.
94. De Wayne Wickham, They Suffer for Doing Right Thing, USA Today (May 16, 2000, 8:39 AM), http://www.usatoday.com/news/opinion/columnists/wickham/wick093. htm (explaining that Ms. Jouhari and her daughter have moved four times to ensure that posters do not find them).
95. Jeremy Waldron, Dignity and Defamation: The Visibility of Hate, 123 Harv. L. Rev. 1596, 1601 (2010).
96. R. A. Lenhardt, Understanding the Mark: Race, Stigma, and Equality in Context, 79 N. Y. U. L. Rev. 803, 844-48 (2004);
97. see also Kenneth L. Karst, Belonging to America: Equal Citizenship and the Constitution 3 (1989) (offering a principle of equal citizenship that suggests people are "entitled to be treated" as "respected, responsible, and participating member[s]"). Jennifer Gordon and R. A. Lenhardt's theory of belonging focuses on formal and informal pathways to the genuine possession and exercise of citizenship in the United States, including political participation, work, and education.
98. Ryan Wilson, HUDALJ 03-98-0692-8, at 24-25 (July 19, 2000).
99. Michele L. Ybarra et al., Linkages Between Internet and Other Media Violence with Seriously Violent Behavior by Youth, 122 PEDIATRICS 929, 933 (2008).
100. See Richard Delgado, Words that Wound: A Tort Action for Racial Insults, Epithets, and Name-Calling, 17 Harv. C. R.-C. L. L. Rev. 133, 147 (1982).
101. Cass R. Sunstein, Republic.com 2.0 145 (2007).
102. Kretzmer, supra note 68, at 505. In this way, prejudice and bigotry fostered by hate speech can produce conscious and unconscious behavioral consequences and thus intensify their targets' disadvantage. Jerry Kang, Trojan Horses of Race, 118 Harv. L. Rev. 1489, 1539-40 (2005) (examining the role played by racial stereotypes in mass media in creating and maintaining biases that result in discriminatory decision-making).
103. see also Erving Goffman, Stigma: Notes on the Management of Spoiled Identity 2-5 (1963) (discussing stigma as creating a spoiled social identity).
104. Danielle Keats Citron, Mainstreaming Privacy Torts, 98 Calif. L. Rev. 1805, 1813 (2010);
105. Jeffrey Rosen, The End of Forgetting, N. Y. Times, July 25, 2010, at MM30.
106. See, e.g., R. A. V. v. City of St. Paul, 505 U. S. 377, 391 (1992) (holding that city ordinance that prohibited expression that "arouses anger, alarm or resentment in others... on the basis of race, color, creed, religion, or gender" impermissibly discriminated on the basis of viewpoint in violation of the First Amendment). Indeed, as some thoughtful commentators have observed, regulatory efforts to constrain hate speech not only face constitutional challenges, but may impose instrumental costs of their own. For example, visible hate speech can remind readers and listeners of bigotry's prevalence and the need to enforce existing antidiscrimination laws.
107. Kingsley R. Browne, Title VII as Censorship: Hostile-Environment Harassment and the First Amendment, 52 Ohio St. L. J. 481, 542 (1991) (arguing that permitting hate speech can contribute to the elimination of prejudice because such speech will expose the poverty of those beliefs). Others suggest that hateful expression may play a role in preventing violence by allowing speakers to let off steam.
108. Vincent Blasi, The Teaching Function of the First Amendment, 87 Colum. L. Rev. 387, 408 (1987)
109. (reviewing Lee C. Bollinger, The Tolerant Society (1986)).
110. But see Dhammika Dharmapala & Richard H. McAdams, Words That Kill? An Economic Model of the Influence of Speech on Behavior (with Particular Reference to Hate Speech), 34 J. Leg. Stud. 93, 132 (2005) (discussing how raising the costs of engaging in hate speech may deter hate crime rather than increase the rate of hate crime). Refraining from regulating hate speech may avoid making martyrs ofand thus invigorating and multiplying - hateful speakers.
111. Graham Hughes, Prohibiting Incitement to Racial Discrimination, 16 U. TORONTO L. J. 361, 365 (1996) (suggesting that regulation creates martyrs and converts to the cause of hatred);
112. Larissa Barnett Lidsky, Where's the Harm?: Free Speech and the Regulation of Lies, 65 Wash. & LEE L. Rev. 1091, 1099-1100 (2008) (concluding that punishing Holocaust denial will paradoxically entrench that view and inspire stronger belief in conspiracy theories). So, too, the expression of hate speech might foster a certain capacity of mind, enabling us to confront our biases, master our irrational passions, and develop further tolerance ourselves.
113. Lee C. Bollinger, The Tolerant Society: Freedom of Speech and Extremist Speech in America 142, 173 (1986).
114. Matthew Shepard and James Byrd, Jr. Hate Crimes Prevention Act, Pub. L. No. 111-84, 123 Stat 2190 (2009)
115. (to be codified at 18 U. S. C. § 249).
116. H. R. Rep. No. 110-406 pt. 1, at 2 (2007).
117. See, e.g., Green v. Am. Online (AOL), 318 F.3d 465, 472 (3d Cir. 2003) (finding that private company AOL is not subject to constitutional free speech guarantees and has not been transformed into a state actor simply because it "provides a connection to the Internet on which government and taxpayer-funded websites are found");
118. Langdon v. Google, Inc., 474 F. Supp. 2d 622, 631 (D. Del. 2007) (ruling that Google, Yahoo!, and Microsoft are private companies not subject to constitutional free speech guarantees even though they may work with state actors like public universities).
119. 47 U. S. C. § 230 (c) (2006);
120. see also Zeran v. Am. Online, Inc., 129 F.3d 327, 330 (4th Cir. 1997)
121. (barring claims against an online service provider under § 230 because defendant did not create the allegedly tortious content). Intermediaries can incur liability for content that they create themselves (e.g., for their own postings that are defamatory or threatening), or for publishing content that violates copyright law. 47 U. S. C. § 230 (e) (1);
122. see also Wendy Seltzer, Free Speech Unmoored in Copyright's Safe Harbor: Chilling Effects of the DMCA on the First Amendment, 24 Harv. J. L. & Tech. 171, 228 (2010) (noting that § 230 "specifically excludes intellectual property and criminal claims from its protections"). For instance, Internet Service Providers (ISPs) and website operators can incur liability under the Digital Millennium Copyright Act for refusing to take down content that they have been notified violates copyright law, whereas they enjoy immunity from liability for defamatory postings created by others.
123. Such intermediaries may explain their actions under the traditional "shareholder primacy" view that understands the corporation's primary (and perhaps exclusive) objective as maximizing shareholder wealth. See, e.g., Mark J. Roe, The Shareholder Wealth Maximization Norm and Industrial Organization, 149 U. Pa. L. Rev. 2063, 2065 (2001); (Pubitemid 33656803)
124. For Whom Corporate Managers Are Trustees: A Note, 45 Harv. L. Rev. 1365, 1367-69 (1932). Along these lines, intermediaries' sense of the bottom-line benefits of addressing hate speech can be shaped by consumers' - i.e., users' - expectations.
125. Although Facebook has now overtaken MySpace in popularity, MySpace remains popular. Search Results for Myspace, Quantcast Corp., http://www.quantcast.com/search?q=myspace (last visited Apr. 16, 2011) (noting that MySpace is the 35th most popular site in the United States).
126. MySpace prohibits content that "promotes or otherwise incites racism, bigotry, hatred or physical harm of any kind against any group or individual... [or] exploits people in a sexual or violent manner." MySpace.com Terms of Use Agreement, MySpace. COM, http://www.myspace.com/help/ terms (last visited Apr. 7, 2011).
127. see also Paul Alan Levy, Stanley Fish Leads the Charge Against Immunity for Internet Hosts - But Ignores the Costs, CONSUMER L. & POL'Y Blog (Jan. 8, 2011), http://pubcit.typepad.com/clpblog/2011/01/stanley-fish-leads-the- charge-against-immunityfor-intemet-hosts-but-ignores-the-costs.html (arguing that websites that fail to provide protections against abuse will find "that the ordinary consumers whom they hope to serve will find it too uncomfortable to spend time on their sites, and their sites will lose social utility (and, perhaps more cynically, they know they will lose page views that help their ad revenue) ").
128. Such decisions may be justified as a matter of corporate law under the social entity theory of the corporation, which permits corporate decision-makers to consider and serve the interests of all the various constituencies affected by the corporation's operation. See Lisa M. Fairfax, Doing Well While Doing Good: Reassessing the Scope of Directors' Fiduciary Obligations in For-Profit Corporations with Non-Shareholder Beneficiaries, 59 Wash. & Lee L. Rev. 409, 412 (2002).
129. Yahoo! lists its company values as including "an infectious sense of mission to make an impact on society and empower consumers in ways never before possible. We are committed to serving both the Internet community and our own communities." Yahoo! - What We Value, Yahoo! Inc., http://docs.yahoo.com/ info/values/ (last visited Apr. 7, 2011). Yahoo! also states that it "empowers] people through corporate social responsibility programs, products, and services to make a positive impact on their communities."
130. Overview, Yahoo! Inc., http://pressroom.yahoo.net/pr/ycorp/overview.aspx (last visited June 1, 2011). Google makes clear that it will pursue policies that may conflict with shortterm shareholder economic gain but that it believes will benefit shareholders in the long term, and which may include benefits other than economic ones.
131. Google Inc., Initial Public Offering Letter: 'An Owner's Manual'for Google's Shareholders (Form S-1/A) (Aug. 18, 2004), available at http://investor.google.com/corporate/2004/ipo-founders-letter.html. Microsoft states: As a global company, we are accountable to millions of customers and stakeholders around the world. As we work to meet their needs, we are committed to creating value for our partners, employees, and wider society, and to managing our business sustainably. This commitment gives focus to our corporate citizenship work and helps us measure our performance over time.
132. Our Commitments, Microsoft, http://www.microsoft.com/about/ corporatecitizenship/enxf/our-commitments/ (last visited Apr. 7, 2011). Following the "Goals" link from that page brings a description of a link titled "Corporate Governance", which states that "considering the interests of other stakeholders - employees, customers, partners, suppliers, and the many communities around the world where we do business - is important to achieving the longterm interests of Microsoft shareholders."
133. Goals, Microsoft, http://www.microsoft.com/about/corporatecitizenship/en- xf/our-commitments/goals/ (last visited Apr. 7, 2011).
134. Rachel Whetstone, Free Expression and Controversial Content on the Web, THE Official Google Blog (Nov. 14, 2007, 3:58 PM), http://googleblog.blogspot. com/2007/11/free-expression-and-controversial.html.
135. Google Blogger requires users to refrain from promoting "hate or violence towards groups based on race, ethnicity, religion, disability, gender, age, veteran status, or sexual orientation/gender identity." Blogger Content Policy, Google, http://www.blogger.com/content.g (last visited Apr. 7, 2011). Google further admonishes Blogger users: "don't write a blog saying that members of Race X are criminals or advocating violence against followers of Religion Y."
136. Hate Book, Hate Book, http://www.hatebook.com/tos.php (last visited Apr. 7, 2011).
137. 47 U. S. C. § 230 (c) (2) (2006).
138. See, e.g., Oren Bracha & Frank Pasquale, Federal Search Commission? Access, Fairness, and Accountability in the Law of Search, 93 Cornell L. Rev. 1149, 1180-82 (2008) (deeming search engine Google a natural monopoly deserving of public regulation). In a series of articles, Frank Pasquale has argued that an Internet Intermediary Regulatory Council should oversee search engines and carriers, assisting the FCC and FTC in carrying out their present missions.
139. Frank Pasquale, Trusting (and Verifying) Online Intermediaries' Policing, in The Next Digital Decade: Essays on the Future of the Internet 347, 348 (Berin Szoka & Adam Marcus eds., 2010), available at http:// nextdigitaldecade.com/readbook/now. Pasquale argues that such regulation of Google is warranted given "Google's dominance of the general search market", the company's indispensable role in economic, cultural, and political life, and the opacity of its practices that immobilize consumer voice options.
140. See, e.g., Barbara van Schewick, Internet Architecture and Innovation 222-51 (2010).
141. See, e.g., Dawn C. Nunziato, Virtual Freedom: Net Neutrality and Free Speech in the Internet Age, at xiv-xv (2009) (arguing that Congress should pass a law, or require the FCC, to prohibit broadband providers from blocking legal content or applications and from engaging in various forms of discrimination and prioritization of packets and that perhaps law should regulate powerful search engines such as Google as well).
142. Past calls for transparency from these entities have focused on legitimacy concerns regarding stealth marketing and undisclosed political and cultural biases. These have included Frank Pasquale, Beyond Innovation and Competition: The Need for Qualified Transparency in Internet Intermediaries, 104 Nw. U. L. Rev. 105, 155 (2010) (discussing particular institutional solutions);
143. Frank Pasquale, Internet Nondiscrimination Principles: Commercial Ethics for Carriers and Search Engines, 2008 U. Chi. Legal F. 263, 268-69 (discussing regulation of search engines and social networks).
144. TOS agreements typically include not only an intermediary's hate speech policy (if any), but also its privacy policies, which typically notify users that they can opt-out of the collection of personally identifiable information. Commentators have criticized TOS privacy policies on the grounds that users do not pay attention to them and thus do not make meaningful choices about their privacy, which can lead to the collection and use of personal information. Danielle Citron, The Boucher Privacy Bill: A Little Something For Everyone Yet Nothing for All?, Concurring Opinions (June 13, 2010, 11:37 AM), http://www.con curringopinions.com/archives/2010/06/the-boucher-privacy-bill-a-little- something-foreveryone-yet-nothing-for-all.html. Ryan Calo's scholarship thoughtfully responds to critiques of notice provisions. See, e.g.
145. M. Ryan Calo, Against Notice Skepticism, 87 Notre Dame L. Rev. (forthcoming 2012), available at http://papers.ssrn. com/sol3/papers.cfm? abstract-id=l 790144%23.
146. Yahoo! Terms of Service, Yahoo! Inc., http://info.yahoo.com/legal/us/ yahoo/utos/utos-173.html (last visited Apr. 7, 2011). These policies apply to some of Yahool's services other than its search engine, such as its Flickr photo-sharing service.
147. For instance, Yahoo! explains that its Answer application is not "a soapbox to vent personal frustrations or rant about issues. We are a community of people with diverse beliefs, opinions, and backgrounds, so please be respectful and keep hateful and incendiary comments off Yahoo! Answers." Yahoo! Answers Community Guidelines, Yahoo! Inc., http://answers.yahoo.com/info/ community-guidelines (last visited Apr. 7, 2011).
148. Xbox LIVE Terms of Use, Microsoft, http://www.xbox.com/en-US/legal/ LiveTOU (last visited Apr. 7, 2011).
149. Xbox LIVE Code of Conduct, Microsoft, http://www.xbox.com/en-US/legal/ codeof conduct (last visited Apr. 7, 2011).
150. For instance, Twitter's Director of Program Development remarked: "What counts as name calling? There are sites that do employ teams of people that do that investigation... but we feel that's a job we wouldn't do well." Anick Jesdanun, On the Internet, Free Speech Is No Guarantee, HamptonRoads.com (July 21, 2008), http://hamptonroads.com/2008/07/internet-free- speech-no-guarantee.
151. See Frank Pasquale, Asterisk Revisited: Debating a Right of Reply on Search Results, 3 J. Bus. & Tech. L. 61, 73 (2008) (recognizing that resource constraints will limit intermediary duties, but recommending some such duties nevertheless);
152. Frank Pasquale, Rankings, Reductionism, and Responsibility, 54 Clev. St. L. Rev. 115, 117 (2006) (cautioning against too easy acceptance of reductionist presentations of reality by intermediaries).
153. Along these lines, Lisa Fairfax has documented the extent to which an institution's written commitments - such as a corporation's rhetoric evincing a responsibility to groups and interests beyond their shareholders - encourage and shape its actual behavior. Lisa M. Fairfax, Easier Said Than Done? A Corporate Law Theory for Actualizing Social Responsibility Rhetoric, 59 Fla. L. Rev. 771, 776 (2007) (debunking "the notion that corporate [social responsibility] rhetoric has no connection to actual practice [and] demonstrat[ing] the manner in which such rhetoric can be used strategically" to shape behavior). Fairfax continues: "when an individual expresses a commitment to a given idea or principle, the human preference for consistency generates internal and external pressures to engage in behavior consistent with that commitment."
154. See, e.g., John T. Nockleby, Hate Speech in Context: The Case of Verbal Threats, 42 Buff. L. Rev. 653, 708 (1994) (urging government regulation of only that universe of hate speech perceived by the listener as threatening violence);
155. Frederick Schauer, Uncoupling Free Speech, 92 COLUM. L. Rev. 1321, 1349 (1992) (defining actionable hate speech as "first, utterances intended to and likely to have the effect of inducing others to commit acts of violence or acts of unlawful discrimination based on the race, religion, gender, or sexual orientation of the victim; and, second, utterances addressed to and intended to harm the listener (or viewer) because of her race, religion, gender, or sexual orientation").
156. See Virginia v. Black, 538 U. S. 343, 365-66 (2003) (holding that the First Amendment permits states to prohibit individuals from burning crosses only when it is done "with the intent to intimidate.");
157. Watts v. United States, 394 U. S. 705, 705-08 (1969) (defining a true threat as that which a reasonable person would consider an expression of the speaker's intent to inflict bodily harm).
158. See United States v. White, 610 F.3d 956, 957, 962 (7th Cir. 2010) (holding that, under Supreme Court precedent, internet speech in which the poster intends to request or solicit a violent crime is not protected by the First Amendment, and declining to dismiss the government's solicitation case based on the defendant's website that "posted personal information about a juror who served on the Matthew Hale jury, along with postings calling for the use of violence on enemies of white supremacy").
159. See Planned Parenthood of Columbia/Willamette, Inc. v. Am. Coal, of Life Activists, 290 F.3d 1058, 1080 (9th Cir. 2002) (en banc) (holding that the Nuremberg Files' website could be characterized as an unprotected true threat, where the site listed the names, addresses, and license plate numbers of abortion providers, with the names of those who had been murdered lined through in black, and the names of those wounded highlighted in grey);
160. see also United States v. Fullmer, 584 F.3d 132, 156 (3d Cir. 2009) (concluding that animal rights activists' website included expression that instilled fears in its targets and thus could be prosecuted as true threats unprotected by the First Amendment).
161. Tom Spring, Digital Hate Speech Roars, PC WORLD, (Sept. 21, 2001, 7:00 PM), http://www.pcworld.com/article/63225/digital-hate-speech-roars.html.
162. Another web hosting company took down sites proclaiming that minorities should be hanged. Raphael Cohen-Almagor & Sharon Haleva-Amir, Bloody Wednesday in Dawson College - The Story of Kimveer Gill, or Why Should We Monitor Certain Websites to Prevent Murder, 2 Stud, in Ethics, L. & Tech. J. no. 3, 2008 at 1, 22-23.
163. James Joyner, Hal Turner and the Limits of Free Speech, OUTSIDE THE BELTWAY (Aug. 16, 2009), http://www.outsidethebeltway.com/hal-turner-and-the- limits-of-free-speech/;
164. see also Tom Hays, NJ Blogger Convicted of Threatening HI Judges, ASSOCIATED Press, Aug. 13, 2010, available at http://www.boston. com/news/local/connecticut/articles/2010/08/13/nj-blogger-convicted-of- threatening-ill-judges.
165. Susy Buchanan & David Holthouse, Extremists Advocate Murder of Immigrants, Politicians, S. Poverty L. Ctr. Intelligence Project (Mar. 30, 2006), http.//www.splcenter.org/intel/news/item.jsp?aid=49.
166. +149 Robin Pomeroy, Facebook Pulls Italian Neo-Nazi Pages After Outcry, REUTERS (Nov. 14, 2008), http://www.reuters.com/article/idUSTRE4AD3KZ20081114.
167. See, e.g., Timothy Zick, Cross Burning, Cockfighting, and Symbolic Meaning: Toward a First Amendment Ethnography, 45 Wm. & MARY L. Rev. 2261, 2346-49 (2004) (describing the use of context and cultural meaning to determine whether cross-burning communicates threats of violence or instead political protest).
168. Virginia v. Black, 538 U. S. 343, 365-66 (2003) (holding that the First Amendment permits states to prohibit individuals from burning crosses "with the intent to intimidate"). Alexander Tsesis argues that cultural symbols of hate, like burning crosses or swastikas, are effective at intimidation because such symbols trigger in victims a well-grounded fear of physical violence.
169. See Tsesis, Dignity and Speech, supra note 67, at 503-04 ("Destructive messages are particularly dangerous when they rely on historically established symbolism, such as burning crosses or swastikas, in order to kindle widely shared prejudices.").
170. Hate Speech and the Belie/net Community, BELIEFNET, http://www.beliefhet. com/Skipped/2004/06/Hate-Speech.aspx (last visited Apr. 8, 2011).
171. Scholars have cautioned that the First Amendment requires a very narrow understanding of this tort to ensure that government does not constrain offensive speech on the basis of viewpoint. See, e.g., Eugene Volokh, Freedom of Speech and the Intentional Infliction of Emotional Distress Tort, 2010 CARDOZO L. Rev. DE NOVO 300, 300-03;
172. Christina Wells, Regulating Offensiveness: Snyder v. Phelps, Emotion, and the First Amendment, 1 Calif. L. Rev. Circuit 71, 72 (2010). Along these lines, the Supreme Court has held that the First Amendment prohibits the tort's application to a defendant who "addressed matters of public import on public property, in a peaceful manner, in full compliance with the guidance of local officials."
173. Snyder v. Phelps, 131 S. Ct. 1207, 1220 (2011).
174. Benjamin Zipursky, Snyder v. Phelps, Outrageousness, and the Open Texture of Tort Law, 60 DePaul L. Rev. (forthcoming 2011) (manuscript at 31), available at http://ssrn. com/abstract=l 687688.
175. See Citron, Cyber Civil Rights, supra note 20, 87-88;
176. see also Nadine Strossen, The Tensions Between Regulating Workplace Harassment and the First Amendment: No Trump, 71 Chi.-Kent L. Rev. 701, 716-17 (1995) (suggesting that prescribable harassment under Title VII focus on workplace speech that directly targets a particular individual and that is so extreme that it amounts to intentional infliction of emotional distress). Along these lines, intermediaries' policies might also address defamatory hate speech. As the Supreme Court's First Amendment doctrine makes clear, the harms of defamatory speech - i.e., culpably false statements of fact that damage the target's reputation - are sufficiently great to justify its regulation by the government under certain circumstances.
177. See, e.g., N. Y. Times Co. v. Sullivan, 376 U. S. 254, 301-02 (1964);
178. see also Bollinger, supra note 107, at 186 (explaining that his tolerance theory permits the regulation of libel because it targets an individual for harm and the purposes of toleration are not served by insisting that an individual - rather than the community as a whole - bear the harm of speech activity).
179. Ryan Wilson, HUDALJ 03-98-0692-8, at 19 (July 19, 2000).
180. See Helen Norton, Regulating Cyberharassment: Some Thoughts on Sexual Harassment 2.0, 87 Denv. U. L. Rev. Online 11, 11-15 (2010) (identifying the difficulties in extending the First Amendment analysis applicable to governmental regulation of harassment at work and school to proposed government regulation of cyber harassment).
181. See R. A. V. v. City of St. Paul, 505 U. S. 377, 389 (1992) (describing Title VII's regulation of harassing speech in the workplace as permissible under the First Amendment as proscribing "sexually derogatory 'fighting words'," within "Title VII's general prohibition against sexual discrimination in employment practices");
182. Wisconsin v. Mitchell, 508 U. S. 476, 487 (1993) (explaining that the Court in R. A. V. "cited Title VII['s prohibition of sexual harassment] as an example of a permissible content-neutral regulation of conduct").
183. Harris v. Forklift Sys., Inc., 510 U. S. 17, 23 (1993) (identifying factors relevant to a conclusion that workplace harassment rises to the level of a Title VII violation).
184. Davis v. Monroe Cnty. Bd. of Educ, 526 U. S. 629, 633 (1999) (interpreting Title VI's prohibition on sex discrimination by federally funded educational activities);
185. see also Racial Incidents and Harassment Against Students at Educational Institutions; Investigative Guidance, 59 Fed. Reg. 11448, 11449 (Mar. 10, 1994) (interpreting Title VI's prohibition on race and national origin discrimination by federally funded activities to include "harassing conduct (e.g., physical, verbal, graphic, or written) that is sufficiently severe, pervasive or persistent so as to interfere with or limit the ability of an individual to participate in or benefit from the services, activities or privileges provided by a recipient").
186. Letter from Jane McAuliffe, President, Bryn Mawr Coll., to William Creeley, Dir. of Legal and Pub. Advocacy, Found, for Individual Rights in Educ. (July 17, 2010), available at http://www.thefire.org/index.php/article/12035. html;
187. see also Emory University Residence Life & Housing Standards & Policies 4-5, http://www.emory.edu/HOUSING/FORMS/form-ugrad.html (follow "download" hyperlink beside "Residence Life & Housing Policies") (last visited Apr. 8, 2011) (defining prohibited harassment to include "objectionable epithets, demeaning depiction or treatment, and threatening or actual abuse or harm").
188. College of William and Mary, Student Handbook 20 n. 3(2010), available at http://www.wm.edu/offices/deanofstudents/services/studentconduct/documents/ studenthand book.pdf.
189. As is true with virtually any proposed definition of hate speech, these efforts are not without controversy, as some argue that even private universities' efforts to address hate speech too often unwisely interfere with the unfettered flow of expression. See, e.g., Azhar Majeed, Defying the Constitution: The Rise, Persistence, and Prevalence of Campus Speech Codes, 7 Geo. J. L. & PUB. Pol'y 481, 483-84 (2009) (criticizing public and private university efforts to regulate hate speech on campus);
190. Nadine Strossen, Regulating Racist Speech on Campus: A Modest Proposal, 1990 Duke L. J. 484, 488-89.
191. See, e.g., J. Peter Byrne, Racial Insults and Free Speech Within the University, 79 Geo. L. J. 399, 416 (1991) (arguing that the university is "a distinct social entity, whose commitment to enhancing the quality of speech justifies setting minimum standards for the manner of speech among its members").
192. Colgate University Student Handbook 2010-2011, at 112-13(2010), available at http://www.colgate.edu/portaldata/imagegallerywww/939d3f45-4876-4ef5-b567- 1082dd4c 58e4/ImageGallery/Student-handbook-201O.pdf.
193. For additional proposals along these lines, see, for example, Yaman Akdeniz, Racism on the Internet 7 (2009) (emphasizing virulent, inflammatory language that is likely to inspire hatred in defining hate speech); Kretzmer, supra note 68, at 454 (urging that responsive hate speech policy focus on "threatening, abusive or insulting" speech that "is likely in the circumstances to stir up hatred against a racial, ethnic, or national group"); Matsuda, supra note 67, at 2357 (defining hate speech as constituting any message of inferiority, "directed at a historically oppressed group", that is "persecutorial, hateful, and degrading").
194. Under the title "Don't be sexist, racist, or a hater", Digg describes its hate speech policy as: "Would you talk to your mom or neighbor like that? Digg defines hate speech as speech intended to degrade, intimidate, or incite violence or prejudicial action against members of a protected group. For instance, racist or sexist content may be considered hate speech." Community Guidelines, DIGG, http://about.digg.com/guidelines (last visited Apr. 8, 2011).
195. YouTube appears to take a similar definitional approach. YouTube Community Guidelines, YouTube, http://www.youtube.com/t/community-guidelines?gl= GB&hl=en-GB (last visited Apr. 8, 2011) ("We encourage free speech and defend everyone's right to express unpopular points of view. But we don't permit hate speech (speech which attacks or demeans a group based on race or ethnic origin, religion, disability, gender, age, veteran status, and sexual orientation/gender identity).").
196. Terms & Conditions, MySpace, http://www.myspace.com/help/terms (last visited Apr. 8, 2011);
197. see also Nora Flanagan, Social Networking: A Place for Hate?, Imagine 2050 (May 19, 2009), http://imagine2050.newcomm.org/2009/05/19/social- networking-aplace-for-hate (describing MySpace's strict enforcement of its hate speech policy); Interview with Nigam, supra note 59. MySpace employs forum moderators who "keep an eye out for anti-semitism and derogatory comments."
198. Michael Arlington, MySpace Wants to Avoid this Whole Holocaust Denial Thing, TechCrunch Blog (May 12, 2009), http://techcrunch.com/2009/05/12/myspace- wants-to-avoid-this-whole-holocaust-denialthing/. Its terms of service explains that it "expressly reserves the right to remove your profile and/or deny, restrict, suspend, or terminate your access to all or any part of the MySpace Services if MySpace determines, in its sole discretion, that you have violated this Agreement."
199. Terms & Conditions, MySpace, http://www.myspace.com/help/terms (last visited Apr. 8, 2011).
200. See supra notes 149-151 and accompanying text (discussing Facebook's removal of pages threatening violence like Kick a Ginger). Assuming that Facebook understands removable hate speech to mean only that which threatens violence, it should say so more clearly in its actual hate speech policy, which instructs users not to "post content that: is hateful, threatening, or pornographic; incites violence; or contains nudity or graphic or gratuitous violence." Statement of Rights and Responsibilites, FACEBOOK, http://www.face book.com/terms.php (last visited June 1, 2011).
201. Video-sharing services and social network sites can remove content, precluding users from seeing them. Social media services can ban users by blocking their IP addresses. Cf. Google and Internet Control in China: A Nexus Between Human Rights and Trade?: Hearing Before the Cong.-Exec. Comm'n on China, 111th Cong. 68-76 (2010) (prepared testimony of Rebecca MacKinnon) (exploring the Chinese government's efforts to censor its citizens' online activities, including through the use of IP address blocking). Search engines can refuse to sell advertising to companies and thus limit their visibility to customers engaging in relevant searches.
202. See Floyd Norris, France Calls Google a Monopoly, N. Y. Times, July 2, 2010, at Bl (describing how Google refused to sell online advertising to French company Navx, which lets French drivers know where the police operate radar traps, because "Google found Navx's business distasteful" - thus search terms like "radar trap" no longer yielded advertisements for the company's product, whose sales "plunged").
203. (reviewing Facebook and Google/YouTube policies on removal of hate speech);
204. (chronicling Facebook's removal of "Kill a Jew" pages).
205. Cf. David A. Hoffman & Salil K. Mehra, Wikitruth Through Wikiorder, 59 Emory L. J. 151, 182 & n. 162 (2009) (discussing Wikipedia's warning of users to stop certain behaviors and placement on probation as well as banning of users).
206. See supra notes 80-85 and accompanying text. Chris Locke operated the blog where the threatening comments and doctored photographs were posted. Chris Locke, Re Kathy Sierra's Allegations, THE EGR WEBLOG (Mar. 27, 2007, 3:16 AM), http://www.rageboy.com/2007/03/re-kathy-sierras-allegations.html. He summarized his reaction to the posts in this way: [T]here were a couple posts - the ones Kathy mentions in her post - that were over the top. I didn't think for a minute that they were "threatening" - and again, they were not my doing - but when I saw mail from her objecting to them, I nuked the entire site rather than censor any individual. I was a conference host on the Well 15 years ago where the core ethos was acronymized to YOYOW - You Own Your Own Words. This has remained a guiding principle for me ever since. I will not take responsibility for what someone else said, nor will I censor what another individual wrote. However, it was clear that Sierra was upset, so it seemed the best course to make the whole site go away. Id. In our view, Locke, as the blog operator and relevant intermediary, should have taken down the posts as soon as he saw them in light of their clear potential to threaten and inflict fear and distress without offering any genuine opportunity for dialogue. Locke's "You Own Your Own Words" philosophy, applied there, seems ironic given that the posters wrote anonymously and thus avoided owning their own words to avoid bearing responsibility for the threats and doctored photographs. See id.
207. See State AG Questions Research on Child-to-Child Online Bullying, WASHINGTON Internet Daily (Warren Comm'ns News, Inc., Washington, D. C.), Dec. 12, 2008.
208. See 'Kill a Jew Day': Spike in Virulent Anti-Jewish Facebook Pages, The New YORK Blueprint (Oct. 3, 2010), http://nyblueprint.com/articles/view.aspx?id= 796. Unfortunately, however, "Kill a Jew" groups continue to appear on Facebook.
209. See Anomaly 100, Two Dozen 'Kill a Jew Day' Pages Found in the Last Seven Days, FreakOutNation (Oct. 3, 2010, 6:09 PM), http://freakoutnation. com/2010/10/03/twodozen-facebook-kill-a-jew-day-pages-found-in-the-last-seven- days.
210. See Jew Watch, http:/jewwatch.com (last visited Apr. 8, 2011).
211. See generally Jew Watch, Wikipedia, http://en. wikipedia.org/wiki/Jew- Watch (last visited Apr. 8, 2011).
212. John Brandon, Dropping the Bomb on Google, WIRED (May 11, 2004), http://www.wired.com/culture/lifestyle/news/2004/05/63380.
213. "Googlebombing" refers to a practice in which users can artificially inflate a page's search ranking by linking to a page in as many other pages as possible. James Grimmelmann, The Google Dilemma, 53 N. Y. U. Sch. L. Rev. 939, 942-43 (2008-09);
214. An Explanation of Our Search Results, GOOGLE, http://www.google.com/ explanation. html (last visited Apr. 8, 2011).
215. Apparently, however, Google does change search results for at least some purposes. Consider the example of a merchant who deliberately engaged in bad behavior because the sheer volume of negative mentions that then appeared on consumer advocacy websites improved his ranking in search results. David Segal, A Bully Finds a Pulpit on the Web, N. Y. Times, Nov. 28, 2010, at BUI. In response, Google changed its algorithm to penalize sites that others link because it provided "extremely poor user experience."
216. Amit Singhal, Being Bad to Your Customers Is Bad for Business, The Official Google Blog (Dec. 1, 2010, 12:06 PM), http://googleblog.blogspot.com/ 2010/12/being-bad-to-yourcustomers-is-bad-for.html. In a blog posting, Google explained that it developed an algorithmic solution to ensure that "being bad is, and hopefully will always be, bad for business in Google's search results."
217. For another example, see David Segal, The Dirty Little Secrets of Search, N. Y. TIMES, Feb. 13, 2011, at BUI (discussing Google's changes in search results to counter the effects of manipulative efforts to maximize J. C. Penney's search result rankings).
218. Google Search for "Jew", #Google, http://www.google.com/search? q=jew (last visited Mar. 26, 2011).
219. An Explanation of Our Search Results, supra note 203. If, however, you type "jew" into Google's German version, google.de, Jew Watch does not appear at all. Grimmelmann, supra note 201, at 948. At the bottom of the results page, a notice explains that Google has removed three results from the page. Id. Google changed its results because German law criminalizes incitement of hatred against minorities. Id. at 947. For a discussion of whether and how countries that have experienced genocide may take more aggressive approaches to hate speech
220. see Jennifer M. Allen & George H. Norris, Is Genocide Different? Dealing with Hate Speech in a Post-Genocide Society, 7 J. INT'L L. & INT'L R. EL. (forthcoming 2011), available at http://ssrn. com/abstract=1640812.
221. An Explanation of Our Search Results, supra note 203.
222. Saeed Ahmed, Google Apologizes for Results of 'Michelle Obama' Image Search, CNN (Nov. 25, 2009, 12:05 PM), http://www.cnn. com/2009/TECH/ll/25/ google.michelle. obama.controversy-2/index.html. A Google forum user flagged the picture. Id. Initially, Google de-indexed the website that posted the photograph on the grounds that "it could spread a malware virus." Id.
223. Bianca Bosker, Michelle Obama Pictures UPDATE: Offensive Image REMOVED, Google SORRY; Huffington Post (updated Mar. 18, 2010, 5:12 AM), http://www.huffingtonpost.com/2009/11/24/michelle-obama-photo-goog-n-368760. html.
224. See Barry Schwartz, Report Offensive Images on Google Does Not Work, SEARCH Engine Roundtable (Apr. 13, 2010, 7:54 AM), http://www.seroundtable.com/ archives/022010.html. Google's inaction on other cases has sparked much criticism.
225. See, e.g., Esra'a Al Shafei, Google Apologizes for Offending Jews Through Search Results, Mideast Youth (Mar. 21, 2007), http://www.mideastyouth.com/ 2007/03/21/google-apologizes-foroffending-jews-through-search-results.
226. Google has bought ads in at least one other instance outside of the context of hate speech: a Google user searching for "suicide" will encounter Google ads featuring suicide prevention resources. Noam Cohen, 'Suicide' Query Prompts Google to Offer Hotline, N. Y. Times, Apr. 5, 2010, at B6. The "icon of a red phone and the toll-free number for the National Suicide Prevention Hotline" appear over the linked results in a way that is "different and more prominent than an advertisement." Id. Google has also provided the telephone number for national poison control in searches like "poison emergency." Id. MySpace has gone further than putting up advertisements when users write about suicide. Interview with Nigam, supra note 59. As Hemanshu Nigam explained, when MySpace identified, or received notice of, users noting a desire to commit suicide, it would contact the National Suicide Prevention hotline and local police to recruit help for the users. Id. According to Nigam, MySpace's intervention helped prevent ninety-three suicides in 2009. Id
227. For additional arguments of the value of counter-speech by powerful speakers in response to hate, see Corey Brettschneider, When the State Speaks, What Should It Say? The Dilemmas of Freedom of Expression and Democratic Persuasion, 8 PERSP. ON Pol. 1005, 1005 (2010) (urging that "a proper theory of the freedom of expression obligates the legitimate state" to respond to hateful but protected speech by emphasizing the importance of respect for equality and dignity);
228. Helen Norton, Campaign Speech Law with a Twist: When Government Is the Speaker, Not the Regulator, 61 EMORY L. J. (forthcoming 2011) (urging government to engage in political speech on contested ballot measures that counters that of powerful private speakers);
229. Charlotte H. Taylor, Hate Speech and Government Speech, 12 U. Pa. J. Const. L. 1115, 1188 (2010) (urging government - generally prohibited by the First Amendment from banning hate speech - to engage in counter-speech to "help forge consensus about the nature of social practices" and "shift the ground under the hateful speaker's feet, robbing her of her confidence that she can invoke an entire system of subordination by using a few cheap words").
230. Sunstein, supra note 100, at 191.
231. Bollinger, supra note 107, at 172-73 (arguing that tolerating the expression of hatred may actually enhance our intellectual capacities and embolden civic courage).
232. Michael Arrington, Facebook Remains Stubbornly Proud of Position on Holocaust Denial, TechCrunch (May 12, 2009), http://techcrunch.com/2009/05/12/ facebookremains-stubbornly-proud-of-position-on-holocaust-denial.
233. See, e.g., Common Pro-N-Arguments, N-MANIA, http://niggermania.com/tom/ niggerarguments/niggerargumentstextpagetwo.htm (last visited Mar. 27, 2011) ("We hate n[-]s" because they are a "failed ape species.").
234. See, e.g., The Beast as Saint: The Truth About "Martin Luther King, Jr.", http://www.martinlutherking.org/thebeast.html (last visited Apr. 8, 2011) (arguing that Dr. Martin Luther King, Jr. was an academic cheat, communist, and sex addict).
235. See, e.g., Jew Watch, supra note 199.
236. Those costs would be comparatively minor in instances where an intermediary can automate counter-speech. Although an intermediary would need to incur the fixed cost of designing, or purchasing, responsive software, it would incur virtually no expenditures for the software's implementation in future cases. Cf. Danielle Keats Citron, Technological Due Process, 85 Wash. U. L. Rev. 1249, 1284-85 (2008).
237. Lorraine Tiven & Partners Against Hate, Hate on the Internet: A Response Guide for Educators and Families 20, 21 (2003) ("[J]ust as fashion editors and e-book publishers have started reaching out to elementary school children and teens... so have hate groups."
238. (quoting Tara McKelvey, Father and Son Target Kids in a Confederacy of Hate, USA Today, Jul. 16, 2001, at 3D)).
239. Simon Wiesenthal Center, supra note 9 (displaying screenshots of video games based in hate speech)
240. See Shiffrin, supra note 74, at 89 ("As an African American father once said to me when I spoke about the contribution of racist speech to the democratic dialogue, 'Tell that to my seven-year-old daughter.'").
241. Social science research demonstrates the significant harm caused by exposing children to violence. See, e.g., Amitai Etzioni, On Protecting Children
242. See, e.g., Susan Engel & Marlene Sandstrom, There's Only One Way to Stop a Bully, N. Y. Times, July 23, 2010, at A23.
243. This recalls the international standards organization for the World Wide Web - the W3C group - that identifies voluntary standards. See W3C Mission, World Wide Web Consortium, http://www.w3.org/Consortium/mission (last visited Apr. 8, 2011). We thank Neil Richards, Berin Szoka, and Chris Wolf for their helpful thoughts on this notion.
244. See, e.g., Fourth Law and Information Society Symposium: Hate Versus Democracy on the Internet, Fordham Law Event Calendar (Mar. 26, 2010), http://law2.fordham.edu/ihtml/cal-2uwcp-calendar-viewitem.ihtml?idc=10320.
245. Indeed, clearer and more transparent policies might have averted the situation where Facebook pulled Sarah Palin's controversial-but-not-hateful posting about proposals to build a mosque near Ground Zero after a number of users responded to a campaign encouraging them to click the "Report Note" hyperlink indicating the posting as hate speech. When Palin questioned the action, Facebook put it back up, apologized for pulling the comment, and promised to modify their process for taking down postings. See Brian Ries, My Facebook War with Palin, The Daily Beast (July 23, 2010, 11:10 AM), http://www.thedailybeast.com/blogs-and-stories/2010-07-23/palins-facebook- ground-zeromosque-post-how-it-disappeared/full/. Transparent policies might thus have additional salutary effects: the prevention of user manipulation of intermediaries' reporting tools.
246. Safety Center: Hateful Content, YouTube, http://www.google.com/support/ youtube/bin/answer.py?hl=en&answer=126264 (last visited Apr. 8, 2011).
247. See, e.g., Jon M. Garon, Wiki Authorship, Social Media, and the Curatorial Advantage, 1 Harv. J. Sports & Ent. L. 95, 99 (2010) ("By expanding opportunity for interaction and fostering behavioral norms of trust among users, these communications tools can expand the reach of social networks for mutual advantage.").
248. Clay Shirky, Cognitive Surplus: Creativity and Generosity in a Connected Age 165 (2010);
249. see also Elinor Ostrom, Governing the Commons: The Evolution of Institutions for Collective Action 88-102 (1990) (identifying the factors key to community regulation of common resources to include institutions well-equipped to gather information about the resource, forums to discuss its management, community participation in developing and enforcing the rules, and appropriate and graduated sanctions to discipline abuse).
250. Jonathan Zittrain, The Future of the Internet - And How to Stop It 134-35 (2008). These norms include "the three-revert rule", in which "an editor should not undo someone else's edits to an article more than three times in one day." Id. at 135.
251. Howard, supra note 7, at 4D. Facebook similarly urges users to provide the company's team of professional reviewers with "accurate and detailed information" so that "you can help us locate and remove abuse on the site as quickly and efficiently as possible." Jessica Ghastin, Responding to Abuse Reports More Effectively, The Facebook Blog (Oct. 14, 2009, 10:43 AM), http://blog.facebook. com/blog.php?post=144628037130.
252. Dan Raywood, YouTube Safety Mode Introduced to Block Inappropriate Content, SC Mag. (U. K.) (Feb. 15, 2010), http://www.scmagazineuk.com/youtube- safety-modeintroduced-to-block-inappropriate-content-but-claims-made-that-it- will-only-have-a-minorimpact/article/163784/;
253. Safety Center: Hateful Content, supra note 239.
254. Safety Center: Hateful Content, supra note 239.
255. How to Create Your Own Persona, MOZILLA, http://www.getpersonas.com/en- US/demo-create (last visited Apr. 8, 2011).
256. Personas is a feature in the Firefox browser that allows a user to select simple-to-use themes, known as Personas, to personalize their browser and status bar. Personas for Firefox, Wikipedia, http://en. wikipedia.org/wiki/ Personas-for-Firefox (last visited Apr. 8, 2011). Over 220, 000 Personas are available for users to choose from on the GetPersonas.com website.
257. E-mail from Julie Martin, Assoc. Gen. Counsel, Mozilla, to Danielle Citron, Professor of Law, Univ. of Maryland School of Law (Aug. 11, 2010) (on file with author).
258. Alexander Halavais, Search Engine Society 136 (2009). Google News Service provides news to users, a service that is separate from its work as a search engine. With respect to its search engine services, Google has also considered "expos[ing] user reviews and ratings for various merchants alongside their results" to address the problem of high rankings for merchants with "extremely poor user experience." Singhai, supra note 204. It ultimately rejected that course of action because it "would not demote poor quality merchants in our results and could still lead users to their websites." Id.
259. In Kind Advertising for Non-profit Organizations, GOOGLE GRANTS, http://www.google.com/grants/ (last visited Apr. 8, 2011) (explaining its "unique in-kind donation program awarding free AdWords advertising to select charitable organizations" that share its "philosophy of community service to help the world in areas such as science and technology").
260. See Lessig, The New Chicago School, supra note 27, at 662-63 (explaining that institutions can shape others' behavior through the development of social norms, as well as through law, markets, and architecture).
261. Jaron Lanier, You Are Not a Gadget 6 (2010) (as originally introduced, the web "emphasized responsibility, because only the owner of a website was able to make sure that their site was available to be visited").
262. Citron, Cyber Civil Rights, supra note 20, at 63-65;
263. See, e.g., Amy J. Schmitz, "Drive-Thru" Arbitration in the Digital Age: Empowering Consumers Through Binding ODR, 62 Baylor L. Rev. 178, 202-04 (2010) (discussing how consumers may be more likely to challenge corporate misbehavior through online vehicles that offer some measure of anonymity and thus protection from retaliation).
264. David Kirkpatrick, The Facebook Effect: The Insider Story of the Company that Is Connecting the World 13 (2010).
265. Richard A. Posner, Just Friends, New Republic, Aug. 12, 2010, at 27
266. (reviewing Kirkpatrick, supra note 272).
267. Chris Matyszczyk, Facebook: Holocaust Denial Repulsive and Ignorant, CNET News Blog (May 6, 2009, 1:04 PM), http://news.cnet.com/8301-17852-3-10234760- 71.html.
268. See, e.g., Levy, supra note 118 (explaining that because the New York Times, but not the Washington Post, devotes staff time to moderating comments before they appear on their blogs, "comments at the Times tend to be much more thoughtful - and hence worth reading - while comments on the Post's political blogs tend to be much more partisan and much more full of rant"); Roy Greenslade, Paper Puts Up a Paywallfor Comments, GREENSLADE Blog (July 13, 2010, 17:23), http://www.guardian. co.uk/media/greenslade/2010/jul/13/paywalls- us-press-publishing; Stephanie Goldberg, News Sites Reining in Nasty User Comments, CNN (July 19, 2000), http://articles.cnn. com/2010-07-19/tech/ commenting.on. news.sites-l-comments-news-sites-credit-card?-s=PM:TECH (discussing news websites like the Huffington Post that require registration or real names as a condition of commenting);
269. Richard Perez-Pefla, News Sites Rethink Anonymous Online Comments, N. Y. Times, Apr. 11, 2010, http://www.nytimes.com/2010/04/12/technology/12comments. html (discussing news websites that are considering real names or otherwise regulating comments to their online news content).