New Insights into Rental Housing Markets across the United States: Web Scraping and Analyzing Craigslist Rental Listings

Journal of Planning Education and Research

Volumn 37, Issue 4, 2017, Pages 457-476

  Boeing, Geoff ^a   Waddell, Paul ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

big data; Craigslist; data science; GIS; housing; urban economics; web scraping

Indexed keywords

EID: 85032331575     PISSN: 0739456X     EISSN: 15526577     Source Type: Journal    
DOI: 10.1177/0739456X16664789     Document Type: Article

References (65)

1. Alexa. 2015. “Craigslist.org Site Overview.” Alexa—Actionable Analytics for the Web. http://www.alexa.com/siteinfo/craigslist.org.
2. Anderson C., 2006. The Long Tail. New York: Hachette Books.
3. Aratani Y., Chau M., Wight V., Addy S., 2011. “Rent Burden, Housing Subsidies and the Well-Being of Children and Youth.” New York: National Center for Children in Poverty.
4. Batty M., Axhausen K., Giannotti F., Pozdnoukhov A., Bazzani A., Wachowicz M., Ouzounis G., Portugali Y., 2012. “Smart Cities of the Future.” European Physical Journal Special Topics 214 (1): 481–518. doi:10.1140/epjst/e2012-01703-3.
5. Bazuin J., Fraser J., 2013. “How the ACS Gets It Wrong: The Story of the American Community Survey and a Small, Inner City Neighborhood.” Applied Geography 45:292–302. doi:10.1016/j.apgeog.2013.08.013.
6. Belsky E., 2013. The Dream Lives on: The Future of Homeownership in America. W13-1. Cambridge, MA: Joint Center for Housing Studies.
7. Bettencourt L., 2013. “The Origins of Scaling in Cities.” Science 340 (6139): 1438–41. doi:10.1126/science.1235823.
8. Bettencourt L., West G., 2010. “A Unified Theory of Urban Living.” Nature 467 (7318): 912–13.
9. Boeing G., 2016. “Honolulu Rail Transit: International Lessons from Barcelona in Linking Form, Design, and Transportation.” Planext 2:28–47. doi:10.17418/planext.2016.3vol.02.
10. Brown C., 2013. “North Dakota Went Boom.” The New York Times Magazine, January 31. http://www.nytimes.com/2013/02/03/magazine/north-dakota-went-boom.html.
11. Brown K., 2014. “In the Rental Market, Craigslist May Be Undisruptable.” San Francisco Chronicle, July 5. http://www.sfgate.com/technology/article/In-the-rental-market-Craigslist-may-be-5601847.php.
12. Cayo M., Talbot T., 2003. “Positional Error in Automated Geocoding of Residential Addresses.” International Journal of Health Geographics 2 (10): 1–12.
13. Ching T.-Y., Ferreira J., 2015. “Smart Cities: Concepts, Perceptions and Lessons for Planners.” In Planning Support Systems and Smart Cities, edited by Geertman S., Ferreira J., Goodspeed R., Stillwell J., 145–68. Cham: Springer International.
14. Craigslist. 2015. “Craigslist Factsheet.” Craigslist. https://www.craigslist.org/about/factsheet.
15. Craigslist Inc. v. 3Taps Inc. 2013, 942 F. Supp. 2d 962. United States District Court for the Northern District of California.
16. Decker N., 2010. “Housing Discrimination and Craigslist.” The Current 14 (1): 43–57.
17. Dowall D., 1981. “Reducing the Cost Effects of Local Land Use Controls.” Journal of the American Planning Association 47 (2): 145–53. doi:10.1080/01944368108977099.
18. Evans-Cowley J., Griffin G., 2012. “Microparticipation with Social Media for Community Engagement in Transportation Planning.” Transportation Research Record 2307:90–98. doi:10.3141/2307-10.
19. Feng J., 2014. “A Week of Mining Seattle’s Craigslist Apartment Pricing.” Jay’s Life, December 23. http://www.racketracer.com/2014/12/23/a-week-of-seattles-craigslist-apartment-pricing/.
20. Folch D., Arribas-Bel D., Koschinsky J., Spielman S., 2014. “Uncertain Uncertainty: Spatial Variation in the Quality of American Community Survey Estimates.” Working Paper 1, Arizona State University, Tempe.
21. Fulton W., Shigley P., 2012. Guide to California Planning, 4th ed. Point Arena, CA: Solano Press.
22. Garde A., 2015. “Affordable by Design? Inclusionary Housing Insights from Southern California.” Journal of Planning Education and Research 0739456X15600033:1–16.
23. Goldman E., 2013. “Craigslist Wins Routine but Troubling Online Trespass to Chattels Ruling in 3Taps Case.” Technology & Marketing Law Blog, September 20. http://blog.ericgoldman.org/archives/2013/09/craigslist_wins_1.htm.
24. Goodspeed R., 2015. “Smart Cities: Moving beyond Urban Cybernetics to Tackle Wicked Problems.” Cambridge Journal of Regions, Economy and Society 8 (1): 79–92. doi:10.1093/cjres/rsu013.
25. Gordon L., 2006. Brokered Deception: The Hidden Perils of Online Real Estate Ads. New York: The Council of the City of New York.
26. Hanson A., Hawley Z., 2011. “Do Landlords Discriminate in the Rental Housing Market? Evidence from an Internet Field Experiment in US Cities.” Journal of Urban Economics 70 (2): 99–114. doi:10.1016/j.jue.2011.02.003.
27. Hau L., 2006. “Newspaper Killer.” Forbes, December 11. http://www.forbes.com/2006/12/08/newspaper-classifield-online-tech_cx-lh_1211craigslist.html.
28. Herman M., 2012. “Scraping Web Pages with Scrapy.” Michael Herman, November 5. http://mherman.org/blog/2012/11/05/scraping-web-pages-with-scrapy/.
29. Holeywell R., 2011. “North Dakota’s Oil Boom Is a Blessing and a Curse.” Governing, August.
30. Jiang B., 2015. “Geospatial Analysis Requires a Different Way of Thinking: The Problem of Spatial Heterogeneity.” GeoJournal 80 (1): 1–13. doi:10.1007/s10708-014-9537-y.
31. Jiang B., Thill J.-C., 2015. “Volunteered Geographic Information: Towards the Establishment of a New Paradigm.” Computers, Environment and Urban Systems 53:1–3. doi:10.1016/j.compenvurbsys.2015.09.011.
32. Joint Center for Housing Studies. 2013. America’s Rental Housing: Evolving Markets and Needs. Cambridge, MA: Harvard University.
33. Kay M., 2008. XSLT 2.0 and XPath 2.0 Programmer’s Reference, 4th ed. Indianapolis, IN: Wrox.
34. Kitchin R., McArdle G., 2016. “What Makes Big Data, Big Data? Exploring the Ontological Characteristics of 26 Datasets.” Big Data & Society 3 (1). doi:10.1177/2053951716631130.
35. Kroft K., Pope D., 2014. “Does Online Search Crowd Out Traditional Search and Improve Matching Efficiency? Evidence from Craigslist.” Journal of Labor Economics 32 (2): 259–303. doi:10.1086/673374.
36. Kurth R., 2007. “Striking a Balance between Protecting Civil Rights and Free Speech on the Internet: The Fair Housing Act vs. the Communications Decency Act.” Cardozo Arts & Entertainment Law Journal 25:805–36.
37. Laney D., 2001. 3D Data Management: Controlling Data Volume, Velocity, and Variety. Report 949. META Group.
38. Lutz C., 2014. “Build and Let Live: 40 Years of Affordable Housing in Aspen.” Aspen Journalism, May 27. http://aspenjournalism.org/2014/05/27/build-and-let-live-40-years-of-affordable-housing-in-aspen/.
39. Macdonald H., 2006. “The American Community Survey: Warmer (More Current), but Fuzzier (Less Precise) Than the Decennial Census.” Journal of the American Planning Association 72 (4): 491–503. doi:10.1080/01944360608976768.
40. Mallach A., 2010. Meeting the Challenge of Distressed Property Investors in America’s Neighborhoods. New York, NY: Local Initiatives Support Corporation.
41. Mayer-Schönberger V., Cukier K., 2013. Big Data: A Revolution That Will Transform How We Live, Work, and Think. Boston, MA: Eamon Dolan/Houghton Mifflin Harcourt.
42. Mitchell R., 2015. Web Scraping with Python: Collecting Data from the Modern Web. Sebastopol, CA: O’Reilly Media.
43. Mossberger K., Tolbert C., Bowen D., Jimenez B., 2012. “Unraveling Different Barriers to Internet Use: Urban Residents and Neighborhood Effects.” Urban Affairs Review 48 (6): 771–810.
44. Oliveri R., 2010. “Discriminatory Housing Advertisements On-Line: Lessons from Craigslist.” Indiana Law Review 43:1125–83.
45. O’Neil C., Schutt R., 2013. Doing Data Science. Sebastopol, CA: O’Reilly Media.
46. Pollock K., 2016. “Policy: Urban Physics.” Nature 531 (7594): S64–S66. doi:10.1038/531S64a.
47. Quigley J., Raphael S., 2004. “Is Housing Unaffordable? Why Isn’t It More Affordable?” Journal of Economic Perspectives 18 (1): 191–214.
48. Rae A., 2015. “Online Housing Search and the Geography of Submarkets.” Housing Studies 30 (3): 453–72. doi:10.1080/02673037.2014.974142.
49. Reid J., 2015. HTML5 Programmer’s Reference. New York: Apress.
50. Scheyder E., 2015. “Reality Hits North Dakota’s Pricey Apartment Market; Rents Drop.” Reuters, March 2. http://www.reuters.com/article/2015/03/02/us-north-dakota-apartments-idUSKBN0LY0TR20150302.
51. Schwartz A., 2010. Housing Policy in the United States, 2nd ed. New York: Routledge.
52. Schweitzer L., 2014. “Planning and Social Media: A Case Study of Public Transit and Stigma on Twitter.” Journal of the American Planning Association 80 (3): 218–38. doi:10.1080/01944363.2014.980439.
53. Scrapy Community. 2015. “Scrapy Documentation.” Scrapy: A Fast and Powerful Scraping and Web Crawling Framework. http://scrapy.org/doc/.
54. Seamans R., Zhu F., 2014. “Responses to Entry in Multi-Sided Markets: The Impact of Craigslist on Local Newspapers.” Management Science 60 (2): 476–93. doi:10.1287/mnsc.2013.1785.
55. Spielman S., Folch D., Nagle N., 2014. “Patterns and Causes of Uncertainty in the American Community Survey.” Applied Geography 46: 147–57. doi:10.1016/j.apgeog.2013.11.002.
56. Splichal C., 2015. “Recent Development: Craigslist and the CFAA: The Untold Story.” Florida Law Review 67 (5): 1845–60.
57. Sugrue T., 2005. The Origins of the Urban Crisis: Race and Inequality in Postwar Detroit. Princeton, NJ: Princeton University Press.
58. Townsend A., 2013. Smart Cities: Big Data, Civic Hackers, and the Quest for a New Utopia. New York: Norton.
59. USHUD (US Department of Housing and Urban Development). 2007. Fair Market Rents for the Section 8 Housing Assistance Payments Program. Washington, DC: Office of Policy Development & Research.
60. USHUD (US Department of Housing and Urban Development). 2014. Rental Burdens: Rethinking Affordability Measures. PD&R Edge, September.
61. USHUD (US Department of Housing and Urban Development). 2015. 50th Percentile Rent Estimates. HUD User Data Sets. http://www.huduser.gov/portal/datasets/50per.html.
62. Watts D., 2007. “A Twenty-First Century Science.” Nature 445 (7127): 489–89.
63. Wegmann J., Chapple K., 2012. Understanding the Market for Secondary Units in the East Bay. WP-2012-03. Berkeley, CA: Institute of Urban and Regional Development.
64. Wolfe N., 2015. “Using the Computer Fraud and Abuse Act to Secure Public Data Exclusivity.” Northwestern Journal of Technology and Intellectual Property 13 (3): 301–16.
65. Zandbergen P., 2008. “A Comparison of Address Point, Parcel and Street Geocoding Techniques.” Computers, Environment and Urban Systems 32 (3): 214–32. doi:10.1016/j.compenvurbsys.2007.11.006.