A primer on theory-driven web scraping: Automatic extraction of big data from the internet for use in psychological research

Psychological Methods

Volumn 21, Issue 4, 2016, Pages 475-492

  Landers, Richard N ^a   Brusso, Robert C ^a,b   Cavanaugh, Katelyn J ^a   Collmus, Andrew B ^a  

^a OLD DOMINION UNIVERSITY   (United States)

^b ICF INTERNATIONAL   (United States)

Author keywords

Big data; Data source theory; Python; Tutorial; Web scraping

Indexed keywords

EXTRACTION; HUMAN; HUMAN EXPERIMENT; INTERNET; PSYCHOLOGIST; PSYCINFO; SCIENTIST; THEORETICAL MODEL; COMPUTER INTERFACE; DATABASE MANAGEMENT SYSTEM; INFORMATION RETRIEVAL; PSYCHOLOGY; RESEARCH;

DATABASE MANAGEMENT SYSTEMS; HUMANS; INFORMATION STORAGE AND RETRIEVAL; INTERNET; PSYCHOLOGY; RESEARCH; USER-COMPUTER INTERFACE;

EID: 84969260511     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/met0000081     Document Type: Article

References (79)

1. American Psychological Association (APA). (2010). Ethical principles of psychologists and code of conduct. Washington, DC: American Psychological Association.
2. Antoniou, G., & Van Harmelen, F. (2008). A semantic web primer (2nd ed.). Cambridge, MA: MIT press.
3. Baker, R. S. J. D., & Yacef, K. (2009). The state of educational data mining in 2009: A review and future visions. Journal of Eductional Data Mining, 1, 3-16.
4. Band, J., & Gerafi, J. (2013). The fair use/fair dealing handbook. Retrieved from http://www.policybandwidth.com/publications
5. Benchell, N. A. (2002). The Digital Millennium Copyright Act: A review of the law and the court's interpretation. Journal of Computer & Information Law, 21, 1-18.
6. Bilton, N. (2014, April 11). Appeals court overturns conviction of AT&T hacker known as "Weev". The New York Times. Retrieved from bits.blogs.nytimes.com/2014/04/11/court-overturns-conviction-of-att-ipad-hacker
7. Binning, J. F., & Barrett, G. V. (1989). Validity of personnel decisions: A conceptual analysis of the inferential and evidential bases. Journal of Applied Psychology, 74, 478-494. http://dx.doi.org/10.1037/0021-9010.74.3.478
8. Boyd, D., & Crawford, K. (2012). Critical questions for big data: Provocations for a cultural, technological, and scholarly phenomenon. Information Communication and Society, 15, 662-679. http://dx.doi.org/10.1080/1369118X.2012.678878
9. Cafarella, M. J., Halevy, A., & Madhavan, J. (2011). Structured data on the web. Communications of the ACM, 54, 72-79. http://dx.doi.org/10.1145/1897816.1897839
10. Chellapilla, K., Larson, K., Simard, P. Y., & Czerwinski, M. (2005). Computers beat humans at single character recognition in reading based Human Interaction Proofs (HIPs). Paper presented at the CEAS, Stanford University, CA.
11. Cleveland, W. S. (2001). Data science: An action plan for expanding the technical areas of the field of statistics. International Statistical Review, 69, 21-26.
12. Cohen, J. (1994). The Earth is round (p <.05). American Psychologist, 49, 997-1003. http://dx.doi.org/10.1037/0003-066X.49.12.997
13. Conway, D., & White, J. M. (2012). Machine learning for hackers. Sebastopol, CA: O'Reilly Media.
14. Cooley, R., Mobasher, B., & Srivastava, J. (1997). Web mining: Information and pattern discovery on the World Wide Web. In Proceedings of the Ninth IEEE International Conference on Tools With Artificial Intelligence, 558-567.
15. Copes, H., & Miller, J. M. (2015). The Routeledge handbook of qualitative criminology. New York, NY: Routeledge.
16. Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281-302. http://dx.doi.org/10.1037/h0040957
17. Culpepper, S. A., & Aguinis, H. (2011). R is for revolution: A cutting-edge, free, open source statistical package. Organizational Research Methods, 14, 735-740.
18. Data Science Association (DSA). (2015). Data science code of profes-sional conduct [PDF document]. Retrieved from http://www.datascienceassn.org/code-of-conduct.html
19. De Corte, W., Lievens, F., & Sackett, P. R. (2007). Combining predictors to achieve optimal trade-offs between selection quality and adverse impact. Journal of Applied Psychology, 92, 1380-1393.
20. Doherty, M. (2013). Legal risk in computer security research in Canada. Unpublished manuscript, Department of Computer Science, Dalhousie University, Nova Scotia, Canada.
21. EBay v. Bidder's Edge, 100 F. Supp. 2d 1058 (N. D. Cal. 2000).
22. Evans, M., Donelle, L., & Hume-Loveland, L. (2012). Social support and online postpartum depression discussion groups: A content analysis. Patient Education and Counseling, 87, 405-410. http://dx.doi.org/10.1016/j.pec.2011.09.011
23. Field, M. A. (1986). Sources of law: The scope of federal common law. Harvard Law Review, 99, 881. http://dx.doi.org/10.2307/1341238
24. Hanson, W. E., Creswell, J. W., Clark, V. L. P., Petska, K. S., & Creswell, J. D. (2005). Mixed methods research designs in counseling psychology. Journal of Counseling Psychology, 52, 224-235.
25. Harrington, P. (2012). Machine learning in action. Shelter Island, N. Y.: Manning Publications Co.
26. Healthboards.com. (n.d.). Depression message board & forum discussions-HealthBoards. Retrieved from http://www.healthboards.com/boards/depression/
27. Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33, 61-83. http://dx.doi.org/10.1017/S0140525X0999152X
28. Herman, M., Rivera, S., Mills, S., Sullivan, J., Guerra, P., Cosmas, A., Kim, M. (2013). The field guide to data science. Retrieved from http://www.boozallen.com/insights/2013/11/data-science-field-guide
29. Jarret, H. M., & Bailie, M. W. (2011). Prosecuting computer crimes OLE Litigation Series. Washington, DC: Office of Legal Education.
30. Jensen, S. (2014). Abusing the Computer Fraud and Abuse Act: Why broad interpretations of the CFAA fail. Hamline Law Review, 36, 81-138.
31. Joyce, C., Leaffer, M., Jaszi, P., & Ochoa, T. (1998). Copyright law (6th ed.). Newark, NJ: LexisNexis.
32. Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196-217. http://dx.doi.org/10.1207/s15327957pspr0203_4
33. Kessler, R. C., McGonagle, K. A., Swartz, M., Blazer, D. G., & Nelson, C. B. (1993). Sex and depression in the National Comorbidity Survey I: Lifetime prevalence, chronicity and recurrence. Journal ofAffective Disorders, 29, 85-96.
34. Kirk, R. E. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746-759. http://dx.doi.org/10.1177/0013164496056005002
35. Kramer, A. D. I., Guillory, J. E., & Hancock, J. T. (2014). Experimental evidence of massive-scale emotional contagion through social networks. Proceedings ofthe National Academies ofScience ofthe United States of America, 111. 8788-8790. http://dx.doi.org/10.1073/pnas.1320040111
36. Kreibich, C., & Crowcroft, J. (2004). Honeycomb: Creating intrusion detection signatures using honeypots. Computer Communication Review, 34, 51-56. http://dx.doi.org/10.1145/972374.972384
37. Krueger, J. (2001). Null hypothesis significance testing. On the survival of a flawed method. American Psychologist, 56, 16-26. http://dx.doi.org/10.1037/0003-066X.56.1.16
38. Landers, R. N., & Behrend, T. S. (2015). An inconvenient truth: Arbitrary distinctions between organizations, Mechanical Turk, and other convenience samples. Industrial and Organizational Psychology: Perspectives on Science and Practice, 8, 142-164. http://dx.doi.org/10.1017/iop.2015.13
39. Landers, R. N., Fink, A., & Collmus, A. B. (in press). Using big data to enhance staffing: Vast untapped resources or tempting honeypot? In J. L. Farr & N. T. Tippins (Eds.), Handbook ofemployee selection (2nd ed.). New York, NY: Routledge.
40. Larson, W. (2008, August). An introduction to compassionate screen scraping. Retrieved from http://lethain.com/
41. Lazer, D., Kennedy, R., King, G., & Vespignani, A. (2014). Big data. The parable of Google Flu: Traps in big data analysis. Science, 343, 1203-1205. http://dx.doi.org/10.1126/science.1248506
42. Lengua, L. J., & Stormshak, E. A. (2000). Gender, gender roles, and personality: Gender differences in the prediction of coping and psychological symptoms. Sex Roles, 43, 787-820. http://dx.doi.org/10.1023/A:1011096604861
43. Lewin, K. (1935). A dynamic theory of personality. New York, NY: McGraw-Hill.
44. Liu, H. (2008). Social network profiles as taste performances. Journal of Computer-Mediated Communication, 13, 252-275. http://dx.doi.org/10.1111/j.1083-6101.2007.00395.x
45. Loukides, M. (2010). What is data science? The future belongs to the companies and people that turn data into products. O'Reilly Radar Report. Retrieved from http://www.verigazeteciligi.com/wp-content/uploads/2014/12/What_is_Data_Science.pdf
46. Madison, M. J. (2004). A pattern-oriented approach to fair use. William and Mary Law Review, 45, 1525-1690.
47. Marres, N. (2015). Why map issues? On controversy analysis as a digital method. Science, Technology & Human Values, 40, 655-686. http://dx.doi.org/10.1177/0162243915574602
48. Marres, N., & Weltevrede, E. (2013). Scraping the social? Issues in live social research. Journal ofCultural Economics, 6, 313-335. http://dx.doi.org/10.1080/17530350.2013.772070
49. Maslow, A. H. (1966). The psychology ofscience. New York, NY: Harper & Row.
50. Mathison, S. (1988). Why triangulate? Educational Researcher, 17, 13-17. http://dx.doi.org/10.3102/0013189X017002013
51. Moorthy, J., Lahiri, R., Biswas, N., Sanyal, D., Ranjan, J., Nanath, K., & Ghosh, P. (2015). Big data: Prospects and challenges. The Journal for Decision Makers, 40, 74-96.
52. Nimmer, D. (2000). A riff on fair use in the Digital Millennium Copyright Act. University of Pennsylvania Law Review, 148, 673-742. http://dx.doi.org/10.2307/3312825
53. Nolen-Hoeksema, S. (2001). Gender differences in depression. Current Directions in Psychological Science, 10, 173-176. http://dx.doi.org/10.1111/1467-8721.00142
54. O'Brien, D., & Torres, A. M. (2012). Social networking and online privacy: Facebook users' perceptions. Irish Journal of Management, 31, 63-97.
55. O'Neil, C., & Schutt, R. (2013). Doing data science: Straight talk from the frontline. Beijing, China: O'Reilly Media.
56. O'Reilly, S. (2007). Nominative fair use and Internet aggregators: Copyright and trademark challenges posed by bots, web crawlers and screen-scraping technologies. Loyola Consumer Law Review, 19, 273-288.
57. Piccinelli, M., & Wilkinson, G. (2000). Gender differences in depression: Critical review. The British Journal ofPsychiatry, 177, 486-492. http://dx.doi.org/10.1192/bjp.177.6.486
58. Piko, B. (2001). Gender differences and similarities in adolescents' ways of coping. The Psychological Record, 51, 223-235.
59. Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal ofApplied Psychology, 88, 879-903. http://dx.doi.org/10.1037/0021-9010.88.5.879
60. Ponzetto, G. A. M., & Fernandez, P. A. (2008). Case law versus statute law: An evolutionary comparison. The Journal ofLegal Studies, 37, 379-430. http://dx.doi.org/10.1086/533421
61. Poole, D. (1989). Explanation and prediction: An architecture for default and abductive reasoning. Computational Intelligence, 5, 97-110. http://dx.doi.org/10.1111/j.1467-8640.1989.tb00319.x
62. Quilter, L. (2002). The continuing expansion of cyberspace trespass to chattels. Berkeley Technology Law Journal, 17, 421-443.
63. R. Core Team. (2015). R: A language and environment for statistical computing [Computer software]. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org/
64. Reid, J. (2011). "We don't Twitter, we Facebook": An alternative pedagogical space that enables critical practices in relation to writing. English Teaching, 10, 58-80.
65. Savage, M., & Burrows, R. (2009). Some further reflections on the coming crisis of empirical sociology. Sociology, 43, 762-772. http://dx.doi.org/10.1177/0038038509105420
66. Schmidt, F. L., & Hunter, J. E. (2001). Meta-analysis. In N. Anderson, D. S. Ones, H. K. Sinangil, & C. Viswesvaran (Eds.), Handbook of industrial, work & organizational psychology: Vol. 1: Personnel psychology (pp. 51-70). Thousand Oaks, CA: Sage.
67. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
68. Shaw, L. H., & Gant, L. M. (2002). In defense of the Internet: The relationship between Internet communication and depression, loneliness, self-esteem, and perceived social support. Cyberpsychology & Behavior, 5, 157-171. http://dx.doi.org/10.1089/109493102753770552
69. Spitzner, L. (2002). Honeypots: Tracking hackers (Vol. 1). Boston, MA: Addison Wesley Reading.
70. Thelwall, M., & Vaughan, L. (2004). A fair history of the Web? Examining country balance in the Internet Archive. Library & Information Science Research, 26, 162-176. http://dx.doi.org/10.1016/j.lisr.2003.12.009
71. The World Bank. (2014). Population, female (% oftotal). Retrieved from http://data.worldbank.org/indicator/SP.POP.TOTL.FE.ZS
72. Ticketmaster Corp., et al. v. Tickets.com, Inc., CV 99-7654 HLH (BQRx) (2000 U.S. Dist.).
73. Von Ahn, L., Blum, M., Hopper, N. J., & Langford, J. (2003). CAPTCHA: Using hard AI problems for security. EUROCRYPT, 2003, 297-311.
74. Wallace, P. (2001). The psychology ofthe Internet. Cambridge, United Kingdom: Cambridge University Press.
75. Wentworth, P., Elkner, J., Downey, A. B., & Meyers, C. (2012). How to think like a computer scientist: Learning with Python 3. The Open Book Project: http://openbookproject.net/thinkcs/python/english3e/
76. World Wide Web Consortium (W3C). (2014, October 28). HTML5. Retrieved from http://www.w3.org/TR/html/
77. Young, S., Dutta, D., & Dommety, G. (2009). Extrapolating psychological insights from Facebook profiles: A study of religion and relationship status. Cyberpsychology & Behavior, 12, 347-350. http://dx.doi.org/10.1089/cpb.2008.0165
78. Youyou, W., Kosinski, M., & Stillwell, D. (2015). Computer-based personality judgments are more accurate than those made by humans. Proceedings ofthe National Academy ofSciences ofthe United States of America, 112. 1036-1040. http://dx.doi.org/10.1073/pnas.1418680112
79. Zetter, K. (2013, March 18). AT&T hacker "Weev" sentenced to 3.5 years in prison. Wired. Retrieved from http://www.wired.com/2013/03/att-hacker-gets-3-years/