Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning

Perspectives on Psychological Science

Volumn 12, Issue 6, 2017, Pages 1100-1122

  Yarkoni, Tal ^a   Westfall, Jacob ^a  

^a UNIVERSITY OF TEXAS AT AUSTIN   (United States)

Author keywords

explanation; machine learning; prediction

Indexed keywords

BEHAVIOR; GOLD STANDARD; HUMAN; MACHINE LEARNING; PREDICTION; RANDOMIZED CONTROLLED TRIAL; PROCEDURES; PSYCHOLOGY;

HUMANS; MACHINE LEARNING; PSYCHOLOGY;

EID: 85030714859     PISSN: 17456916     EISSN: 17456924     Source Type: Journal    
DOI: 10.1177/1745691617693393     Document Type: Article

References (110)

1. Apté C., Weiss S., (1997). Data mining with decision trees and decision rules. Future Generation Computer Systems, 13, 197–210. doi:10.1016/S0167-739X(97)00021-6
2. Baayen R. H., Milin P., Đurdˉević D. F., Hendrix P., Marelli M., (2011). An amorphous model for morphological processing in visual comprehension based on naive discriminative learning. Psychological Review, 118, 438–481. doi:10.1037/a0023851
3. Back M. D., Stopfer J. M., Vazire S., Gaddis S., Schmukle S. C., Egloff B., Gosling S. D., (2010). Facebook profiles reflect actual personality, not self-idealization. Psychological Science, 21, 372–374. doi:10.1177/0956797609360756
4. Bakker M., van Dijk A., Wicherts J. M., (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543–554. doi:10.1177/1745691612459060
5. Balota D. A., Yap M. J., Hutchison K. A., Cortese M. J., Kessler B., Loftis B... Treiman R., (2007). The English Lexicon Project. Behavior Research Methods, 39, 445–459. doi:10.3758/BF03193014
6. Bentley R. A., O’Brien M. J., Brock W. A., (2014). Mapping collective behavior in the big-data era. Behavioral & Brain Sciences, 37, 63–76. doi:10.1017/S0140525X13000289
7. Biswal B. B., Mennes M., Zuo X.- N., Gohel S., Kelly C., Smith S. M... Milham M. P., (2010). Toward discovery science of human brain function. Proceedings of the National Academy of Sciences, USA, 107, 4734–4739. doi:10.1073/pnas.0911855107
8. Bond R. M., Fariss C. J., Jones J. J., Kramer A. D. I., Marlow C., Settle J. E., Fowler J. H., (2012). A 61-million-person experiment in social influence and political mobilization. Nature, 489, 295–298. doi:10.1038/nature11421
9. Breiman L., (2001a). Random forests. Machine Learning, 45, 5–32. doi:10.1023/A:1010933404324
10. Breiman L., (2001b). Statistical modeling: The two cultures. Statistical Science, 16, 199–215.
11. Brewer J. B., Zhao Z., Desmond J. E., Glover G. H., Gabrieli J. D. E., (1998). Making memories: Brain activity that predicts how well visual experience will be remembered. Science, 281, 1185–1187. doi:10.1126/science.281.5380.1185
12. Browne M. W., (2000). Cross-validation methods. Journal of Mathematical Psychology, 44, 108–132. doi:10.1006/jmps.1999.1279
13. Browne M. W., MacCallum R. C., Kim C.- T., Andersen B. L., Glaser R., (2002). When fit indices and residuals are incompatible. Psychological Methods, 7, 403–421. doi:10.1037/1082-989X.7.4.403
14. Brysbaert M., New B., (2009). Moving beyond Kucˇera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English. Behavior Research Methods, 41, 977–990. doi:10.3758/BRM.41.4.977
15. Bunea F., She Y., Ombao H., Gongvatana A., Devlin K., Cohen R., (2011). Penalized least squares regression methods and applications to neuroimaging. NeuroImage, 55, 1519–1527. doi:10.1016/j.neuroimage.2010.12.028
16. Burnham K. P., Anderson D. R., (2004). Multimodel inference understanding AIC and BIC in model selection. Sociological Methods & Research, 33, 261–304. doi:10.1177/0049124104268644
17. Cawley G. C., Talbot N. L. C., (2010). On over-fitting in model selection and subsequent selection bias in performance evaluation. Journal of Machine Learning Research, 11, 2079–2107.
18. Chambers C., Munafo M., (2013, June 5). Trust in science would be improved by study pre-registration. The Guardian. Retrieved from https://www.theguardian.com/science/blog/2013/jun/05/trust-in-science-study-pre-registration
19. Cohen J., (1962). The statistical power of abnormal-social psychological research: A review. The Journal of Abnormal and Social Psychology, 65, 145–153. doi:10.1037/h0045186
20. Cohen J., (1992). A power primer. Psychological Bulletin, 112, 155–159. doi:10.1037/0033-2909.112.1.155
21. Cohen J., Cohen P., West S. G., Aiken L. S., (2013). Applied multiple regression/correlation analysis for the behavioral sciences. Abingdon, United Kingdom: Routledge.
22. Davis-Stober C. P., Dana J., (2013). Comparing the accuracy of experimental estimates to guessing: A new perspective on replication and the “crisis of confidence” in psychology. Behavior Research Methods, 46, 1–14. doi:10.3758/s13428-013-0342-1
23. Dawes R. M., (1979). The robust beauty of improper linear models in decision making. American Psychologist, 34, 571–582. doi:10.1037/0003-066X.34.7.571
24. Deng J., Dong W., Socher R., Li L.- J., Li K., Fei-Fei L., (2009). ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, 2009 (CVPR 2009) (pp. 248–255). New York, NY: Institute of Electrical and Electronics Engineers. doi:10.1109/CVPR.2009.5206848
25. Domingos P., (2012). A few useful things to know about machine learning. Communication of the ACM, 55(10), 78–87. doi:10.1145/2347736.2347755
26. Donoho D., (2015, September). 50 years of data science. Paper presented at the Tukey Centennial workshop, Princeton, NJ.
27. Du S., Tao Y., Martinez A. M., (2014). Compound facial expressions of emotion. Proceedings of the National Academy of Sciences, USA, 111, E1454–E1462. doi:10.1073/pnas.1322355111
28. Dwan K., Altman D. G., Arnaiz J. A., Bloom J., Chan A.- W., Cronin E... Williamson P. R., (2008). Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE, 3(8), e3081. doi:10.1371/journal.pone.0003081
29. Ebersole C. R., Atherton O. E., Belanger A. L., Skulborstad H. M., Adams R. B., Allen J... Nosek B. A., (2015). Many labs 3: Evaluating participant pool quality across the academic semester via replication. Retrieved from osf.io/ct89g
30. Ebstein R. P., Novick O., Umansky R., Priel B., Osher Y., Blaine D... Belmaker R. H., (1996). Dopamine D4 receptor (D4DR) exon III polymorphism associated with the human personality trait of novelty seeking. Nature Genetics, 12, 78–80. doi:10.1038/ng0196-78
31. Ekman P., (1992). An argument for basic emotions. Cognition and Emotion, 6, 169–200. doi:10.1080/02699939208411068
32. Fast L. A., Funder D. C., (2008). Personality as manifest in word use: Correlations with self-report, acquaintance report, and behavior. Journal of Personality and Social Psychology, 94, 334–346. doi:10.1037/0022-3514.94.2.334
33. Ferguson C. J., Heene M., (2012). A vast graveyard of undead theories publication bias and psychological science’s aversion to the null. Perspectives on Psychological Science, 7, 555–561. doi:10.1177/1745691612459059
34. Friedman J. H., (2001). Greedy function approximation: A gradient boosting machine. The Annals of Statistics, 29, 1189–1232.
35. Gelman A., Hill J., (2006). Data analysis using regression and multilevel/hierarchical models (1st ed.). Cambridge, UK: Cambridge University Press.
36. Gelman A., Loken E., (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time. Retrieved from http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf
37. Gelman A., Weakliem D., (2009). Of beauty, sex and power: Too little attention has been paid to the statistical challenges in estimating small effects. American Scientist, 97, 310–316.
38. Gosling S. D., Augustine A. A., Vazire S., Holtzman N., Gaddis S., (2011). Manifestations of personality in online social networks: Self-reported Facebook-related behaviors and observable profile information. Cyberpsychology, Behavior, and Social Networking, 14, 483–488. doi:10.1089/cyber.2010.0087
39. Greenwald A. G., (2012). There is nothing so theoretical as a good method. Perspectives on Psychological Science, 7, 99–108. doi:10.1177/1745691611434210
40. Hagerty M. R., Srinivasan V., (1991). Comparing the predictive powers of alternative multiple regression models. Psychometrika, 56, 77–85. doi:10.1007/BF02294587
41. Hastie T., Tibshirani R., Friedman J., (2009). The elements of statistical learning. New York, NY: Springer. Retrieved from http://link.springer.com/10.1007/978-0-387-84858-7
42. Holbert R. L., Stephenson M. T., (2002). Structural equation modeling in the communication sciences, 1995–2000. Human Communication Research, 28, 531–551. doi:10.1111/j.1468-2958.2002.tb00822.x
43. Horn J. D. V., Toga A. W., (2013). Human neuroimaging as a “Big Data” science. Brain Imaging and Behavior, 8, 323–331. doi:10.1007/s11682-013-9255-y
44. Ioannidis J. P. A., (2008). Why most discovered true associations are inflated. Epidemiology, 19, 640–648. doi:10.1097/EDE.0b013e31818131e7
45. Ioannidis J. P. A., (2012). Why science is not necessarily self-correcting. Perspectives on Psychological Science, 7, 645–654. doi:10.1177/1745691612464056
46. John L. K., Loewenstein G., Prelec D., (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524–532. doi:10.1177/0956797611430953
47. Jonas K. J., Cesario J., (2015). How can preregistration contribute to research in our field? Comprehensive Results in Social Psychology, 1, 1–7. doi:10.1080/23743603.2015.1070611
48. Judd C. M., Westfall J., Kenny D. A., (2016). Experiments with more than one random factor: Designs, analytic models, and statistical power. Annual Review of Psychology, 68, 601–625.
49. Klein R. A., Ratliff K. A., Vianello M., Adams R. B., Jr. Bahník Š., Bernstein M. J... Nosek B. A., (2014). Investigating variation in replicability: A “many labs” replication project. Social Psychology, 45, 142–152. doi:10.1027/1864-9335/a000178
50. Kosinski M., Stillwell D., Graepel T., (2013). Private traits and attributes are predictable from digital records of human behavior. Proceedings of the National Academy of Sciences, USA, 110, 5802–5805. doi:10.1073/pnas.1218772110
51. Krstajic D., Buturovic L. J., Leahy D. E., Thomas S., (2014). Cross-validation pitfalls when selecting and assessing regression and classification models. Journal of Cheminformatics, 6, 1–15. doi:10.1186/1758-2946-6-10
52. Kurtz A. K., (1948). A research test of the Rorschach Test. Personnel Psychology, 1, 41–51. doi:10.1111/j.1744-6570.1948.tb01292.x
53. LeCun Y., Bengio Y., Hinton G., (2015). Deep learning. Nature, 521, 436–444. doi:10.1038/nature14539
54. Lehrer J., (2010, December 13). The truth wears off. The New Yorker. Retrieved from http://www.newyorker.com/magazine/2010/12/13/the-truth-wears-off
55. Lesch K.- P., Bengel D., Heils A., Sabol S. Z., Greenberg B. D., Petri S... Murphy D. L., (1996). Association of anxiety-related traits with a polymorphism in the serotonin transporter gene regulatory region. Science, 274, 1527–1531. doi:10.1126/science.274.5292.1527
56. Loftus E. F., Palmer J. C., (1996). Eyewitness testimony. In Introducing psychological research (pp. 305–309). London, UK: Macmillan Education. Retrieved from http://link.springer.com/chapter/10.1007/978-1-349-24483-6_46
57. McNeish D. M., (2015). Using lasso for predictor selection and to assuage overfitting: A method long overlooked in behavioral sciences. Multivariate Behavioral Research, 50, 471–484. doi:10.1080/00273171.2015.1036965
58. Meehl P. E., (1990). Why summaries of research on psychological theories are often uninterpretable. Psychological Reports, 66, 195–244. doi:10.2466/pr0.1990.66.1.195
59. Miller G., (2012). The smartphone psychology manifesto. Perspectives on Psychological Science, 7, 221–237. doi:10.1177/1745691612441215
60. Mosier C. I., (1951). The need and means of cross validation. I. Problems and designs of cross-validation. Educational and Psychological Measurement, 11, 5–11. doi:10.1177/001316445101100101
61. Munafò M. R., Flint J., (2011). Dissecting the genetic architecture of human personality. Trends in Cognitive Sciences, 15, 395–400. doi:10.1016/j.tics.2011.07.007
62. New B., Pallier C., Brysbaert M., Ferrand L., (2004). Lexique 2: A new French lexical database. Behavior Research Methods, Instruments, & Computers, 36, 516–524. doi:10.3758/BF03195598
63. Nosek B. A., Lakens D., (2014). Registered reports: A method to increase the credibility of published results. Social Psychology, 45, 137–141. doi:10.1027/1864-9335/a000192
64. Nosek B. A., Spies J. R., Motyl M., (2012). Scientific Utopia II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615–631. doi:10.1177/1745691612459058
65. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. doi:10.1126/science.aac4716
66. Pennebaker J. W., King L. A., (1999). Linguistic styles: Language use as an individual difference. Journal of Personality and Social Psychology, 77, 1296–1312. doi:10.1037/0022-3514.77.6.1296
67. Perry C., Ziegler J. C., Zorzi M., (2010). Beyond single syllables: Large-scale modeling of reading aloud with the Connectionist Dual Process (CDP++) model. Cognitive Psychology, 61, 106–151. doi:10.1016/j.cogpsych.2010.04.001
68. Pinheiro J., Bates D., (2000). Mixed-effects models in S and S-PLUS. New York, NY: Springer.
69. Poropat A. E., (2009). A meta-analysis of the five-factor model of personality and academic performance. Psychological Bulletin, 135, 322–338. doi:10.1037/a0014996
70. Rawlings D., Ciancarelli V., (1997). Music preference and the Five-Factor Model of the NEO Personality Inventory. Psychology of Music, 25, 120–132. doi:10.1177/0305735697252003
71. Rentfrow P. J., Gosling S. D., (2003). The do re mi’s of everyday life: The structure and personality correlates of music preferences. Journal of Personality and Social Psychology, 84, 1236–1256. doi:10.1037/0022-3514.84.6.1236
72. Ripke S., Wray N. R., Lewis C. M., Hamilton S. P., Weissman M. M., Breen G... Sullivan P. F., (2013). A mega-analysis of genome-wide association studies for major depressive disorder. Molecular Psychiatry, 18, 497–511. doi:10.1038/mp.2012.21
73. Rissman J., Greely H. T., Wagner A. D., (2010). Detecting individual memories through the neural decoding of memory states and past experience. Proceedings of the National Academy of Sciences, USA, 107, 9849–9854. doi:10.1073/pnas.1001028107
74. Rowe M. L., Goldin-Meadow S., (2009). Early gesture selectively predicts later language learning. Developmental Science, 12, 182–187. doi:10.1111/j.1467-7687.2008.00764.x
75. Schmidhuber J., (2015). Deep learning in neural networks: An overview. Neural Networks, 61, 85–117. doi:10.1016/j.neunet.2014.09.003
76. Schmitt N., Coyle B. W., Rauschenberger J., (1977). A Monte Carlo evaluation of three formula estimates of cross-validated multiple correlation. Psychological Bulletin, 84, 751–758. doi:10.1037/0033-2909.84.4.751
77. Sedlmeier P., Gigerenzer G., (1989). Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 105, 309–316. doi:10.1037/0033-2909.105.2.309
78. Shao J., (1993). Linear model selection by cross-validation. Journal of the American Statistical Association, 88, 486–494. doi:10.1080/01621459.1993.10476299
79. Shear B. R., Zumbo B. D., (2013). False positives in multiple regression unanticipated consequences of measurement error in the predictor variables. Educational and Psychological Measurement, 73, 733–756. doi:10.1177/0013164413487738
80. Shmueli G., (2010). To explain or to predict? Statistical Science, 25, 289–310.
81. Simmons J. P., Nelson L. D., Simonsohn U., (2011). False-positive psychology undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. doi:10.1177/0956797611417632
82. Smith D. J., Escott-Price V., Davies G., Bailey M. E. S., Colodro-Conde L., Ward J... O’Donovan M. C., (2016). Genome-wide analysis of over 106,000 individuals identifies 9 neuroticism-associated loci. Molecular Psychiatry, 21, 749–757.
83. Strube M. J., (2006). SNOOP: A program for demonstrating the consequences of premature and repeated null hypothesis testing. Behavior Research Methods, 38, 24–27. doi:10.3758/BF03192746
84. Tibshirani R., (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58, 267–288.
85. Tibshirani R., (2011). Regression shrinkage and selection via the lasso: A retrospective. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 73, 273–282. doi:10.1111/j.1467-9868.2011.00771.x
86. Van Essen D. C., Smith S. M., Barch D. M., Behrens T. E. J., Yacoub E., Ugurbil K. (2013). The WU-Minn Human Connectome Project: An overview. NeuroImage, 80, 62–79. doi:10.1016/j.neuroimage.2013.05.041
87. Varma S., Simon R., (2006). Bias in error estimation when using cross-validation for model selection. BMC Bioinformatics, 7(1), 91. doi:10.1186/1471-2105-7-91
88. Vazire S., Gosling S. D., (2004). e-Perceptions: Personality impressions based on personal websites. Journal of Personality and Social Psychology, 87, 123–132. doi:10.1037/0022-3514.87.1.123
89. Vitaro F., Arseneault L., Tremblay R. E., (1999). Impulsivity predicts problem gambling in low SES adolescent males. Addiction, 94, 565–575. doi:10.1046/j.1360-0443.1999.94456511.x
90. Vrieze S. I., (2012). Model selection and psychological theory: A discussion of the differences between the Akaike information criterion (AIC) and the Bayesian information criterion (BIC). Psychological Methods, 17, 228–243. doi:10.1037/a0027127
91. Wagenmakers E.- J., Wetzels R., Borsboom D., van der Maas H. L. J., Kievit R. A., (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638. doi:10.1177/1745691612463078
92. Wainer H., (1976). Estimating coefficients in linear models: It don’t make no nevermind. Psychological Bulletin, 83, 213–217. doi:10.1037/0033-2909.83.2.213
93. Westfall J., Kenny D. A., Judd C. M., (2014). Statistical power and optimal design in experiments in which samples of participants respond to samples of stimuli. Journal of Experimental Psychology: General, 143, 2020–2045. doi:10.1037/xge0000014
94. Westfall J., Yarkoni T., (2016). Statistically controlling for confounding constructs is harder than you think. Manuscript submitted for publication.
95. Whelan R., Watts R., Orr C. A., Althoff R. R., Artiges E., Banaschewski T.,.. the IMAGEN Consortium. (2014). Neuropsychosocial profiles of current and future adolescent alcohol misusers. Nature, 512, 185–189. doi:10.1038/nature13402
96. Wherry R. J., (1951). IV. Comparison of cross-validation with statistical inference of betas and multiple R from a single sample. Educational and Psychological Measurement, 11, 23–28. doi:10.1177/001316445101100104
97. Wherry R. J., (1975). Underprediction from overfitting: 45 years of shrinkage. Personnel Psychology, 28, 1–18. doi:10.1111/j.1744-6570.1975.tb00387.x
98. Wolpert D. H., Macready W. G., (1997). No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation, 1, 67–82. doi:10.1109/4235.585893
99. Wood A. R., Esko T., Yang J., Vedantam S., Pers T. H., Gustafsson S... Frayling T. M., (2014). Defining the role of common variation in the genomic and biological architecture of adult human height. Nature Genetics, 46, 1173–1186. doi:10.1038/ng.3097
100. Wood S., (2006). Generalized additive models: An introduction with R (1st ed.). Boca Raton, FL: Chapman & Hall/CRC.
101. Wu S., Harris T. J., Mcauley K. B., (2007). The use of simplified or misspecified models: Linear case. The Canadian Journal of Chemical Engineering, 85, 386–398. doi:10.1002/cjce.5450850401
102. Xu K., Nosek B., Greenwald A. G., (2014). Data from the Race Implicit Association Test on the Project Implicit demo website. Journal of Open Psychology Data, 2(1), e3. doi:10.5334/jopd.ac
103. Yap M. J., Balota D. A., Sibley D. E., Ratcliff R., (2012). Individual differences in visual word recognition: Insights from the English Lexicon Project. Journal of Experimental Psychology: Human Perception and Performance, 38, 53–79. doi:10.1037/a0024177
104. Yarkoni T., (2009). Big correlations in little studies: Inflated fMRI correlations reflect low statistical power—Commentary on Vul et al. (2009). Perspectives on Psychological Science, 4, 294–298. doi:10.1111/j.1745-6924.2009.01127.x
105. Yarkoni T., (2010). Personality in 100,000 words: A large-scale analysis of personality and word use among bloggers. Journal of Research in Personality, 44, 363–373. doi:10.1016/j.jrp.2010.04.001
106. Yarkoni T., (2012). Psychoinformatics new horizons at the interface of the psychological and computing sciences. Current Directions in Psychological Science, 21, 391–397. doi:10.1177/0963721412457362
107. Yarkoni T., (2014). Big Data, n. A kind of black magic. Retrieved from http://www.talyarkoni.org/blog/2014/05/19/big-data-n-a-kind-of-black-magic/
108. Yarkoni T., Ashar Y. K., Wager T. D., (2015). Interactions between donor Agreeableness and recipient characteristics in predicting charitable donation and positive social evaluation. PeerJ, 3, e1089. doi:10.7717/peerj.1089
109. Yarkoni T., Balota D., Yap M., (2008). Moving beyond Coltheart’s N: A new measure of orthographic similarity. Psychonomic Bulletin & Review, 15, 971–979. doi:10.3758/PBR.15.5.971
110. Zeiler M. D., Fergus R., (2014). Visualizing and understanding convolutional networks. In Fleet D., Pajdla T., Schiele B., Tuytelaars T., (Eds.), Computer vision—ECCV 2014 (pp. 818–833): Springer International Publishing. Retrieved from http://link.springer.com/chapter/10.1007/978-3-319-10590-1_53