Interpretations and methods: Towards a more effectively self-correcting social psychology

Journal of Experimental Social Psychology

Volumn 66, Issue , 2016, Pages 116-133

  Jussim, Lee ^a   Crawford, Jarret T ^b   Anglin, Stephanie M ^a   Stevens, Sean T ^a   Duarte, Jose L ^c  

^a RUTGERS UNIVERSITY   (United States)

^b The College of New Jersey   (United States)

^c ARIZONA STATE UNIVERSITY   (United States)

Author keywords

Best practices; Interpretation; Research methods; Scientific integrity; Statistics

Indexed keywords

CONFIRMATION BIAS; DRAWING; HUMAN; HUMAN EXPERIMENT; NARRATIVE; PSYCHOLOGIST; SCIENTIST; SOUND; STATISTICS; VALIDITY;

EID: 84977119985     PISSN: 00221031     EISSN: 10960465     Source Type: Journal    
DOI: 10.1016/j.jesp.2015.10.003     Document Type: Article

References (126)

1. Abramowitz, S.I., Gomes, B., Abramowitz, C.V., Publish or perish: Referee bias in manuscript review. Journal of Applied Social Psychology 5 (1975), 187–200.
2. Allport, F.H., Theories of perception and the concept of structure. 1955, Wiley, NY.
3. Allport, G.W., The nature of prejudice. 1954/1979, Addison-Wesley, Reading, MA.
4. American Psychological Association, Stereotype threat widens achievement gap. Retrieved on 8/20/15 from http://www.apa.org/research/action/stereotype.aspx, 2006.
5. Appel, M., Kronberger, N., Stereotypes and the achievement gap: Stereotype threat prior to test taking. Educational Psychology Review 24 (2012), 609–635.
6. Aronson, J., Fried, C.B., Good, C., Reducing the effects of stereotype threat on African American college students by shaping theories of intelligence. Journal of Experimental Social Psychology 38 (2002), 113–125.
7. Aronson, J., Lustina, M.J., Good, C., Keough, K., Steele, C.M., Brown, J., When white men can't do math: Necessary and sufficient factors in stereotype threat. Journal of Experimental Social Psychology 35 (1999), 29–46.
8. Ashton, M.C., Esses, V.M., Stereotype accuracy: Estimating the academic performance of ethnic groups. Personality and Social Psychology Bulletin 25 (1999), 225–236.
9. Bakker, M., van Dijk, A., Wicherts, J.M., The rules of the game called psychological science. Perspectives on Psychological Science 7 (2012), 543–554.
10. Banerjee, P., Chatterjee, P., Sinha, J., Is it light or dark? Recalling moral behavior changes perception of brightness. Psychological Science 23 (2012), 407–409.
11. Bargh, J.A., Social psychological approaches to consciousness. Zelazo, P.D., Moscovitch, M., Thompson, E., (eds.) The Cambridge handbook of consciousness, 2007, Cambridge University Press, New York, NY, 555–569.
12. Bargh, J.A., Chen, M., Burrows, L., Automaticity of social behavior: Direct effects of trait construct and stereotype activation on action. Journal of Personality and Social Psychology 71 (1996), 239–244.
13. Baron, R.M., Albright, L., Malloy, T.E., Effects of behavioral and social class information on social judgment. Personality and Social Psychology Bulletin 21 (1995), 308–315.
14. Barret, L.F., Psychology is not in crisis. New York Times, 2015, Sept 1 (Retrieved on 10/1/15 from: http://www.nytimes.com/2015/09/01/opinion/psychology-is-not-in-crisis.html?_r=0).
15. Begley, C.G., Ellis, L.M., Drug development: Raise standards for preclinical cancer research. Nature 483 (2012), 531–533.
16. Bem, D., Writing the empirical journal article. Darley, J.M., Zanna, M.P., Roediger, H.L. III, (eds.) The complete academic: A career guide, 2002, American Psychological Association, Washington, DC.
17. Bickel, P.J., Hammel, E.A., O'Connell, J.W., Sex bias in graduate admissions: Data from Berkeley. Science 187 (1975), 396–404.
18. Blanton, H., Jaccard, J., Klick, J., Mellers, B., Mitchell, G., Tetlock, P.E., Strong claims and weak evidence: Reassessing the predictive validity of the IAT. Journal of Applied Psychology, 94, 2009, 567.
19. Brandt, M.J., Reyna, C., Chambers, J.R., Crawford, J.T., Wetherell, G., The ideological-conflict hypothesis: Intolerance among both liberals and conservatives. Current Directions in Psychological Science 23 (2014), 27–34.
20. Brown, R., Prejudice: Its social psychology. 2011, John Wiley & Sons.
21. Bruner, J., On perceptual readiness. Psychological Review 64 (1957), 123–152.
22. Caprara, G.V., Alesandri, G., Eisenberg, N., Prosociality: The contribution of traits, values, and self-efficacy beliefs. Journal of Personality and Social Psychology 102 (2012), 1289–1303.
23. Clark, R.D., Hatfield, E., Gender differences in receptivity to sexual offers. Journal of Psychology & Human Sexuality 2 (1989), 39–55.
24. Cohen, J., Statistical power analysis for the behavioral sciences. 1988, Academic Press, New York.
25. Darley, J.M., Gross, P.H., A hypothesis-confirming bias in labeling effects. Journal of Personality and Social Psychology 44 (1983), 20–33.
26. Ditto, P.H., Lopez, D.F., Motivated skepticism: Use of differential decision criteria for preferred and nonpreferred conclusions. Journal of Personality and Social Psychology 63 (1992), 569–584.
27. Doyen, S., Klein, O., Pichon, C., Cleeremans, A., Behavioral priming: It's all in the mind, but whose mind?. PLoS One, 7, 2012, e29081.
28. Duarte, J.L., Crawford, J.T., Stern, C., Haidt, J., Jussim, L., Tetlock, P.E., Political diversity will improve social psychological science. Behavioral and Brain Sciences 38 (2015), 1–54.
29. Eagly, A.H., The science and politics of comparing women and men. American Psychologist 50 (1995), 145–158.
30. Eagly, A.H., Sex differences in social behavior: A social-role interpretation. 2013, Psychology Press.
31. Epstein, W.M., Informational response bias and the quality of the editorial processes among American social work journals. Research on Social Work Practice 14 (2004), 450–458.
32. Firestone, C., Scholl, B.J., “Top-down” effects where none should be found: The El Greco fallacy in perception research. Psychological Science 25 (2014), 38–46.
33. Firestone, C., Scholl, B.J., Enhanced visual awareness for morality and pajamas? Perception vs. memory in top-down effects. Cognition 136 (2015), 409–416.
34. Firestone, C., Scholl, B.J., Cognition does not affect perception: Evaluating the evidence for ‘top-down’ effects. The Behavioral & Brain Sciences, 2015, 1–77, 10.1017/S014052X15000965 (in press).
35. Fleeson, W., Towards a structure- and process-integrated view of personality: Traits as density distributions of states. Journal of Personality and Social Psychology 37 (2001), 1097–1126.
36. Fleeson, W., Moving personality beyond the person–situation debate. Current Directions in Psychological Science 13 (2004), 83–87.
37. Funder, D.C., Persons, situations, and person–situation interactions. John, O.P., Robins, R.W., Pervin, L.A., (eds.) Handbook of personality: Theory and research, 3rd edition, 2008, Guilford Press, New York, 568–580.
38. Funder, D.C., The perilous plight of the (non)-replicator. Retrieved on 9/13/15 from: http://funderstorms.wordpress.com/2012/10/31/the-perilous-plight-of-the-non-replicator/, 2012.
39. Funder, D.C., Levine, J.M., Mackie, D.M., Morf, C.C., Vazire, S., West, S., Improving the dependability of research in personality and social psychology. Personality and Social Psychology Review 18 (2014), 3–12.
40. Gantman, A.P., Van Bavel, J.J., The moral pop-out effect: Enhanced perceptual awareness of morally relevant stimuli. Cognition 132 (2014), 22–29.
41. Gawronski, B., Theory-based bias correction in dispositional inference: The fundamental attribution error is dead, long live the correspondence bias. European Review of Social Psychology 15 (2004), 183–217.
42. Gelman, A., To understand the replication crisis, imagine a world in which everything was published. Retrieved on 9/11/15 from: http://andrewgelman.com/2015/09/02/to-understand-the-replication-crisis-imagine-a-world-in-which-everything-was-published/, 2015.
43. Gildersleeve, K., Haselton, M.G., Fales, M.R., Meta-analyses and p-curves support robust cycle shifts in women's mate preferences: Reply to Wood and Carden (2014) and Harris, Pashler, and Mickes (2014). Psychological Bulletin 140 (2014), 1272–1280, 10.1037/a0037714.
44. Gonzales, P.M., Blanton, H., Williams, K.J., The effect of stereotype threat and double-minority status on the test performance of Latino women. Personality and Social Psychology Bulletin 28 (2002), 659–670.
45. Greenwald, A.G., Pettigrew, T.F., With malice toward none and charity for some: Ingroup favoritism enables discrimination. American Psychologist 69 (2014), 669–684.
46. Greenwald, A.G., Banaji, M.R., Nosek, B.A., Statistically small effects of the Implicit Association Test can have societally large effects. Journal of Personality and Social Psychology 108 (2015), 553–561.
47. Greenwald, A.G., Poehlman, T.A., Uhlmann, E.L., Banaji, M.R., Understanding and using the Implicit Association Test: III. Meta-analysis of predictive validity. Journal of Personality and Social Psychology, 97, 2009, 17.
48. Greenwald, A.G., Pratkanis, A.R., Leippe, M.R., Baumgardner, M.H., Under what conditions does theory obstruct research progress?. Psychological Review 93 (1986), 216–229.
49. Guimond, S., Crisp, R.J., Oliveria, P.D., Kamiejski, R., Kteily, N.S., Kuepper, B. Zick, A., Diversity policy, social dominance, and intergroup relations: Predicting prejudice in changing social and political contexts. Journal of Personality and Social Psychology 104 (2013), 941–958.
50. Haidt, J., The emotional dog and its rational tail: A social intuitionist approach to moral judgment. Psychological Review 108 (2001), 814–834.
51. Harackiewicz, J.M., Canning, E.A., Tibbetts, Y., Giffen, C.J., Blair, S.S., Rouse, D.I., Hyde, J.S., Closing the social class achievement gap for first-generation students in undergraduate biology. Journal of Educational Psychology 106 (2014), 375–389.
52. Hastorf, A.H., Cantril, H., They saw a game: A case study. Journal of Abnormal and Social Psychology 49 (1954), 129–134.
53. Hyde, J.S., The gender similarities hypothesis. American Psychologist 60 (2005), 581–592.
54. Ioannidis, J.P., Why most published research findings are false. PLoS Medicine, 2, 2005, e124.
55. Ioannidis, J.P., Why science is not necessarily self-correcting. Perspectives on Psychological Science 7 (2012), 645–654.
56. Jacks, J.Z., Devine, P.G., Attitude importance, forewarning of message content, and resistance to persuasion. Basic and Applied Social Psychology 22 (2000), 19–29.
57. Jordan, C.H., Zanna, M.P., Not all experiments are created equal: On conducting and reporting persuasive experiments. Sternberg, R.J., Halpern, D., Roediger, H.L. III, (eds.) Critical thinking in psychology, 2007, Cambridge University Press, New York, 160–176.
58. Jost, J.T., Kruglanski, A.W., The estrangement of social constructionism and experimental social psychology: History of the rift and prospects for reconciliation. Personality and Social Psychology Review 6 (2002), 168–187.
59. Jost, J.T., Glaser, J., Kruglanski, A.W., Sulloway, F.J., Political conservatism as motivated social cognition. Psychological Bulletin 129 (2003), 339–375.
60. Jussim, L., Social perception and social reality: A reflection-construction model. Psychological Review 98 (1991), 54–73.
61. Jussim, L., Social perception and social reality: Why accuracy dominates bias and self-fulfilling prophecy. 2012, Oxford University Press, New York.
62. Jussim, L., Unicorns of social psychology. Retrieved on 9/13/15 from: https://www.psychologytoday.com/blog/rabble-rouse/201208/unicorns-social-psychology, 2012.
63. Jussim, L., Maoz, I., Desperately seeking the “wow effect”: Data interpretation and scientific story-telling as issues of research integrity. Presentation at the October 2014 meeting of the Society for Experimental Social Psychology, 2014.
64. Jussim, L., Crawford, J.T., Rubinstein, R.S., Stereotype (in)accuracy in perceptions of groups and individuals. Current Directions in Psychological Science 24 (2015), 490–497.
65. Jussim, L., Crawford, J.T., Stevens, S.T., Anglin, S.M., Duarte, J.L., Can high moral purposes undermine scientific integrity?. To appear in Forgas, J., Jussim, L., van Lange, P., (eds.) Sydney symposium on the social psychology of morality, 2016, Taylor and Francis, New York, NY (in press).
66. Kahan, D.M., Neutral principles, motivated cognition, and some problems for constitutional law. Harvard Law Review, 125, 2011, 1.
67. Kellow, J.T., Jones, B.D., The effects of stereotypes on the achievement gap: Reexamining the academic performance of African American high school students. Journal of Black Psychology 34 (2008), 94–120.
68. Krosnick, J.A., Issues in scientific integrity. Presentation at the annual conference of the Society for Personality and Social Psychology, Long Beach, CA, 2015.
69. Krueger, J.I., Funder, D.C., Towards a balanced social psychology: Causes, consequences, and cures for the problem-seeking approach to social behavior and cognition. Behavioral and Brain Sciences 27 (2004), 313–376.
70. Krueger, J.I., A componential model of situation effects, person effects, and situation-by-person interaction effects on social behavior. Journal of Research in Personality 43 (2009), 127–136.
71. Kunda, Z., The case for motivated reasoning. Psychological Bulletin 108 (1990), 480–498.
72. Lane, A., Mikolajczak, M., Treinen, E., Samson, D., Corneille, O., de Timary, P., Luminet, O., Failed replication of oxytocin effects on trust: The envelope T ask case. PLoS One, 10(9), 2015, e0137000, 10.1371/journal.pone.0137000.
73. Ledgerwood, A., Haines, E., Ratliff, K., Not nutting up or shutting up: Notes on the demographic disconnect in our field's best practices conversation. Retrieved 9/22/2015 from http://sometimesimwrong.typepad.com/wrong/2015/03/guest-post-not-nutting-up-or-shutting-up.html, 2015.
74. Levine, R., Chein, I., Murphy, G., The relation of the intensity of a need to the amount of perceptual distortion: A preliminary report. The Journal of Psychology 13 (1942), 283–293.
75. Lewandowsky, S., Oberauer, K., Gignac, G.E., NASA faked the moon landing—Therefore, (climate) science is a hoax an anatomy of the motivated rejection of science. Psychological Science 24 (2013), 622–633.
76. Lilienfeld, S.O., Can psychology become a science?. Personality and Individual Differences 49 (2010), 281–288.
77. Loeb, A., Benefits of diversity. Nature: Physics 10 (2014), 616–617.
78. Lord, C.G., Lepper, M.R., Preston, E., Considering the opposite: A corrective strategy for social judgment. Journal of Personality and Social Psychology 47 (1984), 1231–1243.
79. Lord, C.G., Ross, L., Lepper, M., Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence. Journal of Personality and Social Psychology 37 (1979), 2098–2109.
80. MacCoun, R.J., Biases in the interpretation and use of research results. Annual Review of Psychology 49 (1998), 259–287.
81. MacCoun, R.J., Perlmutter, S., Blind analysis as a correction for confirmatory bias in physics and psychology. To appear in Lilienfeld, S.O., Waldman, I., (eds.) Psychological science under scrutiny: Recent challenges and proposed solutions, 2015, John Wiley and Sons (in press).
82. Mahoney, M.J., Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research 1 (1977), 161–175.
83. Malle, B.F., The actor–observer asymmetry in attribution: A (surprising) meta-analysis. Psychological Bulletin 132 (2006), 895–919.
84. McCauley, C., Stitt, C.L., Segal, M., Stereotyping: From prejudice to prediction. Psychological Bulletin 87 (1980), 195–208.
85. McConnell, A.R., Leibold, J.M., Relations among the Implicit Association Test, discriminatory behavior, and explicit measures of racial attitudes. Journal of Experimental Social Psychology 37 (2001), 435–442.
86. McGinnies, E., Emotionality and perceptual defense. Psychological Review 56 (1949), 244–251.
87. McGuire, W.J., A contextualist theory of knowledge: Its implications for innovation and reform in psychological research. Advances in Experimental Social Psychology 16 (1983), 1–47.
88. Meehl, P.E., Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry 1 (1990), 108–141.
89. Mikolajczak, M., Gross, J.J., Lane, A., Corneille, O., de Timary, P., Luminet, O., Oxytocin makes people trusting, not gullible. Psychological Science 21:8 (2010), 1072–1074.
90. Milgram, S., Obedience to authority: An experimental view. 1974, Harper & Row, New York, NY.
91. Miller, G.A., Chapman, J.P., Misunderstanding analysis of covariance. Journal of Abnormal Psychology 110 (2001), 40–48.
92. Munro, G.D., Ditto, P.H., Biased assimilation, attitude polarization, and affect in reactions to stereotype-relevant scientific information. Personality and Social Psychology Bulletin 23 (1997), 636–653.
93. Nickerson, R.S., Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology 2 (1998), 175–220.
94. Open Science Collaboration, Estimating the reproducibility of psychological science. Science, 349, 2015, aac4716, 10.1126/science.aac4716.
95. Oswald, F.L., Mitchell, G., Blanton, H., Jaccard, J., Tetlock, P.E., Predicting ethnic and racial discrimination: A meta-analysis of IAT criterion studies. Journal of Personality and Social Psychology 105 (2013), 171–192.
96. Popper, K., The logic of scientific discovery. 1934, Routledge, New York.
97. Regner, I., Huguet, P., Monteil, J.M., Effects of socioeconomic status (SES) information on cognitive ability inferences: When low-SES students make use of a self-threatening stereotype. Social Psychology of Education 5 (2002), 253–269.
98. Rosenthal, R., Fode, K.L., The effect of experimenter bias on the performance of the albino rat. Behavioral Science 83 (1963), 183–189.
99. Ross, L., The intuitive psychologist and his shortcomings: Distortions in the attribution process. Advances in Experimental Social Psychology 10 (1977), 173–220.
100. Ross, L., Nisbett, R.E., The person and the situation: Perspectives of social psychology. 1991, McGraw-Hill, New York.
101. Ross, L., Lepper, M., Ward, A., History of social psychology: Insights, challenges, and contributions to theory and application. Fiske, S.T., Gilbert, D.T., Lindzey, G., (eds.) Handbook of social psychology, 2010, McGraw-Hill, New York, 3–50.
102. Sackett, P.R., Hardison, C.M., Cullen, M.J., On interpreting stereotype threat as accounting for African American–White differences on cognitive tests. American Psychologist 59 (2004), 7–13.
103. Sampson, R.J., Winship, C., Knight, C., Translating causal claims: Principles and strategies for policy-relevant criminology. Criminology & Public Policy 12 (2013), 587–616.
104. Schmader, T., Johns, M., Forbes, C., An integrated process model of stereotype threat effects on performance. Psychological Review 115 (2008), 336–356.
105. Schneider, D.J., Hastorf, A.H., Ellsworth, P., Person perception. 1979, Addison-Wesley Pub. Co., Reading, MA.
106. Sedikedes, C., Skowronski, J.J., The law of cognitive structure activation. Psychological Inquiry 2 (1991), 169–184.
107. Shen, H., Mind the gender gap. Nature 495 (2013), 22–24.
108. Simmons, J.P., Nelson, L.D., Simonsohn, U., False-positive psychology undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22 (2011), 1359–1366.
109. Simmons, J.P., Nelson, L.D., Simonsohn, U., A 21 word solution. Retrieved from http://ssrn.com/abstract=2160588, 2012.
110. Simpson, E.H., The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society, Series B, 13, 1951, 238-214.
111. Simpson's Paradox (n.d.) In Wikipedia. Retrieved 9/16/15 from https://en.wikipedia.org/wiki/Simpson's_paradox.
112. Snyder, M., Swann, W.B., Behavioral confirmation in social interaction: From social perception to social reality. Journal of Experimental Social Psychology 14 (1978), 148–162.
113. Stanovich, K.E., West, R.F., Toplak, M.E., Myside bias, rational thinking, and intelligence. Current Directions in Psychological Science 22 (2013), 259–264.
114. Steele, C.M., Aronson, J., Stereotype threat and the intellectual test performance of African Americans. Journal of Personality and Social Psychology 69 (1995), 797–811.
115. Steele, C.M., Aronson, J., Stereotype threat does not live by Steele and Aronson (1995) alone. American Psychologist 59 (2004), 47–55.
116. Tetlock, P.E., Political psychology or politicized psychology: is the road to scientific hell paved with good moral intentions?. Political Psychology 15 (1994), 509–529.
117. Trope, Y., Bassok, M., Information-gathering strategies in hypothesis-testing. Journal of Experimental Social Psychology 19 (1983), 560–576.
118. Van Hiel, A., Onraet, E., De Pauw, S.D., The relationship between social-cultural attitudes and behavioral measures of cognitive style: A meta-analytic integration of studies. Journal of Personality 78 (2010), 1765–1800.
119. Walton, G.M., Spencer, S.J., Latent ability grades and test scores systematically underestimate the intellectual ability of negatively stereotyped students. Psychological Science 20 (2009), 1132–1139.
120. Walton, G.M., Spencer, S.J., Erman, S., Affirmative meritocracy. Social Issues and Policy Review 7 (2013), 1–35.
121. Washburn, A.N., Morgan, G.S., Skitka, L.J., A checklist to facilitate objective hypothesis testing. Behavioral and Brain Sciences 38 (2015), 42–43.
122. West, T.V., Kenny, D.A., The truth and bias model of judgment. Psychological Review 118 (2011), 357–378.
123. Wicherts, J.M., Bakker, M., Molenaar, D., Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PLoS One, 6, 2011, e26828, 10.1371/journal.pone.0026828.
124. Wolfe, C., Spencer, S., Stereotypes and prejudice. American Behavioral Scientist 40 (1996), 176–185.
125. Yzerbyt, V.Y., Muller, D., Judd, C.M., Adjusting researchers' approach to adjustment: On the use of covariates when testing interactions. Journal of Experimental Social Psychology 40 (2004), 424–431.
126. Zuwerink, J.R., Devine, P.G., Attitude importance and resistance to persuasion: It's not not just the thought that counts. Journal of Personality and Social Psychology 70 (1996), 931–944.