Sexy but often unreliable: The impact of unreliability on the replicability of experimental findings with implicit measures

Personality and Social Psychology Bulletin

Volumn 37, Issue 4, 2011, Pages 570-583

  Lebel, Etienne P ^a   Paunonen, Sampo V ^a  

^a UNIVERSITY OF WESTERN ONTARIO   (Canada)

Author keywords

Implicit measures; Measurement error; Reliability; Replicability; Replication

Indexed keywords

ARTICLE; ATTITUDE; HUMAN; METHODOLOGY; MONTE CARLO METHOD; PREDICTIVE VALUE; PSYCHOLOGY; PSYCHOMETRY; REPRODUCIBILITY;

ATTITUDE; HUMANS; MONTE CARLO METHOD; PREDICTIVE VALUE OF TESTS; PSYCHOLOGY; PSYCHOMETRICS; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN;

EID: 79953684004     PISSN: 01461672     EISSN: 15527433     Source Type: Journal    
DOI: 10.1177/0146167211400619     Document Type: Article

References (82)

1. Algom, D., Chajut, E., & Shlomo, L. (2004). A rational look at the emotional Stroop phenomenon: A generic slowdown, not a Stroop effect. Journal of Experimental Psychology: General, 133, 323-338.
2. Arkes, H. R., & Tetlock, P. E. (2004). Attributions of implicit prejudice, or would Jesse Jackson 'fail' the Implicit Association Test? Psychological Inquiry, 15, 257-278.
3. Barden, J., Maddux, W. W., Petty, R. E., & Brewer, M. B. (2004). Contextual moderation of racial bias: The impact of social roles on controlled and automatically activated attitudes. Journal of Personality and Social Psychology, 87, 5-22.
4. Bargh, J. A. (1999). The cognitive monster: The case against the controllability of automatic stereotype effects. In S. Chaiken & Y. Trope (Eds.), Dual-process theories in social psychology (pp. 361-382). New York, NY: Guilford.
5. Baugh, F. (2002). Correcting effect sizes for score reliability: A reminder that measurement and substantive issues are linked inextricably. Educational and Psychological Measurement, 62, 254-263.
6. Blanton, H., & Jaccard, J. (2006). Arbitrary metrics in psychology. American Psychologist, 61, 27-41.
7. Blanton, H., & Jaccard, J. (2008). Unconscious racism: A concept in pursuit of a measure. Annual Review of Sociology, 34, 277-297.
8. Blanton, H., Jaccard, J., Gonzales, P., & Christie, C. (2006). Decoding the Implicit Association Test: Perspectives on criterion prediction. Journal of Experimental Social Psychology, 42, 192-212.
9. Blascovich, J., Loomis, J., Beall, A. C., Swinth, K. R., Hoyt, C. L., & Bailenson, J. N. (2002). Immersive virtual environment technology as a methodological tool for social psychology. Psychology Inquiry, 13, 103-124.
10. Borkenau, P., & Mauer, N. (2007). Well-being and the accessibility of pleasant and unpleasant concepts. European Journal of Personality, 21, 169-189.
11. Borkenau, P., Paelecke, M., & Yu, R. (2009). Personality and lexical decision times for evaluative words. European Journal of Personality, 24, 123-136.
12. Bosson, J. K., Lakey, C. E., Campbell, W. K., Zeigler-Hill, V., Jordan, C. H., & Kernis, M. H. (2008). Untangling the links between narcissism and self-esteem: A theoretical and empirical review. Personality and Social Psychology Compass, 2/3, 1415-1439.
13. Bosson, J. K., Swann, W. B., & Pennebaker, J. W. (2000). Stalking the perfect measure of implicit self-esteem: The blind men and the elephant revisited? Journal of Personality and Social Psychology, 79, 631-643.
14. Buchner, A., & Wippich, W. (2000). On the reliability of implicit and explicit memory measures. Cognitive Psychology, 40, 227-259.
15. Campbell, W. K., Bosson, J. K., Goheen, T. W., Lakey, C. E., & Kernis, M. H. (2007). Do narcissists dislike themselves "deep down inside"? Psychological Science, 18, 227-229.
16. Charles, E. (2005). The correction for attenuation due to measurement error: Clarifying concepts and creating confidence sets. Psychological Methods, 10, 206-226.
17. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
18. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 12, 1-16.
19. Cronbach, L. J. (1958). Proposals leading to analytic treatment of social perception scores. In R. Tagiuri & L. Petrullo (Eds.), Person perception and interpersonal behavior (pp. 353-379). Stanford, CA: Stanford University Press.
20. Cronbach, L. J. (2004). My current thoughts on coefficient alpha and successor procedures. Educational and Psychological Measurement, 64, 391-418.
21. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: Wiley.
22. Cumming, G. (2008). Replication and p intervals: p values predict the future only vaguely, but confidence intervals do much better. Perspectives on Psychological Science, 3, 286-300.
23. Cunningham, W. A., Preacher, K. J., & Banaji, M. R. (2001). Implicit attitude measurement: Consistency, stability, and convergent validity. Psychological Science, 12, 163-170.
24. De Houwer, J. (2003). The Extrinsic Affective Simon Task. Experimental Psychology, 50, 77-85.
25. De Houwer, J., & De Bruycker, E. (2007). The identificationEAST as a valid measure of implicit attitudes toward alcohol-related stimuli. Journal of Behavior Therapy and Experimental Psychiatry, 38, 133-143.
26. De Raedt, R., Schacht, R., Franck, E., & De Houwer, J. (2006). Self-esteem and depression revisited: Implicit positive selfesteem in depressed patients? Behaviour Research and Therapy, 44, 1017-1028.
27. DeShon, R. P. (1998). A cautionary note on measurement error corrections in structural equation models. Psychological Methods, 3, 412-423.
28. DeShon, R. P. (2004). Measures are not invariant across groups without error variance homogeneity. Psychology Science, 46, 137-149.
29. Dovidio, J. F., Kawakami, K., & Gaertner, S. L. (2002). Implicit and explicit prejudice and interracial interaction. Journal of Personality and Social Psychology, 82, 62-28.
30. Duhachek, A., & Iacobucci, D. (2004). Alpha's standard error (ASE): An accurate and precise confidence interval estimate. Journal of Applied Psychology, 89, 792-808.
31. Edwards, J. R. (2002). Ten difference score myths. Organizational Research Methods, 4, 265-287.
32. Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175-191.
33. Fazio, R. H., Jackson, J. R., Dunton, B. C., & Williams, C. J. (1995). Variability in automatic activation as an unobtrusive measure of racial attitudes: A bona fide pipeline? Journal of Personality and Social Psychology, 69, 1013-1027.
34. Fazio, R. H., & Olson, M. A. (2003). Implicit measures in social cognition research: Their meaning and use. Annual Review of Psychology, 54, 297-327.
35. Ferguson, M. J., & Bargh, J. A. (2007). Beyond the attitude object: Implicit attitudes spring from object-centered contexts. In B. Wittenbrink & N. Schwarz (Eds.), Implicit measures of attitudes: Procedures and controversies (pp. 216-246). New York, NY: Guilford.
36. Galdi, S., Arcuri, L., & Gawronski, B. (2008). Automatic mental associations predict future choices of undecided decision-makers. Science, 321, 1100-1102.
37. Gawronski, B. (2002). What does the Implicit Association Test measure? A test of the convergent and discriminant validity of prejudice related IATs. Experimental Psychology, 49, 171-180.
38. Gawronski, B., Cunningham, W. A., LeBel, E. P., & Deutsch, R. (2010). Attentional influences on affective priming: Does categorization influence spontaneous evaluations of multiply categorizable objects? Cognition and Emotion, 24, 1008-1025.
39. Gawronski, B., Deutsch, R., & Banse, R. (in press). Response interference tasks as indirect measures of automatic associations. In K. C. Klauer, C. Stahl, & A. Voss (Eds.), Cognitive methods in social psychology. New York, NY: Guilford.
40. Gawronski, B., Deutsch, R., LeBel, E. P., & Peters, K. R. (2008). Response interference as a mechanism underlying implicit measures: Some traps and gaps in the assessment of mental associations with experimental paradigms. European Journal of Psychological Assessment, 24, 218-225.
41. Gawronski, B., LeBel, E. P., & Peters, K. P. (2007). What do implicit measures tell us? Scrutinizing the validity of three common assumptions. Perspectives on Psychological Science, 2, 181-193.
42. Gray, N. S., McCulloch, M. J., Smith, J., Morris, M., & Snowden, R. J. (2003). Violence viewed by psychopathic murderers: Adapting a revealing test may expose those psychopaths who are most likely to kill. Nature, 423, 497-498.
43. Greenwald, A. G., & Banaji, M. R. (1995). Implicit social cognition: Attitudes, self-esteem, and stereotypes. Psychological Review, 102, 4-27.
44. Greenwald, A. G., McGhee, D. E., & Schwartz, J. L. K. (1998). Measuring individual differences in implicit cognition: The Implicit Association Test. Journal of Personality and Social Psychology, 74, 1464-1480.
45. Greenwald, A. G., Nosek, B. A., & Banaji, M. R. (2003). Understanding and using the Implicit Association Test: I. An improved scoring algorithm. Journal of Personality and Social Psychology, 85, 197-216.
46. Hopkins, K. D., & Hopkins, B. R. (1979). The effect of the reliability of the dependent variable on power. Journal of Special Education, 13, 463-466.
47. Jaccard, J., & Wan, C. K. (1995). Measurement error in the analysis of interaction effects between continuous predictors using multiple regression: Multiple indicator and structural equation approaches. Psychological Bulletin, 117, 348-357.
48. Johnson, R. E., & Steinman, L. (2009). Use of implicit measures for organizational research: An empirical example. Canadian Journal of Behavioural Science, 41, 202-212.
49. Jordan, C. H., Spencer, S. J., Zanna, M. P., Hoshino-Browne, E., & Correll, J. (2003). Secure and defensive high self-esteem. Journal of Personality and Social Psychology, 85, 969-978
50. Kashy, D. A., Donnellan, M. B., Ackerman, R. A., & Russell, D. W. (2009). Reporting and interpreting research in PSPB: Practices, principles, and pragmatics. Personality and Social Psychology Bulletin, 35, 1131-1142.
51. Kawakami, K., & Dovidio, J. F. (2001). Implicit stereotyping: How reliable is it? Personality and Social Psychology Bulletin, 27, 212-225.
52. Kopriva, R. J., & Shaw, D. G. (1991). Power estimates: The effect of dependent variable reliability on the power of one-factor ANOVAs. Educational and Psychological Measurement, 51, 585-595.
53. LeBel, E. P., & Gawronski, B. (2009). How to find what's in a name: Scrutinizing the optimality of five scoring algorithms for the name-letter task. European Journal of Personality, 23, 85-106.
54. Lipsey, M. W., & Wilson, D. B. (1993). The efficacy of psychological, education, and behavioral treatment: Confirmation from meta-analysis. American Psychologist, 48, 1181-1209.
55. Liu, Y., & Zumbo, B. D. (2007). The impact of outliers on Cronbach's coefficient alpha estimate of reliability. Educational and Psychological Measurement, 67, 620-634.
56. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
57. Meier, B., & Perrig, W. J. (2000). Low reliability of perceptual priming: Consequences for the interpretation of functional dissociations between explicit and implicit memory. Quarterly Journal of Experimental Psychology, 53, 211-233.
58. Nosek, B. A., & Banaji, M. R. (2001). The Go/No-go Association Task. Social Cognition, 19, 625-666.
59. Nunnally, J. C. (1982). Reliability of measurement. In H. E. Mitzel (Ed.), Encyclopedia of educational research (pp. 1589-1601). New York, NY: Free Press.
60. Nuttin, M. J., Jr. (1985). Narcissism beyond gestalt and awareness: The name letter effect. European Journal of Social Psychology, 64, 723-739.
61. Olson, M. A., & Fazio, R. H. (2003). Relations between implicit measures of prejudice: What are we measuring? Psychological Science, 14, 636-639.
62. Papies, E. K., Stroebe, W., & Aarts, H. (2009). Who likes it more? Restrained eaters' implicit attitudes towards food. Appetite, 53, 279-287.
63. Paunonen, S. V. (1984). The reliability of aggregated measurements: Lessons to be learned from psychometric theory. Journal of Research in Personality, 18, 383-394.
64. Payne, B. K., Cheng, S. M., Govorun, O., & Stewart, B. D. (2005). An inkblot for attitudes: Affect misattribution as implicit measurement. Journal of Personality and Social Psychology, 89, 277-293.
65. Perez, L. A., Peynircio, Z. F., & Blaxton, T. A. (1998). Developmental differences in implicit and explicit memory performance. Journal of Experimental Child Psychology, 70, 167-185.
66. Robie, C., & Komar, S. G. (2007). Simulation, computer approach. In S. G. Rogelberg (Ed.), Encyclopedia of industrial and organizational psychology (pp. 723-724). Thousand Oaks, CA: Sage.
67. Rogers, W. T., & Hopkins, K. D. (1988). Power estimates in the presence of a covariate and measurement error. Educational and Psychological Measurement, 48, 647-56.
68. Schimmack, U., & Diener, E. (2003). Predictive validity of explicit and implicit self-esteem for subjective well-being. Journal of Research in Personality, 37, 100-106.
69. Schmidt, F. L., & Hunter, J. E. (1996). Measurement error in psychological research: Lessons from 27 research scenarios. Psychological Methods, 1, 199-223.
70. Schmukle, S. C. (2005). Unreliability of the dot probe task. EuropeanJournal of Personality, 19, 595-605.
71. Schott, B. H., Henson, R. N., Richardson-Klavehn, A., Becker, C., Thoma, V., Heinze, H.-J., & Düzel, E. (2005). Redefining implicit and explicit memory: The functional neuroanatomy of priming, remembering, and control of retrieval. Proceedings of National Academy Science, 102, 1257-1262.
72. Spruyt, A., Hermans, D., De Houwer, J., Vandekerckhove, J., & Eelen, P. (2007). On the predictive validity of indirect attitude measures: Prediction of consumer choice behavior on the basis of affective priming in the picture-picture naming task. Journal of Experimental Social Psychology, 43, 599-610.
73. Sutcliffe, J. P. (1958). Error of measurement and the sensitivity of a test of significance. Psychometrika, 23, 9-17.
74. Teige, S., Schnabel, K., Banse, R., & Asendorpf, J. B. (2004). Assessment of multiple implicit self-concept dimensions using the Extrinsic Affective Simon Task. European Journal of Personality, 18, 495-520.
75. Thye, S. R. (2000). Reliability in experimental sociology. Social Forces, 78, 1277-1309.
76. Uhlmann, E. L., Pizarro, D. A., & Bloom, P. (2008). Varieties of unconscious social cognition. Journal for the Theory of Social Behaviour, 38, 293-322.
77. Williams, R. H., & Zimmerman, D. W. (1989). Statistical power analysis and reliability of measurement. Journal of General Psychology, 116, 359-369.
78. Wittenbrink, B. (2007). Measuring attitudes through priming. In B. Wittenbrink & N. Schwarz (Eds.), Implicit measures of attitudes: Procedures and controversies (pp. 17-58). New York, NY: Guilford.
79. Wittenbrink, B., Judd, C. M., & Park, B. (1997). Evidence for racial prejudice at the implicit level and its relationship with questionnaire measures. Journal of Personality and Social Psychology, 72, 262-274.
80. Wittenbrink, B., & Schwarz, N. (Eds.). (2007). Implicit measures of attitudes. New York, NY: Guilford.
81. Woltz, D. J., & Shute, V. J. (1993). Individual difference in repetition priming and its relationship to declarative knowledge acquisition. Intelligence, 17, 333-359.
82. Zeigler-Hill, V. (2006). Discrepancies between implicit and explicit self-esteem: Implications for narcissism and self-esteem instability. Journal of Personality, 74, 119-143.