Treating stimuli as a random factor in social psychology: A new and comprehensive solution to a pervasive but largely ignored problem

Journal of Personality and Social Psychology

Volumn 103, Issue 1, 2012, Pages 54-69

  Judd, Charles M ^a   Westfall, Jacob ^a   Kenny, David A ^b  

^a UNIVERSITY OF COLORADO   (United States)

^b UNIVERSITY OF CONNECTICUT   (United States)

Author keywords

Mixed models; Random effects; Stimulus sampling

Indexed keywords

ANALYSIS OF VARIANCE; ASSOCIATION; HUMAN; METHODOLOGY; REVIEW; SOCIAL PSYCHOLOGY; STATISTICS;

ANALYSIS OF VARIANCE; CUES; HUMANS; PSYCHOLOGY, SOCIAL; RESEARCH DESIGN;

EID: 84871563877     PISSN: 00223514     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0028347     Document Type: Article

References (41)

1. Alnosaier, W. (2007). Kenward-Roger approximate F test for fixed effects in mixed linear models (Unpublished doctoral dissertation). Oregon State University, Corvallis, OR.
2. Baayen, R. H. (2008). Analyzing linguistic data: A practical introduction to statistics using R. Cambridge, England: Cambridge University Press.
3. Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59, 390-412. doi:10.1016/j.jml.2007.12.005
4. Bates, D., Maechler, M., & Bolker, B. (2011). lme4: Linear mixed-effects models using S4 classes (R package version 0.999375-39). Retrieved from http://CRAN.R-project.org/package=lme4
5. Bem, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100, 407-425. doi:10.1037/a0021524
6. Blair, I. V., Chapleau, K. M., & Judd, C. M. (2005). The use of Afrocentric features as cues for judgment in the presence of diagnostic information. European Journal of Social Psychology, 35, 59 - 68. doi:10.1002/ ejsp.232
7. Blair, I. V., Judd, C. M., & Chapleau, K. M. (2004). The influence ofAfrocentric facial features in criminal sentencing. Psychological Science, 15, 674-679. doi:10.1111/j.0956-7976.2004.00739.x
8. Blair, I. V., Judd, C. M., Sadler, M. S., & Jenkins, C. (2002). The role of Afrocentric features in person perception: Judging by features and categories. Journal of Personality and Social Psychology, 83, 5-25. doi:10.1037/0022-3514.83.1.5
9. Brunswik, E. (1955). Representative design and probabilistic theory in a functional psychology. Psychological Review, 62, 193-217. doi: 10.1037/h0047470
10. Clark, H. (1973). The language as fixed-effect fallacy: A critique of language statistics in psychological research. Journal of Verbal Learning and Verbal Behavior, 12, 335-359. doi:10.1016/S0022-5371(73) 80014-3
11. Cornfield, J., & Tukey, J. W. (1956). Average values of mean squares in factorials. The Annals of Mathematical Statistics, 27, 907-949. doi: 10.1214/aoms/1177728067
12. Correll, J., Park, B., Judd, C. M., & Wittenbrink, B. (2002). The police officer's dilemma: Using ethnicity to disambiguate potentially threatening individuals. Journal of Personality and Social Psychology, 83, 1314-1329. doi:10.1037/0022-3514.83.6.1314
13. Correll, J., Park, B., Judd, C. M., Wittenbrink, B., Sadler, M. S., & Keesee, T. (2007). Across the thin blue line: Police officers and racial bias in the decision to shoot. Journal of Personality and Social Psychology, 92, 1006-1023. doi:10.1037/0022-3514.92.6.1006
14. Fazio, R. H., Sanbonmatsu, D. M., Powell, M. C., & Kardes, F. R. (1986). On the automatic activation of attitudes. Journal of Personality and Social Psychology, 50, 229-238. doi:10.1037/0022-3514.50.2.229
15. Fears, T. R., Benichou, J., & Gail, M. H. (1996). A reminder of the fallibility of the Wald Statistic. The American Statistician, 50, 226-227. doi:10.2307/2684659
16. Forster, K. I., & Dickenson, R. G. (1976). More on the language-as-fixedeffect fallacy: Monte Carlo estimates of error rates for F1, F2, F , and min F . Journal of Verbal Learning and Verbal Behavior, 15, 135-142. doi:10.1016/0022-5371(76)90014-1
17. Gelman, A., & Hill, J. (2007). Data analysis using regression and multilevel/ hierarchical models. Cambridge, England: Cambridge University Press.
18. Green, B. F., & Tukey, J. W. (1960). Complex analyses of variance: General problems. Psychometrika, 25, 127-152. doi:10.1007/ BF02288577
19. Halekoh, U., & Højsgaard, S. (2011). pbkrtest: Parametric bootstrap Kenward-Roger based methods for mixed model comparison (R package version 0.1.3). Retrieved from http://CRAN.R-project.org/package=pbkrtest
20. Hamilton, D. L., Katz, L. B., & Leirer, V. O. (1980). Cognitive representation of personality impressions: Organizational processes in first impression formation. Journal of Personality and Social Psychology, 39, 1050-1063. doi:10.1037/h0077711
21. Hox, J. J. (2002). Multilevel analysis: Techniques and applications. Mahwah, NJ: Erlbaum.
22. Jaeger, T. F. (2008). Categorical data analysis: Away from ANOVAs (transformation or not) and towards logit mixed models. Journal of Memory and Language, 59, 434-446. doi:10.1016/j.jml.2007.11.007
23. Kenny, D. A. (1985). Quantitative methods for social psychology. In G. Lindzey & E. Aronson (Eds.), Handbook of social psychology (Vol. 1, 3rd ed., pp. 487-508). New York, NY: Random House.
24. Kenny, D. A., Kashy, D. A., & Bolger, N. (1998). Data analysis in social psychology. In D. T. Gilbert, S. T. Fiske, & G. Lindzey (Eds.), Handbook of social psychology (Vol. 1, 4th ed., pp. 233-268). New York, NY: McGraw-Hill.
25. Kenward, M. G., & Roger, J. H. (1997). Small sample inference for fixed effects from restricted maximum likelihood. Biometrics, 53, 983-997. doi:10.2307/2533558
26. Lehrer, J. (2010, December 13). The truth wears off: Is there something wrong with the scientific method? The New Yorker. Retrieved from http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer
27. Nosek, B. A., & Banaji, M. R. (2001). The go/no-go association task. Social Cognition, 19, 625-666. doi:10.1521/soco.19.6.625.20886
28. Pinheiro, J. C., & Bates, D. M. (2000). Mixed-effect models in S and S-PLUS. New York, NY: Springer.
29. Raaijmakers, J. G. W. (2003). A further look at the "language-as-fixedeffect fallacy" [Special issue]. Canadian Journal of Experimental Psychology/Revue canadienne de psychologie expérimentale, 57, 141-151. doi:10.1037/h0087421
30. Raaijmakers, J. G. W., Schrijnemakers, J. M. C., & Gremen, F. (1999). How to deal with "the language-as-fixed-effect fallacy": Common misconceptions and alternative solutions. Journal of Memory and Language, 41, 416-426. doi:10.1006/jmla.1999.2650
31. Raudenbusch, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, CA: Sage.
32. Rietveld, T., & van Hout, R. (2007). Analysis of variance for repeated measures designs with words as a nested random or fixed factor. Behavior Research Methods, 39, 735-747. doi:10.3758/BF03192964
33. Rouder, J. N., Lu, J., Sun, D., Speckman, P., Morey, R., & Naveh-Benjamin, M. (2007). Signal detection models with random participant and item effects. Psychometrika, 72, 621-642. doi:10.1007/s11336-005- 1350-6
34. Schaalje, G. B., McBride, J. B., & Fellingham, G. W. (2002). Adequacy of approximations to distributions of test statistics in complex mixed linear models using SAS Proc MIXED. Journal of Agricultural, Biological, and Environmental Statistics, 7, 512-524. doi:10.1198/108571102726
35. Singer, J. D., & Willett, J. B. (2003). Applied longitudinal data analysis: Modeling change and event occurrence. New York, NY: Oxford University Press.
36. Snijders, T., & Bosker, R. (1999). An introduction to basic and advanced multilevel modeling. Thousand Oaks, CA: Sage.
37. Spilke, J., Piepho, H.-P., & Hu, X. (2005). A simulation study on tests of hypotheses and confidence intervals for fixed effects in mixed models for blocked experiments with missing data. Journal of Agricultural, Biological, and Environmental Statistics, 10, 374-389. doi:10.1198/ 108571105X58199
38. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011). Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of Personality and Social Psychology, 100, 426-432. doi:10.1037/a0022790
39. Wells, G. L., & Windschitl, P. D. (1999). Stimulus sampling and social psychological experimentation. Personality and Social Psychological Bulletin, 25, 1115-1125. doi:10.1177/01461672992512005
40. Wickens, T. D., & Keppel, G. (1983). On the choice of design and of test statistics in the analysis of experiments with sampled materials. Journal of Verbal Learning and Verbal Behavior, 22, 296-309. doi:10.1016/ 0S022-5371(83)90208-6
41. Winer, B. J. (1971). Statistical principles in experimental design. New York, NY: McGraw-Hill.