Estimating the Expected Replicability of a Pattern of Correlations and Other Measures of Association

Multivariate Behavioral Research

Volumn 49, Issue 1, 2014, Pages 17-40

  Sherman, Ryne A ^a   Wood, Dustin ^b  

^a FLORIDA ATLANTIC UNIVERSITY   (United States)

^b WAKE FOREST UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84894288519     PISSN: 00273171     EISSN: None     Source Type: Journal    
DOI: 10.1080/00273171.2013.822785     Document Type: Article

References (84)

1. Arulampalam, W., Hartog, J., MaCurdy, T. and Theeuwes, J. 1997. Replication and re-analysis. Labour Economics, 4: 99 - 105.
2. Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Deinissen, J. J. A., Fiedler, K. and Wicherts, J. M. 2013. Recommendations for increasing replicability in psychology. European Journal of Personality, 27: 108 - 119.
3. Bem, D. J. 2011. Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100: 407 - 425.
4. Block, J. 1961. The Q-sort method in personality assessment and psychiatric research, Springfield, IL: Charles C. Thomas.
5. Block, J. and Block, J. H. 2006. Nursery school personality and political orientation two decades later. Journal of Research in Personality, 40: 734 - 749.
6. Block, J., Block, J. H. and Keyes, S. 1988. Longitudinal foretelling drug usage in adolescence: Early childhood personality and environmental precursors. Child Development, 59: 336 - 355.
7. Brooks, A., Roper, M., Wood, M., Daly, J. and Miller, J. 2000. Replication of software engineering experiments: Empirical foundations of computer science technical report., University of Strathclyde, Department of Computer and Information Sciences.
8. Brown, W. 1910. Some experimental results in the correlation of mental abilities. British Journal of Psychology, 3: 296 - 322.
9. Carlson, E. N., Furr, R. M. and Vazire, S. 2010. Do we know the first impressions we make? Evidence for idiographic meta-accuracy and calibration of first impressions. Social Psychological and Personality Science, 1: 94 - 98.
10. Chmielewski, M. and Watson, D. 2009. What is being assessed and why it matters: The impact of transient error on trait research. Journal of Personality and Social Psychology, 97: 186 - 202.
11. Cohen, J. 1988. Statistical power analysis for the behavioral sciences, (2nd ed.), Hillsdale, NJ: Erlbaum.
12. Cohen, J., Cohen, P., West, S. G. and Aiken, L. S. 2003. Applied multiple regression/correlation analysis for the behavioral sciences, (3rd ed.), Mahwah, NJ: Erlbaum.
13. Colvin, C. R. 1993. "Judgable" people: Personality, behavior, and competing explanations. Journal of Personality and Social Psychology, 64: 861 - 873.
14. Costa, P. T. Jr. and McCrae, R. R. 1992. The NEO-PI-R professional manual, Odessa, FL: Psychological Assessment Resources.
15. Cronbach, L. J. 1951. Coefficient alpha and the internal structure of tests. Psychometrika, 16: 297 - 343.
16. de Moor, M. H., Costa, P. T., Terracciano, A., Krueger, R. F., de Geus, E. J. C., Toshiko, T. and Metspalu, A. 2010. Meta-analysis of genome-wide association studies for personality. Molecular Psychiatry, 17: 337 - 349.
17. Eastwick, P. W. and Finkel, E. J. 2008. Sex differences in mate preferences revisited: Do people know what they initially desire in a romantic partner?. Journal of Personality and Social Psychology, 94: 245 - 264.
18. Fast, L. A. and Funder, D. C. 2008. Personality as manifest in word use: Correlations with self-report, acquaintance-report, and behavior. Journal of Personality and Social Psychology, 94: 334 - 346.
19. Fast, L. A. and Funder, D. C. 2010. Gender differences in the correlates of self-referent word use: Authority, entitlement and depressive symptoms. Journal of Personality, 78: 313 - 338.
20. Fournier, M. A., Moskowitz, D. S. and Zuroff, D. C. 2008. Integrating dispositions, signatures, and the interpersonal domain. Journal of Personality and Social Psychology, 94: 531 - 545.
21. Fournier, M. A., Moskowitz, D. S. and Zuroff, D. C. 2009. The interpersonal signature. Journal of Research in Personality, 43: 155 - 162.
22. Funder, D. C., Block, J. H. and Block, J. 1983. Delay of gratification: Some longitudinal personality correlates. Journal of Personality and Social Psychology, 44: 1198 - 1213.
23. Funder, D. C. and Dobroth, K. M. 1987. Differences between traits: Properties associated with interjudge agreement. Journal of Personality and Social Psychology, 52: 409 - 418.
24. Funder, D. C., Furr, R. M. and Colvin, C. R. 2000. The Riverside Behavioral Q-sort: A tool for the description of social behavior. Journal of Personality, 68: 451 - 489.
25. Furr, R. M. 2008. A framework for profile similarity: Integrating similarity, normativeness, and distinctiveness. Journal of Personality, 76: 1267 - 1316.
26. Furr, R. M. and Funder, D. C. 2004. Situational similarity and behavioral consistency: Subjective, objective, variable-centered, and person-centered approaches. Journal of Research in Personality, 38: 421 - 447.
27. Genz, A., Bretz, F., Bickel, P., Diggle, P., Fienberg, S. E., Gather, U., Olkin, I. and Zeger, S. 2009. " Computation of multivariate normal and t probabilities ". In Lecture notes in statistics (Vol. 195), Heidelberg, Germany: Springer-Verlag.
28. Genz, A., Bretz, F., Miwa, T., Mi, X., Leisch, F., Scheipl, F. and Hothorn, T. 2012. Mvtnorm: Multivariate normal and t distributions (R Package Version 0.9-9992) [Software]. Retrieved from http://CRAN.R-project.org/package=mvtnorm
29. Goldberg, L. R. 2008. The Eugene-Springfield community sample: Information available from the research participants (Tech. Rep. Vol. 48, No. 1), Eugene, OR: Oregon Research Institute.
30. Herrnson, P. S. 1995. Replication, verification, secondary analysis, and data collection in political science. Political Science and Politics, 28: 452 - 455.
31. Human Genome Research Institute. 2011. Genome-wide association studies Retrieved from http://www.genome.gov/20019523
32. Human, L. J. and Biesanz, J. C. 2011. Target adjustment and self-other agreement: Utilizing trait observability to disentangle judgability and self-knowledge. Journal of Personality and Social Psychology, 101: 202 - 216.
33. Jackson, J. J., Bogg, T., Walton, K., Wood, D., Harms, P. D., Lodi-Smith, J. L. and Roberts, B. W. 2009. Not all conscientiousness scales change alike: A multi-method, multi-sample study of age differences in the facets of conscientiousness. Journal of Personality and Social Psychology, 96: 446 - 459.
34. John, O. P. and Robins, R. W. 1993. Determinants of interjudge agreement on personality traits: The Big Five domains, observability, evaluativeness, and the unique perspective of the self. Journal of Personality, 61: 521 - 551.
35. John, O. P. and Srivastava, S. 1999. " The Big Five trait taxonomy: History, measurement, and theoretical perspectives ". In Handbook of personality: Theory and research, (2nd ed., Edited by: Pervin, L. A. and John, O. P. 102 - 139. New York, NY: Guilford Press.
36. Kelly, C. D. 2006. Replicating empirical research in behavioral ecology: How and why it should be done but rarely ever is. The Quarterly Review of Biology, 81: 221 - 236.
37. Klimstra, T. A., Luyckx, K., Hale, W. W. III, Goossens, L. and Meeus, W. H. 2010. Longitudinal associations between personality profile stability and adjustment in college students: Distinguishing among overall stability, distinctive stability, and within-time normativeness. Journal of Personality, 78: 1163 - 1184.
38. Koning, A. J. and Franses, P. H. 2003. Confidence intervals for Cronbach's alpha coefficient values. ERIM Report Series Reference No. ERS-2003-041-MKT, Available at SSRN: http//ssrn.com/abstract=423658
39. Lehrer, J. 2011, December. The truth wears off. The New Yorker, Retrieved from http://www.newyorker.com/
40. Levelt Committee. 2011. Interim report regarding the breach of scientific integrity committed by Prof. D. A. Stapel Retrieved from http://www.tilburguniversity.edu/nl/nieuws-en-agenda/commissie-levelt/interim-report.pdf
41. Lord, F. and Novick, M. R. 1968. Statistical theories of mental tests, New York, NY: Addison-Wesley.
42. Luminent, O., Bagby, R. M., Wagner, H. L., Taylor, G. J. and Parker, J. D. A. 1999. Relation between alexithymia and the Five Factor model of personality: A facet level analysis. Journal of Personality Assessment, 73: 345 - 358.
43. Mehl, M. R., Gosling, S. D. and Pennebaker, J. W. 2006. Personality in its natural habitat: Manifestations and implicit folk theories of personality in daily life. Journal of Personality and Social Psychology, 90: 862 - 877.
44. Morrell, K., Loan-Clarke, J. and Wilkinson, A. 2004. Go forth and replicate: A revisionist account of the value of replication in management studies., Loughborough University's Institutional Repository. Retrieved from https://dspace.lboro.ac.uk/dspace-jspui/
45. Morrell, K. and Lucas, J. W. 2012. The replication problem and its implications for policy studies. Critical Policy Studies, 6: 182 - 200.
46. Nave, C. S., Sherman, R. A., Funder, D. C., Hampson, S. E. and Goldberg, L. R. 2010. On the contextual independence of personality: Teachers' assessments predict directly observed behavior after four decades. Social and Personality Psychology Science, 1: 327 - 334.
47. Open Science Collaboration. 2012. An open, large-scale, collaborate effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7: 657 - 660.
48. Open Science Collaboration. in press. " The Reproducibility Project: A model of large-scale collaboration for empirical research on reproducibility ". In Implementing reproducible computational research (a volume in the R series), Edited by: Stodden, V., Leisch, F. and Peng, R. New York, NY: Taylor & Francis.
49. Park, C. L. 2004. What is the value of replicating other studies?. Research Evaluation, 13: 189 - 195.
50. Pavot, W. and Diener, E. 2008. The satisfaction with life scale and the emerging construct of life satisfaction. Journal of Positive Psychology, 3: 137 - 152.
51. Pennebaker, J. W., Francis, M. E. and Booth, R. J. 2001. Linguistic Inquiry and Word Count (LIWC): LIWC2001, Mahwah, NJ: Erlbaum.
52. R Development Core Team. 2012. " R: A language and environment for statistical computing [Computer software] ". Vienna, Austria: R Foundation for Statistical Computing.
53. Raizada, R. D. S. and Kriegeskorte, N. 2010. Pattern information fMRI: New questions which it opens up and challenges it faces. International Journal of Imaging Studies and Technology, 20: 31 - 41.
54. Raskin, R. and Terry, H. 1988. A principal-components analysis of the Narcissistic Personality Inventory and further evidence of its construct validity. Journal of Personality and Social Psychology, 54: 890 - 902.
55. Roberts, B. W. 2011, November. Personality psychology has a serious problem (and so do many other areas of psychology). P: The Online Newsletter for Personality Science, 6 Retrieved from http://www.personality-arp.org/
56. Roberts, B. W., Bogg, T., Walton, K. E., Chernyshenko, O. S. and Stark, S. E. 2004. A lexical investigation of the lower-order structure of conscientiousness. Journal of Research in Personality, 38: 164 - 178.
57. Rosenthal, R. 1990. Replication in behavioral research. Journal of Social Behavior and Personality, 5: 1 - 30.
58. Rosenthal, R. and Rosnow, R. L. 2008. Essentials of behavioral research: Methods and data analysis, (3rd ed.), New York, NY: McGraw-Hill.
59. Ryff, C. D. 1989a. Beyond Ponce de Leon and life satisfaction: New directions in quest of successful ageing. International Journal of Behavioral Development, 12: 35 - 55.
60. Ryff, C. D. 1989b. Happiness is everything, or is it? Explorations on the meaning of psychological well-being. Journal of Personality and Social Psychology, 57: 1069 - 1081.
61. Schimmack, U., Oishi, S., Furr, R. M. and Funder, D. C. 2004. Personality and life satisfaction: A facet level analysis. Personality and Social Psychological Bulletin, 30: 1062 - 1075.
62. Schmidt, S. 2009. Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13: 90 - 100.
63. Shedler, J. and Westen, D. 2007. The Shedler-Westen assessment procedure (SWAP): Making personality diagnosis clinically meaningful. Journal of Personality Assessment, 89: 41 - 55.
64. Sherman, R. A. and Funder, D. C. 2009. Evaluating correlations in studies of personality and behavior: Beyond the number of significant findings to be expected by chance. Journal of Research in Personality, 43: 1053 - 1063.
65. Sherman, R. A., Nave, C. S. and Funder, D. C. 2010. Situational similarity and personality predict behavioral consistency. Journal of Personality and Social Psychology, 99: 330 - 343.
66. Sherman, R. A., Nave, C. S. and Funder, D. C. 2012. Properties of persons and situations related to overall and distinctive personality-behavior congruence. Journal of Research in Personality, 46: 87 - 101.
67. Shull, F., Basili, V., Carver, J., Maldonado, J. C., Travassos, G. H., Mendonca, M. and Fabbri, S. 2002. Replicating software engineering experiments: Addressing the tacit knowledge problem. Proceedings on International Symposium Empirical Software Engineering,: 7 - 16.
68. Simmons, J. P., Nelson, L. D. and Simonsohn, U. 2011. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22: 1359 - 1366.
69. Spearman, C. C. 1910. Correlation calculated from faulty data. British Journal of Psychology, 3: 271 - 295.
70. SPSP Task Force on Responsible Conduct in Research Report. 2012. Retrieved from http://spsp.site-ym.com/resource/resmgr/files/task_force_on_responsible_co.pdf
71. Steiger, J. H. 1980. Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87: 245 - 251. doi: 10.1037/0033-2909.87.2.245
72. Vazire, S. 2010. Who knows what about a person? The self-other knowledge (SOKA) model. Journal of Personality and Social Psychology, 98: 281 - 300.
73. Vul, E., Harris, C., Winkielman, P. and Pashler, H. 2009. Puzzling high correlations in fMRI studies of emotion, personality, and social cognition. Perspectives on Psychological Science, 4: 274 - 290.
74. Wagenmakers, E.-J. 2012. A year of horrors. De Psychonoom, 27: 12 - 13.
75. Wagenmakers, E.-J., Wetzels, R., Borsboom, D. and van der Maas, H. L. J. 2011. Why psychologists must change the way they analyze their data: The case of psi. Comments on Bem (2011). Journal of Personality and Social Psychology, 100: 426 - 432.
76. Watson, D. 2001. Procrastination and the five-factor model: a facet level analysis. Personality and Individual Differences, 30: 149 - 158.
77. Watson, D., Hubbard, B. and Wiese, D. 2000. Self-other agreement in personality and affectivity: Effects of acquaintanceship, trait visibility, and assumed similarity. Journal of Personality and Social Psychology, 78: 546 - 558.
78. Wilcox, R. 2011. Modern statistics for the social and behavioral sciences: A practical introduction, Boca Raton, FL: CRC Press.
79. Wood, D. and Brumbaugh, C. C. 2009. Using revealed mate preferences to evaluate market force and differential preference explanations for mate selection. Journal of Personality and Social Psychology, 6: 1226 - 1244.
80. Wood, D., Nye, C. and Saucier, G. 2010. Identification and measurement of a more comprehensive set of person-descriptive trait markers from the English lexicon. Journal of Research in Personality, 44: 258 - 272.
81. Wood, D. and Wortman, J. 2012. Trait means and desirabilities as artifactual and real sources of differential stability of personality traits. Journal of Personality, 80: 665 - 701.
82. Wortman, J. and Wood, D. 2011. The personality traits of liked people. Journal of Research in Personality, 45: 519 - 528.
83. Yarkoni, T. 2010. The abbreviation of personality, or how to measure 200 personality scales with 200 items. Journal of Research in Personality, 44: 363 - 373.
84. Yong, E. 2012. Bad copy: In the wake of high-profile controversies, psychologists are facing up to problems with replication. Nature, 485: 298 - 300.