Simpson's paradox in psychological science: A practical guide

Frontiers in Psychology

Volumn 4, Issue AUG, 2013, Pages

  Kievit, Rogier A ^a,b   Frankenhuis, Willem E ^c   Waldorp, Lourens J ^a   Borsboom, Denny ^a  

^a UNIVERSITY OF AMSTERDAM   (Netherlands)

^b MEDICAL RESEARCH COUNCIL   (United Kingdom)

^c RADBOUD UNIVERSITY NIJMEGEN   (Netherlands)

Author keywords

Ecological fallacy; Measurement; Paradox; Reductionism; Simpson's paradox; Statistical inference

Indexed keywords

EID: 84885440627     PISSN: None     EISSN: 16641078     Source Type: Journal    
DOI: 10.3389/fpsyg.2013.00513     Document Type: Article

References (110)

1. Aldrich, J. (1995). Correlations genuine and spurious in Pearson and Yule. Stat. Sci. 10, 364-376.
2. Anderson, K. J., Revelle, W., and Lynch, M. J. (1989). Caffeine, impulsivity, and memory scanning: a comparison of two explanations for the Yerkes-Dodson Effect. Motiv. Emot. 13, 1-20. doi: 10.1007/BF00995541
3. Bandyoapdhyay, P. S., Nelson, D., Greenwood, M., Brittan, G., and Berwald, J. (2011). The logic of Simpson's paradox. Synthese 181, 185-208. doi: 10.1007/s11229-010-9797-0
4. Batty, G. D., Deary, I. J., Schoon, I., Emslie, C., Hunt, K., and Gale, C. R. (2008). Childhood mental ability and adult alcohol intake and alcohol problems: the 1970 British Cohort Study. Am. J. Public Health 98, 2237-2243. doi: 10.2105/AJPH.2007.109488
5. Bauer, D. B., and Curran, P. J. (2003). Distributional assumptions of growth mixture models: implications for overextraction of latent trajectory classes. Psychol. Methods 8, 338-363. doi: 10.1037/1082-989X.8.3.338
6. Benjamin, L. T., Cavell, T. A., and Shallenberger, W. R. (1987). "Staying with initial answers on objective tests: is it a myth?," in Handbook on Student Development: Advising, Career Development, and Field Placement, eds M. E. Ware, and R. J. Millard (Hillsdale, NJ: Lawrence Erlbaum), 45-53.
7. Bickel, P. R., Hammel, E. A., and O'Connell, J. W. (1975). Sex bias in graduate admissions: data from Berkeley. Science 187, 398-404.
8. Block, N. (1995). How heritability misleads about race. Cognition 56, 99-128. doi: 10.1016/0010-0277(95)00678-R
9. Blyth, C. R. (1972). On Simpson's paradox and the sure-thing principle. J. Am. Statist. Assoc. 67, 364-366. doi: 10.1080/01621459.1972.10482387
10. Borsboom, D., Kievit, R. A., Cervone, D. P., and Hood, S. B. (2009). "The two disciplines of scientific psychology, or: the disunity of psychology as a working hypothesis," in Developmental Process Methodology in the Social and Developmental Sciences, eds J. Valsiner, P. C. M. Molenaar, M. C. D. P. Lyra, and N. Chaudary (New York, NY: Springer), 67-89.
11. Breusch, T. S., and Pagan, A. R. (1979). Simple test for heteroscedasticity and random coefficient variation. Econometrica 47, 1287-1294. doi: 10.2307/1911963
12. Cartwright, N. (1979). Causal laws and effective strategies. Nous 13, 419-437. doi: 10.2307/2215337
13. Cattell, R. B. (1952). The three basic factor-analytic research designs-their interrelations and derivatives. Psychol. Bull. 49, 499-520. doi: 10.1037/h0054245
14. Chater, N., Vlaev, I., and Grinberg, M. (2008). A new consequence of Simpson's paradox: stable cooperation in one-shot prisoner's dilemma from populations of individualistic learners. J. Exp. Psychol. Gen. 137, 403-421. doi: 10.1037/0096-3445.137.3.403
15. Chuang, J. S., Rivoire, O., and Leibler, S. (2009). Simpson's paradox in a synthetic microbial system. Science 323, 272-275. doi: 10.1126/science.1166739
16. Cohen, M. R., and Nagel, E. (1934). An Introduction to Logic and Scientific Method. New York, NY: Harcourt, Brace and Company.
17. Conger, A. J. (1974). A revised definition for suppressor variables: a guide to their identification and interpretation. Educ. Psychol. Meas. 34, 35-46. doi: 10.1177/001316447403400105
18. Curley, S. P., and Browne, G. J. (2001). Normative and descriptive analyses of Simpson's paradox in decision making. Organ. Behav. Hum. Decis. Process. 84, 308-333. doi: 10.1006/obhd.2000.2928
19. Dawid, A. P. (1979). Conditional independence in statistical theory. J. Roy. Stat. Soc. Ser. B (Methodol.) 41, 1-31.
20. Dienes, Z. (2011). Bayesian versus orthodox statistics: which side are you on? Perspect. Psychol. Sci. 6, 274-290. doi: 10.1177/1745691611406920
21. Dutilh, G., Wagenmakers, E. J., Visser, I., and van der Maas, H. L. (2011). A phase transition model for the speed-accuracy trade-offin response time experiments. Cogn. Sci. 35, 211-250. doi: 10.1111/j.1551-6709.2010.01147.x
22. Fiedler, K. (2000). Beware of samples!: a cognitive-ecological sampling approach to judgment biases. Psychol. Rev. 107, 659-676. doi: 10.1037/0033-295X.107.4.659
23. Fiedler, K. (2008). The ultimate sampling dilemma in experience- based decision making. J. Exp. Psychol. Learn. Mem. Cogn. 34, 186-203. doi: 10.1037/0278-7393.34.1.186
24. Fiedler, K., Freytag, P., and Meisder, T. (2009). Pseudocontingencies: an integrative account of an intriguing cognitive illusion. Psychol. Rev. 116, 187-206. doi: 10.1037/a0014480
25. Fiedler, K., Walther, E., Freytag, P., and Nickel, S. (2003). Inductive reasoning and judgment interference: experiments on Simpson's paradox. Pers. Soc. Psychol. Bull. 29, 14-27. doi: 10.1177/0146167202238368
26. Fisher, R. A. (1925). Statistical Methods for Research Workers. Edinburgh: Oliver and Boyd.
27. Fitts, P. M. (1954). The information capacity of the human motor system in controlling the amplitude of movement. J. Exp. Psychol. 47, 381-391. doi: 10.1037/h0055392
28. Fraley, C., and Raftery, A. E. (1998a). MCLUST: Software for Model-Based Cluster and Discriminant Analysis. Department of Statistics, University of Washington: Technical Report No. 342.
29. Fraley, C., and Raftery, A. E. (1998b). How many clusters? Which clustering method? Answers via model-based cluster analysis. Comput J. 41, 578-588.
30. Friendly, M., Monette, G., and Fox, J. (2013). Elliptical insights: understanding statistical methods through elliptical geometry. Stat. Sci. 28(1), 1-39. doi: 10.1214/12-STS402
31. Froelich, W. (2013). "Mining association rules from database tables with the instances of Simpson's paradox," in Advances in Databases and Information Systems, eds M. Tadeusz, H. Theo, and W. Robert (Berlin; Heidelberg: Springer), 79-90.
32. Greenland, G. (2010). Simpson's paradox from adding constants in contingency tables as an example of bayesian noncollapsibility. Am. Stat. 64, 340-344. doi: 10.1198/tast.2010.10006
33. Halpin, P. F., Dolan, C. V., Grasman, R. P., and De Boeck, P. (2011). On the relation between the linear factor model and the latent profile model. Psychometrika 76, 564-583. doi: 10.1007/s11336-011-9230-8
34. Hamaker, E. L., Nesselroade, J. R., and Molenaar, P. C. M. (2007). The integrated trait- state model. J. Res. Pers. 41, 295-315. doi: 10.1016/j.jrp.2006.04.003
35. Hernán, M. A., Clayton, D., and Keiding, N. (2011). The Simpson's paradox unraveled. Int. J. Epidemiol. 40, 780-785. doi: 10.1093/ije/dyr041
36. Herrnstein, R. J., and Murray, C. (1994). Bell curve: Intelligence and class structure in American life. (New York, NY: Free Press).
37. Hintzman, D. L. (1980). Simpson's paradox and the analysis of memory retrieval. Psychol. Rev. 87, 398-410. doi: 10.1037/0033-295X.87.4.398
38. Hintzman, D. L. (1993) On variability, Simpson's paradox, and the relation between recognition and recall: reply to Tulving and Flexser. Psychol. Rev. 100, 143-148.
39. Imai, K., Lu, Y., and Strauss, A. (2011). Eco: R package for ecological inference in 2x2 tables. J. Stat. Softw. 42, 1-23.
40. Jensen, A. R. (1998). The g Factor: The Science of Mental Ability. Westport, CT: Praeger.
41. Julious, S. A., and Mullee, M. A. (1994). Confounding and Simpson's paradox. Br. Med. J. 209, 1480-1481. doi: 10.1136/bmj.309.6967.1480
42. Kanazawa, S. (2010). Evolutionary psychology and intelligence research. Am. Psychol. 65, 279-289. doi: 10.1037/a0019378
43. Kaufman, L., and Rousseeuw, P. J. (2008) Introduction, in Finding Groups in Data: An Introduction to Cluster Analysis. Hoboken, NJ: John Wiley and Sons, Inc.
44. Kievit, R. A., and Epskamp, S. (2012). Simpsons: Detecting Simpson's Paradox. R package version 0.1.0. Available online at: http://CRAN.R-project.org/package=Simpsons
45. Kievit, R. A., Romeijn, J. W., Waldorp, L. J., Scholte, H. S., Wicherts, J. M., and Borsboom, D. (2011). Mind the gap: a psychometric approach to the reduction problem. Psychol. Inq. 22, 67-87. doi: 10.1080/1047840X.2011.550181
46. Kievit, R. A., van Rooijen, H., Wicherts, J. M., Waldorp, L. J., Kan, K. J., Scholte, H. S., et al. (2012a). Intelligence and the brain: a model-based approach. Cogn. Neurosci. 3, 89-97. doi: 10.1080/17588928.2011.628383
47. Kievit, R. A., Waldorp, L. J., Kan, K. J., and Wicherts, J. M. (2012b). Causality: populations, individuals, and assumptions. Eur. J. Pers. 26, 400-401.
48. King, G. (1997). A Solution to the Ecological Inference Problem: Reconstructing Individual Behavior from Aggregate Data. Princeton: Princeton University Press.
49. King, G. (2004). EI: a program for ecological inference. J. Stat. Softw. 11, 1-38.
50. King, G., and Roberts, M. (2012). EI: a (n R) program for ecological inference. 1-27. Available online at: http://gking.harvard.edu/files/gking/files/ei.pdf
51. Kraemer, H. C., Wilson, K. A., and Hayward, C. (2006). Lifetime prevalence and pseudocomorbidity in psychiatric research. Arch. Gen. Psychiatry 63, 604-608. doi: 10.1001/archpsyc.63.6.604
52. Kramer, G. H. (1983). The ecological fallacy revisited: aggregate-versus individual-level findings on economics and elections, and sociotropic voting. Am. Polit. Sci. Rev. 77, 92-111. doi: 10.2307/1956013
53. Lewontin, R. C. (2006). The analysis of variance and the analysis of causes. Int. J. Epidemiol. 35, 520-525. doi: 10.1093/ije/dyl062
54. Loftus, G. R. (1993). A picture is worth a thousand p values: on the irrelevance of hypothesis testing in the microcomputer age. Behav. Res. Methods Instrum. Comput. 25, 250-256. doi: 10.3758/BF03204506
55. Lubke, G. H., and Muthen, B. (2005). Investigating population heterogeneity with factor mixture models. Psychol. Methods 10, 21-39. doi: 10.1037/1082-989X.10.1.21
56. MacKay, D. G. (1982). The problems of flexibility, fluency, and speed-accuracy trade-offin skilled behavior. Psychol. Rev. 89, 483-506. doi: 10.1037/0033-295X.89.5.483
57. MacKinnon, D. P., Fairchild, A. J., and Fritz, M. S. (2007). Mediation analysis. Annu. Rev. Psychol. 58, 593.
58. MacKinnon, D. P., Krull, J. L., and Lockwood, C. M. (2000). Equivalence of the mediation, confounding and suppression effect. Prev. Sci. 1, 173-181. doi: 10.1023/A:1026595011371
59. Martin, E. (1981). Simpson's paradox resolved: a reply to Hintzman. Psychol. Rev. 88, 372-374. doi: 10.1037/0033-295X.88.4.372
60. Mazur, A., and Booth, A. (1998). Testosterone and dominance in men. Behav. Brain Sci. 21, 353-397. doi: 10.1017/S0140525X98001228
61. McCrae, R. R., and Costa, P. T. (2008). "Empirical and theoretical status of the five-factor model of personality traits," in The SAGE Handbook of Personality Theory and Assessment, Vol. 1: Personality Theories and Models, eds G. J. Boyle, G. Matthews, and D. H. Saklofske (London: SAGE publishers), 273-294. doi: 10.4135/9781849200462.n13
62. McCrae, R. R., and John, O. P. (1992). An introduction to the five-factor model and its applications. J. Pers. 60, 175-215. doi: 10.1111/j.1467-6494.1992.tb00970.x
63. Miller, G. (2012). The smartphone psychology manifesto. Perspect. Psychol. Sci. 7, 221-237.
64. Molenaar, P. C. M. (2004). A manifesto on psychology as idiographic science: bringing the person back into scientific psychology, this time forever. Measurement 2, 201-218.
65. Molenaar, P. C. M., and Campbell, C. G. (2009). The new person-specific paradigm in psychology. Curr. Dir. Psychol. Sci. 18, 112-117. doi: 10.1111/j.1467-8721.2009.01619.x
66. Molenaar, P. C. M., Huizenga, H. M., and Nesselroade, J. R. (2003). "The relationship between the structure of inter-individual and intra-individual variability: a theoretical and empirical vindication of developmental systems theory," in Understanding Human Development: Dialogues with Lifespan Psychology, eds U. M. Staudinger and U. Lindenberger (Boston: Kluwer Academic Publishers), 339-360).
67. Na, J., Grossmann, I., Varnum, M. E., Kitayama, S., Gonzalez, R., and Nisbett, R. E. (2010). Cultural differences are not always reducible to individual differences. Proc. Natl. Acad. Sci. U.S.A. 107, 6192-6197. doi: 10.1073/pnas.1001911107
68. Otte, R. (1985). Probabilistic causality and Simpson's Paradox. Philos. Sci. 52, 110-125. doi: 10.1086/289225
69. Pavlides, M. G., and Perlman, M. D. (2009). How likely is Simpson's paradox? Am. Stat. 63, 226-233.
70. Pearl, J. (1999). Simpson's Paradox: An Anatomy. UCLA Cognitive Systems Laboratory, Technical Report.
71. Pearl, J. (2000). Causality: Models, Reasoning and Inference. Cambridge: Cambridge University Press.
72. Pearl, J. (2013). Linear models: a useful "microscope" for causal analysis. J. Causal Inference 1, 155-170.
73. Pearson, K., Lee, A., and Bramley-Moore, L. (1899). Genetic reproductive selection: inheritance of fertility in man. Philos. Trans. R. Soc. A. 192, 257-330.
74. Penke, L., Johnson, W., Kievit, R. A., Wicherts, J. M., Ploeger, A., and Borsboom, D. (2011). Evolutionary psychology and intelligence research cannot be integrated the way Kanazawa (2010) suggests. Am. Psychol. 66, 916-917. doi: 10.1037/a0024626
75. Reid, J. A., and Sullivan, C. J. (2012). Unraveling victim-offender overlap: exploring profiles and constellations of risk. Vict. Offenders. 7, 327-360. doi: 10.1080/15564886.2012.685216
76. Robinson, W. S. (1950). Ecological correlations and the behavior of individuals. Am. Sociol. Rev. 15, 351-357. doi: 10.2307/2087176
77. Rosenbaum, P. R., and Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika 70, 41-55. doi: 10.1093/biomet/70.1.41
78. Rubin, D. B. (1997). Estimating causal effects from large data sets using propensity scores. Ann. Intern. Med. 127, 757-763. doi: 10.7326/0003-4819-127-8_Part_2-199710151-00064
79. Rubin, D. R. (2011). An alternative to pooling Kaplan-Meier curves in time-to-event meta-analysis. Int. J. Biostat. 7, 1-26. doi: 10.2202/1557-4679.1289
80. Rücker, G., and Schumacher, M. (2008). Simpson's paradox visualized: the example of the rosiglitazone meta-analysis. BMC Med. Res. Methodol. 8:34. doi: 10.1186/1471-2288-8-34
81. Schaller, M. (1992). Sample size, aggregation, and statistical reasoning in social inference. J. Exp. Soc. Psychol. 28, 65-85. doi: 10.1016/0022-1031(92)90032-F
82. Scheiner, S. M., Cox, S. B., Willig, M., Mittelbach, G. G., Osenberg, C., and Kaspari, M. (2000). Species richness, species-area curves and Simpson's paradox. Evol. Ecol. Res. 2, 791-802.
83. Schield, M. (1999). Simpson's paradox and Cornfield's conditions. ASA Proc. Sect. Stat. Educ. 106-111.
84. Schneiter, K., and Symanzik, J. (2013). J. Stat. Educ. 21, 1-20.
85. Schwarz, G. (1978). Estimating the dimension of a model. Ann. Stat. 6, 461-464. doi: 10.1214/aos/1176344136
86. Shaw, P., Greenstein, D., Lerch, J., Clasen, L., Lenroot, R., Gogtay, N., et al. (2006). Intellectual ability and cortical development in children and adolescents. Nature 440, 676-679. doi: 10.1038/nature04513
87. Shweder, R. (1973). The between and within of cross-cultural research. Ethos 1, 531-545. doi: 10.1525/eth.1973.1.4.02a00150
88. Simpson, E. H. (1951). The interpretation of interaction in contingency tables. J. R. Stat. Soc. Ser. B 13, 238-241.
89. Spellman, B. A. (1996a). Acting as intuitive scientists: contingency judgments are made while controlling for alternative potential causes. Psychol. Sci. 7, 337-334.
90. Spellman, B. A. (1996b). Conditioning causality. Psychol. Learn. Motiv. 34, 167-206.
91. Spellman, B. A., Price, C. M., and Logan, J. (2001). How two causes are different from one: the use of (un)conditional information in Simpson's paradox. Mem. Cogn. 29, 193-208. doi: 10.3758/BF03194913
92. Spirtes, P., Glymour, C., and Scheines, R. (2000). Causation, prediction, and search (Vol. 81). The MIT Press.
93. Suh, G. H. (2009). The use of atypical antipsychotics in dementia: rethinking Simpson's paradox. Int. Psychogeriatr. 21, 616-621. doi: 10.1017/S1041610209008485
94. Tamnes, C. K., Fjell, A. M., østby, Y., Westlye, L. T., Due-Tønnessen, P., Bjørnerud, A., et al (2011). The brain dynamics of intellectual development: waxing and waning white and gray matter. Neuropsychologia 49, 3605-3611. doi: 10.1016/j.neuropsychologia.2011.09.012
95. Team, R. D. C. (2013). R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing.
96. Tu, Y. K., Gunnell, D. J., and Gilthorpe, M. S. (2008). Simpson's paradox, Lord's paradox, and suppression effects are the same phenomenon-the reversal paradox. Emerg. Themes Epidemiol. 5, 2. doi: 10.1186/1742-7622-5-2
97. Tzambazis, K., and Stough, C. (2000). Alcohol impairs speed of information processing and simple and choice reaction time and differentially impairs higher-order cognitive abilities. Alcohol Alcohol. 35, 197-201. doi: 10.1093/alcalc/35.2.197
98. van der Linden, W. J., Jeon, M., and Ferrara, S. (2011). A paradox in the study of the benefits of test-item review. J. Educ. Meas. 48, 380-398. doi: 10.1111/j.1745-3984.2011.00151.x
99. Van Rijn, P. (2008). Categorical Time Series in Psychological Measurement. University of Amsterdam, Unpublished doctoral dissertation.
100. Vermunt, J. K., and Magidson, J. (2002). "Latent class cluster analysis," in Applied Latent Class Analysis, eds J. A. Hagenaars and A. L. McCutcheon (Cambridge, UK: Cambridge University Press), 89-106.
101. Wagner, C. H. (1982). Simpson's paradox in real life. Am. Stat. 36, 46-48.
102. Wainer, H. (1986). Minority contributions to the SAT score turnaround: an example of Simpson's paradox. J. Educ. Behav. Stat. 11, 239-244. doi: 10.3102/10769986011004239
103. Waldmann, M. R., and Hagmayer, Y. (1995). "When a cause simultaneously produces and prevents an effect," in Proceedings of the Seventeenth Annual Conference of the Cognitive Science Society, J. D. Moore and J. F. Lehman (Hillsdale, NJ: Erlbaum), 425-430.
104. Wardrop, R. L. (1995). Simpson's paradox and the hot hand in basketball. Am. Stat. 49, 24-28.
105. Wicherts, J. M., Bakker, M., and Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PLoS ONE 6:e26828. doi: 10.1371/journal.pone.0026828
106. Wicherts, J. M., Borsboom, D., Kats, J., and Molenaar, D. (2006). The poor availability of psychological research data for reanalysis. Am. Psychol. 61, 726-728. doi: 10.1037/0003-066X.61.7.726
107. Wickham, H. (2009). ggplot2: Elegant Graphics for Data Analysis. Springer Publishing Company, Incorporated.
108. Wilmoth, D. R. (2012). Intelligence and past use of recreational drugs. Intelligence 40, 15-22.
109. Yaari, G., and Eisenmann, S. (2011). The hot (invisible?) hand: can time sequence patterns of success/failure in sports be modeled as repeated random independent trials?. PLoS ONE 6:e24532. doi: 10.1371/journal.pone.0024532
110. Yule, G. U. (1903). Notes on the theory of association of attributes in statistics. Biometrika 2, 121-134. doi: 10.1093/biomet/2.2.121