A psychometric analysis of the reading the mind in the eyes test: Toward a brief form for research and applied settings

Frontiers in Psychology

Volumn 6, Issue OCT, 2015, Pages

  Olderbak, Sally ^a   Wilhelm, Oliver ^a   Olaru, Gabriel ^a   Geiger, Mattis ^a   Brenneman, Meghan W ^b   Roberts, Richard D ^c  

^a UNIVERSITY OF ULM   (Germany)

^b Educational Testing Service   (United States)

^c Professional Examination Service   (United States)

Author keywords

Cognitive empathy; Emotion perception; Psychometric analysis; Reading the mind in the eyes test; Short form; Theory of mind; Vocabulary

Indexed keywords

EID: 84949643164     PISSN: None     EISSN: 16641078     Source Type: Journal    
DOI: 10.3389/fpsyg.2015.01503     Document Type: Article

References (85)

1. Adams, Jr. R. B., Rule, N.O., Franklin, Jr. R. G., Wang, E., Stevenson, M.T., Yoshikawa, S., et al. (2009). Cross-cultural reading the mind in the eyes: an fMRI investigation. J. Cogn. Neurosci. 22, 97-108. doi: 10.1162/jocn.2009.21187.
2. Ahmed, F.S., and Stephen Miller, L. (2011). Executive function mechanisms of Theory of Mind. J. Autism Dev. Disord. 41, 667-678. doi: 10.1007/s10803-010-1087-7.
3. Astington, J.W. (2003). "Sometimes necessary, never sufficient: false-belief understanding and social competence," in Individual Differences in Theory of Mind: Implications for Typical And Atypical Development. Macquarie Monographs in Cognitive Science, eds B. Repacholi and V. Slaughter (New York, NY: Psychology Press), 13-38.
4. Astington, J.W., and Jenkins, J.M. (1999). A longitudinal study of the relation between language and theory-of-mind development. Dev. Psychol. 35, 1311. doi: 10.1037/0012-1649.35.5.1311.
5. Baron-Cohen, S., Jolliffe, T., Mortimore, C., and Robertson, M. (1997). Another advanced test of theory of mind: evidence from very high functioning adults with autism or Asperger syndrome. J. Child Psychol. Psychiatry 38, 813-822. doi: 10.1111/j.1469-7610.1997.tb01599.x.
6. Baron-Cohen, S., Knickmeyer, R.C., and Belmonte, M.K. (2005). Sex differences in the brain: implications for explaining austism. Science 310, 819-823. doi: 10.1126/science.1115455.
7. Baron-Cohen, S., Leslie, A.M., and Frith, U. (1985). Does the autistic child have a "Theory of Mind"? Cognition 21, 37-46. doi: 10.1016/0010-0277(85)90022-8.
8. Baron-Cohen, S., Wheelwright, S., Hill, J., Raste, Y., and Plumb, I. (2001). The "Reading the Mind in the Eyes" test revised version: a study with normal adults, and adults with Asperger syndrome or high-functioning autism. J. Child Psychol. Psychiatry 42, 241-251. doi: 10.1111/1469-7610.00715.
9. Batki, A., Baron-Cohen, S., Wheelwright, S., Connellan, J., and Ahluwalia, J. (2000). Is there an innate gaze module? Evidence from human neonates. Infant Behav. Dev. 23, 223-229. doi: 10.1016/S0163-6383(01)00037-6.
10. Beauducel, A., and Herzberg, P.Y. (2006). On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA. Struct. Equ. Modeling 13, 186-203. doi: 10.1207/s15328007sem1302_2.
11. Bentler, P.M. (1990). Comparative fit indices in structural models. Psychol. Bull. 107, 238-246. doi: 10.1037/0033-2909.107.2.238.
12. Berinsky, A.J., Huber, G.A., and Lenz, G.S. (2012). Evaluating online labor markets for experimental research: Amazon. com's Mechanical Turk. Polit. Anal. 20, 351-368. doi: 10.1093/pan/mpr057.
13. Bora, E., Yucel, M., and Pantelis, C. (2009). Theory of mind impairment in schizophrenia: meta-analysis. Schizophr. Res. 109, 1-9. doi: 10.1016/j.schres.2008.12.020.
14. Bosseler, A., and Massaro, D.W. (2003). Development and evaluation of a computer-animated tutor for vocabulary and language learning in children with autism. J. Autism Dev. Disord. 33, 653-672. doi: 10.1023/B:JADD. 0000006002.82367.4f.
15. Buhrmester, M., Kwang, T., and Gosling, S.D. (2011). Amazon's Mechanical Turk: a new source of inexpensive, yet high-quality data? Perspect. Psychol. Sci. 6, 3-5. doi: 10.1177/1745691610393980.
16. Calder, A.J., Lawrence, A.D., Keane, J., Scott, S.K., Owen, A.M., Christoffels, I., et al. (2002). Reading the mind from eye gaze. Neuropsychologia 40, 1129-1138. doi: 10.1016/S0028-3932(02)00008-8.
17. Casler, K., Bickel, L., and Hackett, E. (2013). Separate but equal? A comparison of participants and data gathered via Amazon's MTurk, social media, and face-to-face behavioral testing. Comput. Hum. Behav. 29, 2156-2160. doi: 10.1016/j.chb.2013.05.009.
18. Clark, L.A., and Watson, D. (1995). Constructing validity: basic issues in objective scale development. Psychol. Assess. 7, 309-319. doi: 10.1037/1040-3590.7.3.309.
19. Cortina, J.M. (1993). What is coefficient alpha? An examination of theory and applications. J. Appl. Psychol. 78, 98-104. doi: 10.1037/0021-9010.78.1.98.
20. Cronbach, L.J. (1951). Coefficient Alpha and the internal structure of tests. Psychometrika 16, 297-334. doi: 10.1007/BF02310555.
21. Davies, M. (2008). The Corpus of Contemporary American English: 450 Million Words, 1990-present [Data file and code book]. Available online at: http://corpus.byu.edu/coca/
22. Davis, M.H. (1980). A multidimensional approach to individual differences in empathy. JSAS Catalog Sel. Doc. Psychol. 10, 85-104.
23. Dehning, S., Girma, E., Gasperi, S., Meyer, S., Tesfaye, M., and Siebeck, M. (2012). Comparative cross-sectional study of empathy among first year and final year medical students in Jimma University, Ethiopia: steady state of the heart and opening of the eyes. BMC Med. Educ. 12:34. doi: 10.1186/1472-6920-12-34.
24. Duval, C., Piolino, P., Bejanin, A., Eustache, F., and Desgranges, B. (2011). Age effects on different components of theory of mind. Conscious. Cogn. 20, 627-642. doi: 10.1016/j.concog.2010.10.025.
25. Ekstrom, R.B., French, J.W., Harman, H.H., and Dermen, D. (1976). Manual for Kit of Factor-Referenced Cognitive Tests. Princeton, NJ: Educational Testing Service.
26. Emery, N.J. (2000). The eyes have it: the neuroethology, function and evolution of social gaze. Neurosci. Biobehav. Rev. 24, 581-604. doi: 10.1016/S0149-7634(00)00025-7.
27. Ferguson, F.J., and Austin, E.J. (2010). Associations of trait and ability emotional intelligence with performance on theory of mind tasks in an adult sample. Pers. Individ. Dif. 49, 414-418. doi: 10.1016/j.paid.2010.04.009.
28. Fernández-Abascal, E.G., Cabello, R., Fernández-Berrocal, P., and Baron-Cohen, S. (2013). Test-retest reliability of the "Reading the Mind in the Eyes" test: a one-year follow-up study. Mol. Autism 4, 1-6. doi: 10.1186/2040-2392-4-33.
29. Flora, D.B., and Curran, P.J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychol. Methods 9, 466-491. doi: 10.1037/1082-989X.9.4.466.
30. Girli, A. (2014). Psychometric properties of the Turkish child and adult form of "Reading the Mind in the Eyes Test". Psychology 5, 1321-1337. doi: 10.4236/psych.2014.511143.
31. Golan, O., Baron-Cohen, S., Hill, J.J., and Golan, Y. (2006). The "reading the mind in films" task: complex emotion recognition in adults with and without autism spectrum conditions. Soc. Neurosci. 1, 111-123. doi: 10.1080/17470910600980986.
32. Grove, R., Baillie, A., Allison, C., Baron-Cohen, S., and Hoekstra, R.A. (2014). The latent structure of cognitive and emotional empathy in individuals with autism, first-degree relatives and typical individuals. Mol. Autism 5, 1-10. doi: 10.1186/2040-2392-5-42.
33. Guastella, A.J., Einfeld, S.L., Gray, K.M., Rinehart, N.J., Tonge, B.J., Lambert, T.J., et al. (2010). Intranasal oxytocin improves emotion recognition for youth with Autism spectrum disorders. Biol. Psychiatry 67, 692-694. doi: 10.1016/j.biopsych.2009.09.020.
34. Hallerbäck, M.U., Lugnegård, T., Hjärthag, F., and Gillberg, C. (2009). The reading the mind in the eyes test: test-retest reliability of a Swedish version. Cogn. Neuropsychiatry 14, 127-143. doi: 10.1080/13546800902901518.
35. Harkness, K., Sabbagh, M., Jacobson, J., Chowdrey, N., and Chen, T. (2005). Enhanced accuracy of mental state decoding in dysphoric college students. Cogn. Emot. 19, 999-1025. doi: 10.1080/02699930541000110.
36. Harkness, K.L., Jacobson, J.A., Duong, D., and Sabbagh, M.A. (2010). Mental state decoding in past major depression: effect of sad versus happy mood induction. Cogn. Emot. 24, 497-513, doi: 10.1080/02699930902750249.
37. Harms, M.B., Martin, A., and Wallace, G.L. (2010). Facial emotion recognition in autism spectrum disorders: a review of behavioral and neuroimaging studies. Neuropsychol. Rev. 20, 290-322. doi: 10.1007/s11065-010-9138-6.
38. Heene, M., Hilbert, S., Draxler, C., Ziegler, M., and Bühner, M. (2011). Masking misfit in confirmatory factor analysis by increasing unique variances: a cautionary note on the usefullness of cutoff values of fit indices. Psychol. Methods 16, 319-336. doi: 10.1037/a0024917.
39. Henry, J.D., Phillips, L.H., Beatty, W.W., McDonald, S., Longley, W.A., Joscelyne, A., et al. (2009). Evidence for deficits in facial affect recognition and theory of mind in multiple sclerosis. J. Int. Neuropsychol. Soc. 15, 277-285. doi: 10.1017/S1355617709090195.
40. Hu, L., and Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct. Equ. Modeling 6, 1-55. doi: 10.1080/10705519909540118.
41. Jenkins, J.M., and Astington, J.W. (1996). Cognitive factors and family structure associated with theory of mind development in young children. Dev. Psychol. 32, 70. doi: 10.1037/0012-1649.32.1.70.
42. Khorashad, B.S., Baron-Cohen, S., Roshan, G.M., Kazemian, M., Khazai, L., Aghili, Z., et al. (2015). The "Reading the Mind in the Eyes" test: investigation of psychometric properties and test-retest reliability of the persian version. J. Autism Dev. Disord. 45, 2651-2666. doi: 10.1007/s10803-015-2427-4.
43. Kirkland, R.A., Baker, C.A., Johnson, C., Peterson, E., and Putos, S. (2012a). "Meta-analysis reveals a moderate relationship between tests of theory of mind and the Eyes Test," in Poster Presented at the Association of Psychological Science (Chicago).
44. Kirkland, R.A., Baker, C.A., Miller, S., Pulos, S., and Peterson, E. (2012b). "Meta-analysis reveals female superiority in "Reading the Mind in the Eyes Test"," in Poster Presented at the Association of Psychological Science (Chicago).
45. Kline, R.B. (2005). Principles and Practice of Structural Equation Modeling, 2nd Edn. New York, NY: The GuilfordPress.
46. Konrath, S., Corneille, O., Bushman, B.J., and Luminet, O. (2014). The relationship between narcissistic exploitativeness, dispositional empathy, and emotion recognition abilities. J. Nonverbal Behav. 38, 129-143. doi: 10.1007/s10919-013-0164-y.
47. Kuder, G.F., and Richardson, M.W. (1937). The theory of the estimation of test reliability. Psychometrika 2, 151-116. doi: 10.1007/BF02288391.
48. Kunihira, Y., Senju, A., Dairoku, H., Wakabayashi, A., and Hasegawa, T. (2006). 'Autistic' traits in non-autistic Japanese populations: relationships with personality traits and cognitive ability. J. Autism Dev. Disord. 36, 553-566. doi: 10.1007/s10803-006-0094-1.
49. Lawrence, E.J., Shaw, P., Baker, D., Baron-Cohen, S., and David, A.S. (2004). Measuring empathy: reliability and validity of the Empathy Quotient. Psychol. Med. 34, 911-920. doi: 10.1017/S0033291703001624.
50. Leite, W.L., Huang, I.C., and Marcoulides, G.A. (2008). Item selection for the development of short forms of scales using an ant colony optimization algorithm. Multivariate Behav. Res. 43, 411-431. doi: 10.1080/00273170802285743.
51. Little, T.D. (1997). Mean and covariance structures (MACS) analyses of cross-cultural data: practical and theoretical issues. Multivariate Behav. Res. 32, 53-76. doi: 10.1207/s15327906mbr3201_3.
52. Little, T.D., Cunningham, W.A., Shahar, G., and Widaman, K.F. (2002). To parcel or not to parcel: exploring the question, weighing the merits. Struct. Equ. Modeling 9, 151-173. doi: 10.1207/S15328007SEM0902_1.
53. Marcoulides, G.A., and Drezner, Z. (2003). Model specification searchers using ant colony optimization algorithms. Struct. Equ. Modeling 10, 154-164. doi: 10.1207/S15328007SEM1001_8.
54. Maurage, P., Grynberg, D., Noël, X., Joassin, F., Hanak, C., Verbanck, P., et al. (2011). The "Reading the Mind in the Eyes" test as a new way to explore complex emotions decoding in alcohol dependence. Psychiatry Res. 190, 375-378. doi: 10.1016/j.psychres.2011.06.015.
55. McDonald, R.P. (1999). Test Theory: A Unified Treatment. Mahwah, NJ: Lawrence Erlbaum Associates
56. Miu, A.C., Pană, S.E., and Avram, J. (2012). Emotional face processing in neurotypicals with autistic traits: implications for the broad autism phenotype. Psychiatry Res. 198, 489-494. doi: 10.1016/j.psychres.2012.01.024.
57. Muthén, L.K. (2014). Re: Model Fit Index WRMR [Online forum comment]. Available online at: http://www.statmodel.com/discussion/messages/9/5096.html?1429189065
58. Muthén, L.K., and Muthén, B.O. (2011). Mplus User's Guide, 6th Edn. Los Angeles, CA: Muthén & Muthén.
59. Nowicki, Jr. S., and Duke, M.P. (1994). Individual differences in the nonverbal communication of affect: the diagostic analysis of nonverbal accuracy scale. J. Nonverbal Behav. 18, 9-35. doi: 10.1007/BF02169077.
60. Nunnally, I.H., and Bernstein, N.J. (1994). Psychometric Theory. New York, NY: McGraw-Hill.
61. Olaru, G., Witthöft, M., and Wilhelm, O. (in press). Methods matter: testing competing models for designing short-scale big-five assessments. J. Res. Pers. doi: 10.1016/j.jrp.2015.09.001.
62. Olderbak, S., Sassenrath, C., Keller, J., and Wilhelm, O. (2014). An emotion-differentiated perspective on empathy with the emotion specific empathy questionnaire. Front. Psychol. 5:653. doi: 10.3389/fpsyg.2014.00653.
63. Peterson, E., and Miller, S.F. (2012). The eyes test as a measure of individual differences: how much of the variance reflects verbal IQ? Front. Psychol. 3:220. doi: 10.3389/fpsyg.2012.00220.
64. Petroni, A., Canales-Johnson, A., Urquina, H., Guex, R., Hurtado, E., Blenkmann, A., et al. (2011). The cortical processing of facial emotional expression is associated with social cognition skills and executive functioning: a preliminary study. Neurosci. Lett. 505, 41-46. doi: 10.1016/j.neulet.2011.09.062.
65. Piedmont, R.L., and Hyland, M.E. (1993). Inter-item correlation frequency distribution analysis: a method for evaluating scale dimensionality. Educ. Psychol. Meas. 53, 369-378. doi: 10.1177/0013164493053002006.
66. Posner, M.I. (1980). Orienting of attention. Q. J. Exp. Psychol. 32, 3-25. doi: 10.1080/00335558008248231.
67. Premack, D., and Woodruff, G. (1978). Does the chimpanzee have a theory of mind? Behav. Brain Sci. 1, 515-526. doi: 10.1017/S0140525X00076512.
68. Prevost, M., Carrier, M.E., Chowne, G., Zelkowitz, P., Joseph, L., and Gold, I. (2014). The Reading the Mind in the Eyes test: validation of a French version and exploration of cultural variations in a multi-ethnic city. Cogn. Neuropsychiatry 19, 189-204. doi: 10.1080/13546805.2013.823859.
69. Ragsdale, G., and Foley, R.A. (2011). A maternal influence on Reading the Mind in the Eyes mediated by executive function: differential parental influences on full and half-siblings. PLoS ONE 6:e23236. doi: 10.1371/journal.pone.0023236.
70. Rigdon, E.E. (1996). CFI versus RMSEA: a comparison of two fit indexes for structural equation modeling. Struct. Equ. Modeling 3, 369-379. doi: 10.1080/10705519609540052.
71. Savalei, V. (2012). The relationship between root mean square error of approximation and model misspecification in confirmatory factor analysis models. Educ. Psychol. Meas. 72, 910-932. doi: 10.1177/0013164412452564.
72. Schmidt, J.Z., and Zachariae, R. (2009). PTSD and impaired eye expression recognition: a preliminary study. J. Loss Trauma 14, 46-56. doi: 10.1080/15325020802537096.
73. Slaughter, B., and Repacholi, B. (2003). "Individual differences in theory of mind: what are we investigating?" in Individual Differences in Theory of Mind: Implications for Typical and Atypical Development. Macquarie Monographs in Cognitive Science, eds B. Repacholi and V. Slaughter (New York, NY: Psychology Press), 1-12.
74. Söderstrand, P., and Almkvist, O. (2012). Psychometric data on the Eyes Test, the Faux Pas Test, and the Dewey Social Stories Test in a population-based Swedish adult sample. Nordic Psychol. 64, 30-43. doi: 10.1080/19012276.2012.693729.
75. Spreng, R.N., McKinnon, M.C., Mar, R.A., and Levine, B. (2009). The Toronto Empathy Questionnaire: scale development and initial validation of a factor-analytic solution to multiple empathy measures. J. Pers. Assess. 91, 62-71. doi: 10.1080/00223890802484381.
76. Steiger, J.H., and Lind, J.C. (1980). "Statistically based tests for the number of common factors," in Paper Presented at the Annual Meeting of the Psychometric Society (Iowa City, IA).
77. Torralva, T., Roca, M., Gleichgerrcht, E., Bekinschtein, T., and Manes, F. (2009). A neuropsychological battery to detect specific executive and social cognitive impairments in early frontotemporal dementia. Brain 132, 1299-1309. doi: 10.1093/brain/awp041.
78. Tucker, L.R., and Lewis, C. (1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika 38, 1-10. doi: 10.1007/BF02291170.
79. Vellante, M., Baron-Cohen, S., Melis, M., Marrone, M., Petretto, D.R., Masala, C., et al. (2012). The "Reading the Mind in the Eyes" test: systematic review of psychometric properties and a validation study in Italy. Cogn. Neuropsychiatry 18, 1-12. doi: 10.1080/13546805.2012.721728.
80. Vogindroukas, I., Chelas, E.N., and Petridis, N.E. (2014). Reading the Mind in the Eyes Test (children's version): a comparison study between children with typical development, children with high-functioning autism and typically developed adults. Folia Phoniatrica Logopaedica 66, 18-24. doi: 10.1159/000363697.
81. Voracek, M., and Dressler, S.G. (2006). Lack of correlation between digit ratio (2D:4D) and Baron-Cohen's "Reading the Mind in the Eyes" test, empathy, systemising, and autism-spectrum quotients in a general population sample. Pers. Individ. Dif. 41, 1481-1491. doi: 10.1016/j.paid.2006.06.009.
82. Wellman, H.M., Cross, D., and Watson, J. (2001). Meta-analysis of theory-of-mind development: the truth about false belief. Child Dev. 72, 655-684. doi: 10.1111/1467-8624.00304.
83. Yildirim, E.A., Kaşar, M., Güdük, M., Ateş, E., Küçüparlak, I., and Ozalmete, E.O. (2011). Investigation of the reliability of the "Reading the Mind in the Eyes Test" in a Turkish population. Turk. J. Psychiatry 22, 1-8. doi: 10.5080/u6500.
84. Yu, C.Y. (2002). Evaluating Cutoff Criteria of Model Fit Indices for Latent Variable Models with Binary and Continuous Outcomes. Doctoral dissertation, University of California Los Angeles.
85. h. Appl. Psychol. Meas. 30, 121-144. doi: 10.1177/0146621605278814.