A trifactor model for integrating ratings across multiple informants

Psychological Methods

Volumn 18, Issue 4, 2013, Pages 475-493

  Bauer, Daniel J ^a   Howard, Andrea L ^a   Baldasaro, Ruth E ^a   Curran, Patrick J ^a   Hussong, Andrea M ^a   Chassin, Laurie ^b   Zucker, Robert A ^c  

^a University of North Carolina at Chapel Hill   (United States)

^b ARIZONA STATE UNIVERSITY   (United States)

^c UNIVERSITY OF MICHIGAN   (United States)

Author keywords

Factor analysis; Informants; Observers; Raters; Sources

Indexed keywords

ADOLESCENT; ADULT; CHILD; FEMALE; HUMAN; MALE; PRESCHOOL CHILD; PROCEDURES; PSYCHOLOGICAL MODEL; PSYCHOMETRY; STATISTICAL ANALYSIS;

ADOLESCENT; ADULT; CHILD; CHILD, PRESCHOOL; DATA INTERPRETATION, STATISTICAL; FEMALE; HUMANS; MALE; MODELS, PSYCHOLOGICAL; PSYCHOMETRICS;

EID: 84892447072     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0032475     Document Type: Article

References (98)

1. Achenbach, T. M. (2011). Commentary: Definitely more than measurement error: But how should we understand and deal with informant discrepancies? Journal of Clinical Child and Adolescent Psychology, 40, 80-86. doi:10.1080/15374416.2011.533416
2. Achenbach, T. M., and Edelbrock, C. S. (1978). The classification of child psychopathology: A review and analysis of empirical efforts. Psychological Bulletin, 85, 1275-1301. doi:10.1037/0033-2909.85.6.1275
3. Achenbach, T. M., Krukowski, R. A., Dumenci, L., and Ivanova, M. Y. (2005). Assessment of adult psychopathology: Meta-analyses and implications of cross-informant correlations. Psychological Bulletin, 131, 361-382. doi:10.1037/0033-2909.131.3.361
4. Achenbach, T. M., McConaughy, S. H., and Howell, C. T. (1987). Child/ adolescent behavioral and emotional problems: Implications of crossinformant correlations for situational specificity. Psychological Bulletin, 101, 213-232. doi:10.1037/0033-2909.101.2.213
5. Alessandri, G., Vecchione, M., Tisak, J., and Barbaranelli, C. (2011). Investigating the nature of method factors through multiple informants: Evidence for a specific factor? Multivariate Behavioral Research, 46, 625-642. doi:10.1080/00273171.2011.589272
6. Anderson, T. W., and Rubin, H. (1956). Statistical inference in factor analysis. In J. Neyman (Ed.), Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability (pp. 111-150). Berkeley: University of California Press.
7. Asparouhov, T., and Muthén, B. (2006). Robust chi square difference testing with mean and variance adjusted test statistics (Mplus Web Notes No. 10). Retrieved from http://statmodel.com/download/webnotes/ webnote10.pdf
8. Asparouhov, T., and Muthén, B. (2010). Weighted least squares estimation with missing data. Retrieved from http://www.statmodel.com/download/ GstrucMissingRevision.pdf
9. Barbaranelli, C., Fida, R., Paciello, M., Di Giunta, L., and Caprara, G. V. (2008). Assessing personality in early adolescence through self-report and other-ratings in a multitrait-multimethod analysis of the BFQ-C. Personality and Individual Differences, 44, 876-886. doi:10.1016/j.paid.2007.10.014
10. Bartholomew, D. J., and Knott, M. (1999). Latent variable models and factor analysis (2nd ed.). London, England: Arnold.
11. Bauer, D. J., and Hussong, A. M. (2009). Psychometric approaches for developing commensurate measures across independent studies: Traditional and new models. Psychological Methods, 14, 101-125. doi:10.1037/a0015583
12. Biesanz, J. C., and West, S. G. (2004). Towards understanding assessments of the Big Five: Multitrait-multimethod analyses of convergent and discriminant validity across measurement occasion and type of observer. Journal of Personality, 72, 845-876. doi:10.1111/j.0022-3506.2004.00282.x
13. Bollen, K. A. (2002). Latent variables in psychology and the social sciences. Annual Review of Psychology, 53, 605-634. doi:10.1146/annurev.psych.53.100901.135239
14. Bollen, K. A., and Paxton, P. (1998). Detection and determinants of bias in subjective measures. American Sociological Review, 63, 465-478. doi: 10.2307/2657559
15. Boyle, M. H., and Pickles, A. R. (1997). Influence of maternal depressive symptoms on ratings of childhood behavior. Journal of Abnormal Child Psychology, 25, 399-412. doi:10.1023/A:1025737124888
16. Browne, M. W. (1984). Asymptotically distribution-free methods for the analysis of covariance structures. British Journal of Mathematical and Statistical Psychology, 37, 62- 83. doi:10.1111/j.2044-8317.1984.tb00789.x
17. Bullock, B. M., Deater-Deckard, K., and Leve, L. D. (2006). Deviant peer affiliation and problem behavior: A test of genetic and environmental influences. Journal of Abnormal Child Psychology, 34, 27-39. doi:10.1007/s10802-005-9004-9
18. Cai, L. (2010a). High-dimensional exploratory item factor analysis by a Metropolis-Hastings Robbins-Monro algorithm. Psychometrika, 75, 33-57. doi:10.1007/s11336-009-9136-x
19. Cai, L. (2010b). Metropolis-Hastings Robbins-Monro algorithm for confirmatory item factor analysis. Journal of Educational and Behavioral Statistics, 35, 307-335. doi:10.3102/1076998609353115
20. Cai, L. (2010c). A two-tier full-information item factor analysis model with applications. Psychometrika, 75, 581-612. doi:10.1007/s11336-010-9178-0
21. Cai, L., Yung, J. S., and Hansen, M. (2011). Generalized full-information item bifactor analysis. Psychological Methods, 16, 221-248. doi: 10.1037/a0023350
22. Campbell, D. T., and Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105. doi:10.1037/h0046016
23. Chassin, L., Rogosch, F., and Barrera, M. (1991). Substance use and symptomatology among adolescent children of alcoholics. Journal of Abnormal Psychology, 100, 449-463. doi:10.1037/0021-843X.100.4.449
24. Cheung, G. W., and Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233-255. doi:10.1207/S15328007SEM0902_5
25. Christoffersson, A. (1975). Factor analysis of dichotomized variables. Psychometrika, 40, 5-32. doi:10.1007/BF02291477
26. Conway, J. M., and Huffcutt, A. I. (1997). Psychometric properties of multisource performance ratings: A meta-analysis of subordinate, supervisor, peer, and self-ratings. Human Performance, 10, 331-360. doi:10.1207/s15327043hup1004_2
27. Conway, J. M., Lievens, F., Scullen, S. E., and Lance, C. E. (2004). Bias in the correlated uniqueness model for MTMM data. Structural Equation Modeling, 11, 535-559. doi:10.1207/s15328007sem1104_3
28. Curran, P. J., and Hussong, A. M. (2009). Integrative data analysis: The simultaneous analysis of multiple data sets. Psychological Methods, 14, 81-100. doi:10.1037/a0015914
29. Curran, P. J., Hussong, A. M., Cai, L., Huang, W., Chassin, L., Sher, K. J., and Zucker, R. A. (2008). Pooling data from multiple longitudinal studies: The role of item response theory in integrative data analysis. Developmental Psychology, 44, 365-380. doi:10.1037/0012-1649.44.2.365
30. De Los Reyes, A. (2011). Introduction to the special section: More than measurement error: Discovering meaning behind informant discrepancies in clinical assessments of children and adolescents. Journal of Clinical Child and Adolescent Psychology, 40, 1-9. doi:10.1080/15374416.2011.533405
31. Edwards, M. C. (2010). A Markov chain Monte Carlo approach to confirmatory item factor analysis. Psychometrika, 75, 474-497. doi:10.1007/s11336-010-9161-9
32. Eid, M. (2000). A multitrait-multimethod model with minimal assumptions. Psychometrika, 65, 241-261. doi:10.1007/BF02294377
33. Eid, M., Lischetzke, T., Nussbeck, F. W., and Trierweiler, L. I. (2003). Separating trait effects from trait-specific method effects in multitrait- multimethod models: A multiple-indicator CT-C(M _ 1) model. Psychological Methods, 8, 38-60. doi:10.1037/1082-989X.8.1.38
34. Eid, M., Nussbeck, F. W., Geiser, C., Cole, D. A., Gollwitzer, M., and Lischetzke, T. (2008). Structural equation modeling of multitrait- multimethod data: Different models for different types of methods. Psychological Methods, 13, 230-253. doi:10.1037/a0013219
35. Fan, X., and Sivo, S. A. (2009). Using goodness-of-fit indexes in assessing mean structure invariance. Structural Equation Modeling, 16, 54-69. doi:10.1080/10705510802561311
36. Fergusson, D. M., Lunskey, M. T., and Horwood, L. J. (1993). The effect of maternal depression on maternal ratings of child behavior. Journal of Abnormal Child Psychology, 21, 245-269. doi:10.1007/BF00917534
37. Flora, D. B., and Curran, P. J. (2004). An evaluation of alternative methods for confirmatory factor analysis with ordinal data. Psychological Methods, 9, 466-491. doi:10.1037/1082-989X.9.4.466
38. Geiser, C., and Lockhart, G. (2012). A comparison of four approaches to account for method effects in latent state-trait analyses. Psychological Methods, 17, 255-283. doi:10.1037/a0026977
39. Gibbons, R. D., Bock, R. D., Hedeker, D., Wiess, D. J., Segawa, E., Bhaumik, D. K.,. .. Stover, A. (2007). Full-information item bifactor analysis of graded response data. Applied Psychological Measurement, 31, 4-19. doi:10.1177/0146621606289485
40. Gibbons, R. D., and Hedeker, D. R. (1992). Full-information item bi-factor analysis. Psychometrika, 57, 423-436. doi:10.1007/BF02295430
41. Glas, C. A. W. (1998). Detection of differential item functioning using Lagrange multiplier tests. Statistica Sinica, 8, 647-667.
42. Gonzalez, R., and Griffin, D. W. (1999). The correlation analysis of dyadlevel data in the distinguishable case. Personal Relationships, 6, 449- 469. doi:10.1111/j.1475-6811.1999.tb00203.x
43. Hewitt, J. K., Silberg, J. L., Neale, M. C., Eaves, L. J., and Erickson, M. (1992). The analysis of parental ratings of children's behavior using LISREL. Behavior Genetics, 22, 293-317. doi:10.1007/BF01066663
44. Hoffman, B., Lance, C. E., Bynum, B., and Gentry, W. A. (2010). Rater source effects are alive and well after all. Personnel Psychology, 63, 119-151. doi:10.1111/j.1744-6570.2009.01164.x
45. Holzinger, K. J., and Swineford, F. (1937). The bi-factor method. Psychometrika, 2, 41-54. doi:10.1007/BF02287965
46. Horton, N. J., and Fitzmaurice, G. M. (2004). Regression analysis of multiple source and multiple informant data from complex survey samples. Statistics in Medicine, 23, 2911-2933. doi:10.1002/sim.1879
47. Hussong, A. M., Jones, D. J., Stein, G. L., Baucom, D. H., and Boeding, S. (2011). An internalizing pathway to alcohol use and disorder. Psychology of Addictive Behaviors, 25, 390-404. doi:10.1037/a0024519
48. Kenny, D. A. (1979). Correlation and causality. New York, NY: Wiley.
49. Kenny, D. A. (1991). A general model of consensus and accuracy in interpersonal perception. Psychological Review, 98, 155-163. doi: 10.1037/0033-295X.98.2.155
50. Kenny, D. A., and Kashy, D. A. (1992). Analysis of the multitrait- multimethod matrix by confirmatory factor analysis. Psychological Bulletin, 112, 165-172. doi:10.1037/0033-2909.112.1.165
51. Kenny, D. A., Kashy, D. A., and Cook, W. L. (2006). Dyadic data analysis. New York, NY: Guilford Press.
52. Kraemer, H. C., Measelle, J. R., Ablow, J. C., Essex, M. J., Boyce, W. T., and Kupfer, D. J. (2003). A new approach to integrating data from multiple informants in psychiatric assessment and research: Mixing and matching contexts and perspectives. American Journal of Psychiatry, 160, 1566-1577. doi:10.1176/appi.ajp.160.9.1566
53. Lance, C. E., Hoffman, B. J., Gentry, W. A., and Baranik, L. E. (2008). Rater source factors represent important subcomponents of the criterion construct space, not rater bias. Human Resource Management Review, 18, 223-232. doi:10.1016/j.hrmr.2008.03.002
54. Lance, C. E., Noble, C. L., and Scullen, S. E. (2002). A critique of the correlated trait-correlated method and correlated uniqueness models for multitrait-multimethod data. Psychological Methods, 7, 228-244. doi: 10.1037/1082-989X.7.2.228
55. Loken, E. (2005). Identification constraints and inference in factor models. Structural Equation Modeling, 12, 232-244. doi:10.1207/ s15328007sem1202_3
56. MacCallum, R. C., Roznowski, M., and Necowitz, L. B. (1992). Model modifications in covariance structure analysis: The problem of capitalization on chance. Psychological Bulletin, 111, 490-504. doi:10.1037/0033-2909.111.3.490
57. Marsh, H. W. (1989). Confirmatory factor analyses of multitrait- multimethod data: Many problems and a few solutions. Applied Psychological Measurement, 13, 335-361. doi:10.1177/014662168901300402
58. Marsh, H. W. (1993). Multitrait-multimethod analyses: Inferring each trait-method combination with multiple indicators. Applied Measurement in Education, 6, 49-81. doi:10.1207/s15324818ame0601_4
59. Marsh, H. W., and Bailey, M. (1991). Confirmatory factor analyses of multitrait-multimethod data: A comparison of the behavior of alternative models. Applied Psychological Measurement, 15, 47-70. doi:10.1177/014662169101500106
60. Marsh, H. W., and Grayson, D. (1995). Latent variable models of multitrait- multimethod data. In R. H. Hoyle (Ed.), Structural equation modeling: Concepts, issues, and applications (pp. 177-198). Thousand Oaks, CA: Sage.
61. Marsh, H. W., and Hocevar, D. (1988). A new, more powerful approach to multitrait-multimethod analyses: Application of second-order confirmatory factor analysis. Journal of Applied Psychology, 73, 107-117. doi:10.1037/0021-9010.73.1.107
62. Mount, M. K., Judge, T. A., Scullen, S. E., Sytsma, M. R., and Hezlett, S. A. (1998). Trait, rater and level effects in 360-degree performance ratings. Personnel Psychology, 51, 557-576. doi:10.1111/j.1744-6570.1998.tb00251.x
63. Muthén, B. O. (1978). Contributions to factor analysis of dichotomous variables. Psychometrika, 43, 551-560. doi:10.1007/BF02293813
64. Muthén, B. O., du Toit, S. H. C., and Spisic, D. (1997). Robust inference using weighted least squares and quadratic estimating equations in latent variable modeling with categorical and continuous outcomes. Unpublished manuscript.
65. Muthén, L. K., and Muthén, B. O. (2010). Mplus user's guide (6th ed.). Los Angeles, CA: Muthén and Muthén.
66. Najman, J. M., Williams, G. M., Nikels, J., Spence, S., Bor, W., O'Callaghan, M.,. .. Andersen, M. J. (2000). Mothers' mental illness and child behavior problems: Cause-effect association or observation bias? Journal of the American Academy of Child and Adolescent Psychiatry, 39, 592-602. doi:10.1097/00004583-200005000-00013
67. Neale, M. C., and Stevenson, J. (1989). Rater bias in the EASI temperament scales: A twin study. Journal of Personality and Social Psychology, 56, 446-455. doi:10.1037/0022-3514.56.3.446
68. Nussbeck, F. W., Eid, M., Geiser, C., Courvoisier, D. S., and Lischetzke, T. (2009). A CTC(M _ 1) model for different types of raters. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 5, 88-98. doi:10.1027/1614-2241.5.3.88
69. Nussbeck, F. W., Eid, M., and Lischetzke, T. (2006). Analysing multitrait- multimethod data with structural equation models for ordinal variables applying the WLSMV estimator: What sample size is needed for valid results? British Journal of Mathematical and Statistical Psychology, 59, 195-213. doi:10.1348/000711005X67490
70. Olsson, U. (1979). Maximum likelihood estimation of the polychoric correlation coefficient. Psychometrika, 44, 443- 460. doi:10.1007/BF02296207
71. Pohl, S., and Steyer, R. (2010). Modeling common traits and method effects in multitrait-multimethod analysis. Multivariate Behavioral Research, 45, 45-72. doi:10.1080/00273170903504729
72. Pohl, S., Steyer, R., and Kraus, K. (2008). Modeling method effects as individual causal effects. Journal of the Royal Statistical Society: Series A. Statistics in Society, 171, 41-63. doi:10.1111/j.1467-985X.2007.00517.x
73. Reininghaus, U., McCabe, R., Burns, T., Croudace, T., and Priebe, S. (2011). Measuring patients' views: A bifactor model of distinct patient-reported outcomes in psychosis. Psychological Medicine, 41, 277-289. doi:10.1017/S0033291710000784
74. Reise, S. P., Ventura, J., Keefe, R. S. E., Baade, L. E., Gold, J. M., Green, M. F.,. .. Bilder, R. (2011). Bifactor and item response theory analyses of interviewer report scales of cognitive impairment in schizophrenia. Psychological Assessment, 23, 245-261. doi:10.1037/a0021501
75. Renk, K. (2005). Cross-informant ratings of the behavior of children and adolescents: The "gold standard". Journal of Child and Family Studies, 14, 457-468. doi:10.1007/s10826-005-7182-2
76. Rowe, D. C., and Kandel, D. (1997). In the eye of the beholder? Parental ratings of externalizing and internalizing symptoms. Journal of Abnormal Child Psychology, 25, 265-275. doi:10.1023/A:1025756201689
77. Satorra, A. (2000). Scaled and adjusted restricted tests in multi-sample analysis of moment structures. In R. D. H. Heijmans, D. S. G. Pollock, and A. Satorra (Eds.), Innovations in multivariate statistical analysis: A Festschrift for Heinz Neudecker (pp. 233-247). Dordrecht, the Netherlands: Kluwer Academic.
78. Satorra, A., and Bentler, P. M. (1999). A scaled difference chi-square test statistic for moment structure analysis. Retrieved from University of California, Los Angeles, Department of Statistics website: http:// statistics.ucla.edu/preprints/uclastat-preprint-1999:19
79. Schafer, J. L., and Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7, 147-177. doi:10.1037/1082-989X .7.2.147
80. Scullen, S. E., Mount, M. K., and Goff, M. (2000). Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 85, 956-970. doi:10.1037/0021-9010.85.6.956
81. Seiffge-Krenke, I., and Kollmar, F. (1998). Discrepancies between mothers' and fathers' perceptions of sons' and daughters' problem behaviour: A longitudinal analysis of parent-adolescent agreement on internalising and externalising problem behaviour. Journal of Child Psychology and Psychiatry, 39, 687-697.
82. Simonoff, E., Pickles, A., Hewitt, J., Silberg, J., Rutter, M., Loeber, R.,. .. Eaves, L. (1995). Multiple raters of disruptive child behavior: Using a genetic strategy to examine shared views and bias. Behavior Genetics, 25, 311-326. doi:10.1007/BF02197280
83. Skrondal, A., and Rabe-Hesketh, S. (2004). Generalized latent variable modeling: Multilevel, longitudinal and structural equation models. Boca Raton, FL: Chapman and Hall/CRC. doi:10.1201/9780203489437
84. Spearman, C. (1904). General intelligence, objectively determined and measured. American Journal of Psychology, 15, 201-293. doi:10.2307/1412107
85. Steyer, R. (1989). Models of classical psychometric test theory as stochastic measurement models: Representation, uniqueness, meaningfulness, identifiability, and testability. Methodika, 3, 25-60.
86. Steyer, R., Ferring, D., and Schmitt, M. J. (1992). States and traits in psychological assessment. European Journal of Psychological Assessment, 8, 79-98.
87. Thissen, D., and Orlando, M. (2001). Item response theory for items scored in two categories. In D. Thissen and H. Wainer (Eds.), Test scoring (pp. 73-140). Mahwah, NJ: Erlbaum.Thomson, G. H. (1936). Some points of mathematical technique in the factorial analysis of ability. Journal of Educational Psychology, 27, 37-54. doi:10.1037/h0062007
88. Thomson, G. H. (1951). The factorial analysis of human ability (5th ed.). Boston, MA: Houghton Mifflin.
89. Thurstone, L. L. (1935). The vectors of the mind: Multiple-factor analysis for the isolation of primary traits. Chicago, IL: University of Chicago Press. doi:10.1037/10018-000
90. van der Valk, J. C., van den Oord, E. J. C. G., Verhulst, F. C., and Boomsma, D. I. (2001). Using parental ratings to study the etiology of 3-year-old twins' problem behaviors: Different views or rater bias? Journal of Child Psychology and Psychiatry, 42, 921-931. doi:10.1111/1469-7610.00788
91. van Dulmen, M. H. M., and Egeland, B. (2011). Analyzing multiple informant data on child and adolescent behavior problems: Predictive validity and comparison of aggregation procedures. International Journal of Behavioral Development, 35, 84-92. doi:10.1177/0165025410392112
92. Widaman, K. F. (1985). Hierarchically nested covariance structure models for multitrait-multimethod data. Applied Psychological Measurement, 9, 1-26. doi:10.1177/014662168500900101
93. Wirth, R. J., and Edwards, M. C. (2007). Item factor analysis: Current approaches and future directions. Psychological Methods, 12, 58-79. doi:10.1037/1082-989X.12.1.58
94. Woehr, D. J., Sheehan, M. K., and Bennett, W., Jr. (2005). Assessing measurement equivalence across rating sources: A multitrait-multirater approach. Journal of Applied Psychology, 90, 592-600. doi:10.1037/ 0021-9010.90.3.592
95. Yoon, M., and Millsap, R. E. (2007). Detecting violations of factorial invariance using data-based specification searches: A Monte Carlo study. Structural Equation Modeling, 14, 435- 463. doi:10.1080/ 10705510701301677
96. Youngstrom, E. A., Izard, C., and Ackerman, B. (1999). Dysphoria-related bias in maternal ratings of children. Journal of Consulting and Clinical Psychology, 67, 905-916. doi:10.1037/0022-006X.67.6.905
97. Zucker, R. A., Ellis, D. A., Fitzgerald, H. E., Bingham, C. R., and Sanford, K. (1996). Other evidence for at least two alcoholisms II: Life course variation in antisociality and heterogeneity of alcoholic outcome. Development and Psychopathology, 8, 831- 848. doi:10.1017/ S0954579400007458
98. Zucker, R. A., Fitzgerald, H. E., Refior, S. K., Puttler, L. I., Pallas, D. M., and Ellis, D. A. (2000). The clinical and social ecology of childhood for children of alcoholics: Description of a study and implications for a differentiated social policy. In H. E. Fitzgerald, B. M. Lester, and B. S. Zuckerman (Eds.), Children of addiction: Research, health, and policy issues (pp. 109-141). New York, NY: Routledge Falmer.