Motivation and engagement in science around the globe: Testing measurement invariance with multigroup structural equation models across 57 countries using PISA 2006

Handbook of International Large-Scale Assessment: Background, Technical Issues, and Methods of Data Analysis

Volumn , Issue , 2013, Pages 317-344

  Nagengast, Benjamin ^a   Marsh, Herbert W ^b,c,d  

^a UNIVERSITY OF TÜBINGEN   (Germany)

^b UNIVERSITY OF WESTERN SYDNEY   (Australia)

^c UNIVERSITY OF OXFORD   (United Kingdom)

^d KING SAUD UNIVERSITY   (Saudi Arabia)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85128126086     PISSN: None     EISSN: None     Source Type: Book    
DOI: None     Document Type: Chapter

References (76)

1. Artelt, C. 2005. Cross-cultural approaches to measuring motivation. Educational Assessment, 10, 231-255.
2. Artelt, C., Baumert, J., Julius-McElvany, N., and Peschar, J. 2003. Learners for Life: Student Approaches to Learning. Results from PISA 2000. Paris: Organisation for Economic Co-operation and Development.
3. Beauducel, A. and Herzberg, P. Y. 2006. On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA. Structural equation Modeling, 13, 186-203. doi: 10.1207/ s15328007sem1302_2
4. Bentler, P. M. 1990. Comparative fit indexes in structural models. Psychological Bulletin, 107, 238-246. doi:10.1037/0033-2909.107.2.238
5. Bentler, P. M. and Bonett, D. G. 1980. Significance tests and goodness-of-fit in the analysis of covariance structures. Psychological Bulletin, 88, 588-606.
6. Bong, M. 2001. Between- and within-domain relations of academic motivation among middle and high school students: Self-efficacy, task value, and achievement goals. Journal of Educational Psychology, 93, 23-34. doi:10.1037/0022-0663.93.1.23
7. Browne, M. W. and Cudeck, R. 1993. Alternative ways of assessing model fit. In K. A. Bollen and J. S. Long (Eds.), Testing Structural Equation Models (pp. 136-162). Newbury Park, CA: Sage.
8. Brunner, M., Keller, U., Hornung, C., Reichert, M., and Martin, R. 2009. The cross-cultural generalizability of a new structural model of academic self-concepts. Learning and Individual Differences, 19, 387-403.
9. Byrne, B. M., Shavelson, R. J., and Muthén, B. O. 1989. Testing for the equivalence of factor covariance and mean structures-the issue of partial measurement invariance. Psychological Bulletin, 105, 456-466.
10. Chen, F. F. 2007. Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14, 464-504.
11. Chen, F. F. 2008. What happens if we compare chopsticks with forks? The impact of making inappropriate comparisons in cross-cultural research. Journal of Personality and Social Psychology, 95, 1005-1018.
12. Chen, F. F., Curran, P. J., Bollen, K. A., Kirby, J., and Paxton, P. 2008. An empirical evaluation of the use of fixed cutoff points in RMSEA test statistic in structural equation models. Sociological Methods & Research, 36, 462-494. doi: 10.1177/0049124108314720.
13. Cheung, G. W. and Rensvold, R. B. 2002. Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233-255. doi:10.1207/S15328007SEM0902_5
14. Chiu, M. M. and Klassen, R. M. 2010. Relations of mathematics self-concept and its calibration with mathematics achievement: Cultural differences among fifteen-year-olds in 34 countries. Learning and Instruction, 20, 2-17. doi:10.1016/j.learninstruc.2008.11.002
15. Dolan, C. V. 1994. Factor analysis of variables with 2, 3, 5 and 7 response categories: A comparison of categorical variable estimators using simulated data. British Journal of Mathematical and Statistical Psychology, 47, 309-326. doi:10.1111/j.2044-8317.1994.tb01039.x
16. Eccles (Parsons), J. S. 1983. Expectancies, values, and academic behaviours. In J. T. Spence (Ed.), Achievement and Achievement Motivation (pp. 75-146). San Francisco, CA: W. H. Freeman.
17. Enders, C. K. 2010. Applied Missing Data Analysis. New York: Guilford.
18. Graham, J. W. 2009. Missing data analysis: Making it work in the real world. Annual Review of Psychology, 60, 549-576.
19. Heene, M., Hilbert, S., Draxler, C., Ziegler, M., and Bühner, M. 2011. Masking misfit in confirmatory factor analysis by increasing unique variances: A cautionary note on the usefulness of cutoff values of fit indices. Psychological Methods, 16, 319-336. doi:10.1037/a0024917
20. Hu, L. and Bentler, P. M. 1999. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55. doi:10.1080/10705519909540118.
21. IBM. 2011. IBM SPSS Statistics. Version 20 [Computer Software].
22. Jöreskog, K. G. and Goldberger, A. S. 1975. Estimation of a model with multiple indicators and multiple causes of a single latent variable. Journal of the American Statistical Association, 70, 631-639.
23. Kaplan, D. 2009. Structural Equation Models. Foundations and Extensions (2nd ed.). Thousand Oaks, CA: Sage.
24. Lee, J., Little, T. D., and Preacher, K. J. 2011. Methodological issues in using structural equation models for testing differential item functioning. In E. Davidov, P. Schmitt, and J. Billiet (Eds.), Cross-Cultural Analysis. Methods and Applications (Vol. 1, pp. 55-85). New York: Routledge. doi:10.1109/TSC.2008.18
25. Lei, P.-W. 2009. Evaluating estimation methods for ordinal data in structural equation modeling. Quality & Quantity, 43, 495-507. doi: 10.1007/s11135-007-9133-z.
26. Little, T. D. 1997. Mean and covariance structures (MACS) analyses of cross-cultural data: Practical and theoretical issues. Multivariate Behavioral Research, 32, 53-76.
27. Lubke, G. H. and Muthén, B. O. 2004. Applying MG confirmatory factor models for continuous outcomes to Likert scale data complicates meaningful group comparisons. Structural equation Modeling, 11, 514-534. doi:10.1207/s15328007sem1104_2
28. Lumley, T. 2004. Analysis of complex survey samples. Journal of Statistical Software, 9, 1-19.
29. Lumley, T. 2011. Survey: Analysis of Complex Survey Samples. R package version 3.26.
30. Marsh, H. W. 1986. Verbal and math self-concepts: An internal/external frame of reference model. American Educational Research Journal, 23, 129-149.
31. Marsh, H. W. 1994. Sp1ort motivation orientations: Beware of the jingle-jangle fallacies. Journal of Sport and Exercise Psychology, 16, 365-380.
32. Marsh, H. W. 2006. Self-Concept Theory, Measurement and Research into Practice: The Role of Self-Concept in Educational Psychology. Leicester, UK: British Psychological Society.
33. Marsh, H. W., Balla, J. R., and Hau, K. T. 1996. An evaluation of incremental fit indexes: A clarification of mathematical and empirical properties. In G. A. Marcoulides and R. E. Schumacker (Eds.), Advanced Structural Equation Modeling Techniques (pp. 315-353. Mahwah, NJ: Lawrence Erlbaum.
34. Marsh, H. W., Balla, J. R., and McDonald, R. P. 1988. Goodness-of-fit indexes in confirmatory factor analysis: The effect of sample size. Psychological Bulletin, 103, 391-410.
35. Marsh, H. W. and Craven, R. G. 2006. Reciprocal effects of self-concept and performance from a multidimensional perspective: Beyond seductive pleasure and unidimensional perspectives. Perspectives on Psychological Science, 1, 133-163.
36. Marsh, H. W., Craven, R. G., and Debus, R. L. 1998. Structure, stability, and development of young children's self-concepts. Child Development, 69, 1030-1053.
37. Marsh, H. W., Craven, R. G., Hinkley, J. W., and Debus, R. L. 2003. Evaluation of the Big-Two-Factor Theory of academic motivation orientations: An evaluation of jingle-jangle fallacies. Multivariate Behavioral Research, 38, 189-224.
38. Marsh, H. W. and Hau, K.-T. 2004. Explaining paradoxical relations between academic self-concepts and achievements: Cross-cultural generalizability of the internal/external frame of reference predictions across 26 countries. Journal of Educational Psychology, 96, 56-67.
39. Marsh, H. W., Hau, K. T., and Wen, Z. 2004. In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indices and dangers in overgeneralizing Hu and Bentler's 1999 findings. Structural Equation Modeling, 11, 320-341.
40. Marsh, H. W. and Grayson, D. 1994. Longitudinal stability of latent means and individual differences: A unified approach. Structural Equation Modeling, 1, 317-359.
41. Marsh, H. W., Hau, K.-T., Artelt, C., Baumert, J., and Peschar, J. 2006. OECD's brief self-report measure of educational psychology's most useful affective constructs: Cross-cultural, psychometric comparisons across 25 countries. International Journal of Testing, 6, 311-360. doi:10.1207/s15327574ijt0604_1
42. Marsh, H. W., Muthén, B. O., Asparouhov, T., Lüdtke, O., Robitzsch, A., Morin, A. J. S., and Trautwein, U. 2009. Exploratory structural equation modeling, integrating CFA and EFA: Application to students' evaluations of university teaching. Structural Equation Modeling, 16, 439-476.
43. Marsh, H. W., Seaton, M., Trautwein, U., Lüdtke, O., Hau, K.-T., O'Mara, A. J., and Craven, R. G. 2008. The big-fish-little-pond effect stands up to critical scrutiny: Implications for theory, methodology, and future research. Educational Psychology Review, 3, 319-350.
44. Marsh, H. W. and Shavelson, R. 1985. Self-concept: Its multifaceted, hierarchical structure. Educational Psychologist, 20, 107-125.
45. Marsh, H. W., Trautwein, U., Lüdtke, O., Köller, O., and Baumert, J. 2006. Integration of multidimensional self-concept and core personality constructs: Construct validation and relations to well-being and achievement. Journal of Personality, 74, 403-456.
46. Marsh, H. W. and Yeung, A. S. 1997. Causal effects of academic self-concept on academic achievement: Structural equation models of longitudinal data. Journal of Educational Psychology, 89, 41-54.
47. Muthén, L. K. and Muthén, B. O. (1998-2010). Mplus User's Guide (6th ed.). Los Angeles, CA: Muthén & Muthén.
48. Meredith, W. 1993. Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58, 525-543.
49. Nagengast, B. and Marsh, H. W. 2011. The negative effect of school-average ability on science self-concept in the UK, the UK countries and the world: The Big-Fish-Little-Pond-Effect for PISA 2006. Educational Psychology, 31, 629-656.
50. Nagengast, B. and Marsh, H. W. 2012. Big fish in little ponds aspire more: Mediation and cross-cultural generalizability of school-average ability effects on self-concept and career aspirations in science. Journal of Educational Psychology, 104, 1033-1053. doi:10.1037/a0027697
51. Nagengast, B., Marsh, H. W., Scalas, L. F., Xu, M. K., Hau, K.-T., and Trautwein, U. 2011. Who took the "x" out of expectancy-value theory? A psychological mystery, a substantive-methodological synergy, and a cross-national Generalization. Psychological Science, 22, 1058-1066. doi:10.1177/0956797611415540
52. OECD. 2007. PISA 2006 Science Competencies for Tomorrow's World. Paris: OECD.
53. OECD. 2009. PISA 2006. Technical Report. Paris, France: OECD.
54. OECD. 2010. TALIS 2008. Technical Report. Paris, France: OECD.
55. Peschar, J. L., Veenstra, R., and Molenaar I. W. 1999. Self Regulated Learning as a Cross-Curricular Competency. The Construction of Instruments in 22 Countries for the PISA Main Study 2000. Paris: Organisation for Economic Co-operation and Development.
56. Pintrich, P. R. 2003. A motivational science perspective on the role of student motivation in learning and teaching contexts. Journal of Educational Psychology, 95, 667-686.
57. Pituch, K. A. and Stapleton, L. M. 2011. Hierarchical linear and structural equation modeling approaches to mediation analysis in randomized field experiments. In M. Williams and P. Vogt (Eds.), The Sage Handbook of Innovation in Social Research Methods (pp. 590-619). Thousand Oaks, CA: Sage.
58. Preacher, K. J., Zyphur, M. J., and Zhang, Z. 2010. A general multilevel SEM framework for assessing multilevel mediation. Psychological Methods, 15, 209-233. doi:10.1037/a0020141
59. R Development Core Team. 2011. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Retrieved from http://www.R-project.org/.
60. Raykov, T. 2009. Evaluation of scale reliability for unidimensional measures using latent variable modeling. Measurement and Evaluation in Counseling and Development, 42, 223-232.
61. Raykov, T. and Shrout, P. E. 2002. Reliability of scales with general structure: Point and interval estimation using a structural equation modeling approach. Structural Equation Modeling, 9, 195-212. doi:10.1207/S15328007SEM0902_3
62. Revelle, W. and Zinbarg, R. E. 2009. Coefficients alpha, beta, omega, and the glb: Comments on Sijtsma. Psychometrika, 74, 145-154. doi:10.1007/s11336-008-9102-z
63. Rhemtulla, M., Brosseau-Liard, P. E., and Savalei, V. 2012. When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17, 354-373. doi: 10.1037/a0029315
64. Rubin, D. B. 1987. Multiple Imputation for Nonresponse in Surveys. Hoboken, NJ: Wiley.
65. Schafer, J. L. 1997. Analysis of Incomplete Multivariate Data. Boca Raton, FL: Chapman & Hall/CRC.
66. Seaton, M., Marsh, H. W., and Craven, R. G. 2009. Earning its place as a pan-human theory: Universality of the big-fish-little-pond effect across 41 culturally and economically diverse countries. Journal of Educational Psychology, 101, 403-419. doi:10.1037/a0013838
67. Shavelson, R. J., Hubner, J. J., and Stanton, G. C. 1976. Validation of construct interpretations. Review of Educational Research, 46, 407-441.
68. Sijtsma, K. 2009. On the use, the misuse, and the very limited usefulness of Cronbach's alpha. Psychometrika, 74, 107-120.
69. Sörbom, D. 1974. A general method for studying differences in factor means and factor structures between groups. British Journal of Mathematical and Statistical Psychology, 27, 229-239.
70. Steiger, J. H. and Lind, J. C. 1980, May. Statistically Based Tests for the Number of Common Factors. Paper presented at the annual meeting of the Psychometric Society, Iowa City, IA.
71. Valentine, J. C. and DuBois, D.L. 2005. Effects of self-beliefs on academic achievement and vice versa. In H.W. Marsh, R.G. Craven, and D.M. McInerney (Eds.), The New Frontiers of Self Research (pp. 53-77). Greenwich, CT: Information Age.
72. van de Vijver, F. and Leung, K. 2000. Methodological issues in psychological research on culture. Journal of Cross-Cultural Psychology, 31, 33-51.
73. Vandenberg, R. J. and Lance, C. E. 2000. A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4-70.
74. Widaman, K. F. and Reise, S. P. 1997. Exploring the measurement invariance of psychological instruments: Applications in the substance use domain. In K.J. Bryant, M. Windle, and S.G. West (Eds.), The Science of Prevention: Methodological Advances from Alcohol and Substance Abuse Research (pp. 281-324). Washington, DC: American Psychological Association.
75. Widaman, K. F. and Thompson, J. S. 2003. On specifying the null model for incremental fit indices in structural equation modeling. Psychological Methods, 8, 16-37. doi:10.1037/1082-989X.8.1.16
76. Wigfield, A. and Eccles, J. S. 2000. Expectancy-value theory of achievement motivation. Contemporary Educational Psychology, 25, 65-81.