Five methods to score the teacher observation of classroom adaptation checklist and to examine group differences

Journal of Experimental Education

Volumn 83, Issue 1, 2015, Pages 24-50

  Wang, Ze ^a   Rohrer, David ^a   Chuang, Chi Ching ^a   Fujiki, Mayo ^a   Herman, Keith ^a   Reinke, Wendy ^a  

^a UNIVERSITY OF MISSOURI   (United States)

Author keywords

Factor analysis; Measurement invariance; Ordered categorical variables; Scoring methods; TOCA C

Indexed keywords

EID: 84920280151     PISSN: 00220973     EISSN: 19400683     Source Type: Journal    
DOI: 10.1080/00220973.2013.876230     Document Type: Article

References (62)

1. Adams, R. J., Wilson, M., & Wang, W.-C. (1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21, 1-23. doi:10.1177/0146621697211001
2. Arbuckle, J. L. (1996). Full information estimation in the presence of incomplete data. In G. A. Marcoulides & R. E. Schumacker (Eds.), Advanced structural equation modeling: Issues and techniques (pp. 243-277). Mahwah, NJ: Erlbaum.
3. Banks, K. (2013). A synthesis of the peer-reviewed differential bundle functioning research. Educational Measurement: Issues and Practice, 32, 43-55. doi:10.1111/emip.12004
4. Beauducel, A., & Herzberg, P. Y.(2006). On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA. Structural Equation Modeling, 13, 186-203.
5. Bock, R. D., & Aitkin, M. (1981 ). Marginal maximum likelihood estimation of item parameters: An application of the EM algorithm. Psychometrika, 46, 443-459.
6. Bollen, K. A., Bauer, D. J., & Christ, S. L. (2010). Overview of structural equation models and recent extensions. In S. Kolenikov, D. Steinley, & L. Thombs (Eds.), Statistics in the social sciences: Current methodological developments (pp. 37-79). Hoboken, NJ: Wiley.
7. Bovaird, J. A., & Koziol, N. A. (2012). Measurement models for ordered-categorical indicators. In R. H. Hoyle (Ed.), Handbook of structural equation modeling (pp. 495-511). New York, NY: Guilford Press.
8. Bradshaw, C. P., Schaeffer, C. M., Petras, H., & Ialongo, N. (2010). Predicting negative life outcomes from early aggressive-disruptive behavior trajectories: Gender differences in maladaptation across life domains. Journ al of Youth and Adolescence, 39, 953-966. doi:10.1007/s10964-009-9442-8
9. Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York, NY: Guilford Press.
10. Byrne, B. M. (2010). Structural equation modeling with AMOS: Basic concepts, applications, and programming (2nd ed.). New York, NY: Routledge.
11. Cai, L. (2010). Metropolis-Hastings Robbins-Monro algorithm for confirmat ory item factor analysis. Journal of Educational and Behavioral Statistics, 35, 307-335. doi:10.3102/1076998609353115
12. Cai, L., du Toit, S. H. C., & Thissen, D. (2011). IRTPRO: Flexible, multidimensional, multiple categorical IRT modeling [Computer software]. Chicago, IL: SSI International.
13. Cohen, J. (1988). Statistical power analysis for the bahavioral sciences (2nd ed.). Mahwah, NJ: Erlbaum.
14. Cooke, D. J., Kosson, D. S., & Michie, C. (2001). Psychopathy and ethnicity: Structural, item, and test generalizability of the Psychopathy Checklist-Revised (PCL-R) in Caucasian and African American participants. Psychologic al Assessment, 13, 531-542. doi:10.1037/1040-3590.13.4.531
15. Dimitrov,D.M. (2010). Testing for factorial invariance in the context of construct validation. Measurement and Evaluation in Counseling and Development, 43 , 121-149. doi:10.1177/0748175610373459
16. Dorman, J. P. (2003). Cross-national validation of the what is happening in this class (WIHIC) questionnaire using confirmatory factor analysis. Learning Environments Research, 6 , 231-245. doi:10.1023/a:1027355123577
17. Edwards, M. C., Wirth, R. J., Houts, C. R., & Xi, N. (2012). Categorical data in the structural equation modeling framework. In R. H. Hoyle (Ed.), Handbook of stru ctural equation modeling (pp. 195-208). New York, NY: Guilford Press.
18. Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Mahwah, NJ: Erlbaum.
19. Flora D. B.,Curran , P. J. 2004 An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data Psychological Methods 9 466-491. 10.1037/1082-989X.9.4.466
20. Forero, C. G., Maydeu-Olivares, A., & Gallardo-Pujol, D. (2009). Factor analysis with ordinal indicators: A Monte Carlo study comparing DWLS and ULS estimation. Structural Equation Modeling, 16, 625-641.doi:10.1080/107055109032 03573
21. Greener, S., & Crick, N. R. (1999). Normative beliefs about prosocial behavior in middle childhood: What does it mean to be nice Social Development, 8, 349-363. doi:10.1111/1467-9507.00100
22. Gregorich, S. E. (2006). Do self-report instruments allow meaningful comparisons across diverse population groups Testing measurement invariance using the confirmatory factor analysis framework. Medical Care, 44, S78-S94 .
23. Hambleton, R. K., & Swaminathan, H. (1985). Item response theory: Principles and applications. Boston, MA: Kluwer-Nijoff.
24. Harter, S. (1981). A new self-report scale of intrinsic versus e xtrinsic orientation in the classroom: Motivational and informational components. Developmental Psychology, 17, 300-312. doi:10.1037/0012-1649.17.3.300
25. Hu, L.-T., & Bentler, P.M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3, 424-453. doi:10.1037/1082-989x.3.4.424
26. Hu, L.-T., & Bentler, P. M. (1999). Cutoff cri teria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.
27. Jensen, A. R. (1976). Test bias and construct validity. Phi Delta Kappan, 58, 340-346.
28. Jöreskog, K. G. (1990). New developments in LISREL: Analysis of ordinal variables using polychoric correlations and weighted least squares. Quality and Quantity, 24, 387-404. doi:10.1007/bf00152012
29. Jöreskog , K. G., & Moustaki, I. (2001). Factor analysis of ordinal variables: A comparison of three approaches. Multivariate Behavioral Research, 36, 347-387. doi:10.1207/s15327906347-387
30. Kim, E. S., & Yoon, M. (2011). Testing measurement invariance: A comparison of multiple-group categorical CFA and IRT. Structural Equation Modeling, 18, 212-228. doi:10.1080/10705511.2011.557337
31. Koth, C. W., Bradshaw, C. P., & Leaf, P. J. (2009). Teacher Observation of Classroom Adaptation-Checklist: Development and factor structure. Measurement and Evaluation in Counseling and Development, 42, 15-30. doi:10.1177/0748175609333560
32. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
33. McDonald, R. P. (1999). Test theory: A unified treatment. Mahwah, NJ: Erlbaum.
34. McG rew, K. S., & Woodcock, R. W. (2001). Technical manual. Woodcock-Johnson III. Itasca, IL: Riverside.
35. McKeon, J. J. (1966). Canonical analysis: Some relations between canonical correlation, factor analysis, discriminant function analysis, and scaling theory. Sarasota, FL: Psychometric Society.
36. Millsap, R. E. (2011). Statistical approach to measurement invariance. New York, NY: Routledge.
37. Millsap, R. E., & Olivera-Aguilar, M. (2012). Investigating measurement invariance using confirmatory factor analysis. In R. H. Hoyle (Ed.), Handbook of structural equation modeling (pp. 380-392). New York, NY: Guilford Press.
38. Millsap, R. E., & Yun-Tein, J. (2004). Assessing factorial invariance in ordered-categorical measures. Multivariate Behavioral Research, 39, 479-515.
39. Mislevy, R. J., Johns on, E. G., & Muraki, E. (1992). Scaling procedures in NAEP. Journal of Educational Statistics, 17, 131-154. doi:10.2307/1165166.
40. Muthén, B. 1984 A general structural equation model with dichotomous, ordered categorical, and continuous latent variable indicators Psychometrika 49 115-132.
41. Muthén, B. (1993). Goodness of fit with categorical and other non-normal variables. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 205-234). Newbury Park, CA: Sage.
42. Muthén, L. K.,&Muthén, B. O. (2012). Mplus user's guide (7th ed.). Los Angeles, CA: Muthén & Muthén.
43. Olsson, U. (1979).Maximum likelihood estimation of the polychoric correlation coefficient. Psychometrika, 44, 443-460. doi:10.1007/bf02296207
44. Osterlind, S. J.,&Wang, Z. (2012). Item response theory inmeasurement, assessment and evaluation for higher education. In C. Secolsky & D. B. Denison (Eds.), Handbook on measurement, assessment, and evaluation in higher education (pp. 150-160). New York, NY: Routledge.
45. Ostrov, J. M., Crick, N. R., & Keating, C. F. (2005). Gender-biased perceptions of preschoolers' behavior: How much is aggression and prosocial behavior in the eye of the beholder Sex Roles, 52, 393-398. doi:10.1007/ s11199-005-2681-6
46. Petras, H., Ialongo, N., Lambert, S. F., Barrueco, S., Schaeffer, C. M., Chilcoat, H., & Kellam, S. (2005). The utility of elementary school TOCA-R scores in identifying later criminal c ourt violence among adolescent females. Journal of the American Academy of Child & Adolescent Psychiatry, 44, 790-797. doi:10.1097/01.chi.0000166378.22651.63
47. Raykov, T. (2012). Scale construction and development using structural equation modeling. In R. H. Hoyle (Ed.), Handbook of structural equation modeling (pp. 472-492). New York, NY: Guilford Press.
48. Reckase, M. D. (2009). Multidimensional item response theory. New York, NY: Springer.
49. Reynolds, C. R., & Ramsay, M. C. (2003). Bias in psychological assessment: An empirical review and recommendations. In J. R. Gr aham & J. A. Naglieri (Eds.), Handbook of psychology (Vol. 10, pp. 67-93). Hoboken, NJ: Wiley.
50. Samejima, F. (1969). Estimation of latent trait ability using a response pattern of graded scores (Psychometrika Monograph No. 17). Richmond, VA: Psychometric Society.
51. Schilling, S., & Bock, R. D. (2005). High-dimensional maximum marginal likelihood item factor analysis by adaptive quadrature. Psychometrika, 70, 533-555.
52. Steinmetz, H., Schmidt, P., Tina-Booh, A., Wieczorek, S., & Schwartz, S. (2009). Testing measurement invariance using multigroup CFA: differences between educational groups in human values measurement. Quality & Quantity, 43, 599-616. doi:10.1007/s11135-007-9143-x
53. Webster-Stratton, C. (1994). The Incredible Years Teacher Training Series. Seattle, WA: Incredible Years.
54. Werthamer-Larsson, L., Kell am, S. & Oveson-McGregor, K. E. (1990). Teacher interview: Teacher Observation of Classroom Adaptation-Revised (TOCA-R). Johns Hopkins Prevention Center Training Manual. Baltimore, MD: Johns Hopkins University.
55. Werthamer-La rsson, L., Kellam, S., & Wheeler, L. (1991). Effect of first-grade classroom environment on shy behavior, aggressive behavior, and concentration problems. American Journal of Community Psychology, 19, 585-602. do i:10.1007/bf00937993
56. Wirth, R. J., & Edwards, M. C. (2007). Item factor analysis: Current approaches and future directions. Psychological Methods, 12, 58-79. doi:10.1037/1082-989x.12.1.58
57. Wu, A. D., Li, Z., & Zumbo, B. D. (2007). Decoding the meaning of factorial invariance and updating the practice of multi-group confirmatory factor analyses: A demonstration with TIMSS data. Practical Assessment, Research, and Evaluation , 12(3), 1-26.
58. Wu, J., King, K. M.,Witkiewitz, K., Racz, S. J., & McMahon, R. J. (2012). Item analysis and differential item functioning of a brief conduct problem screen. Psychological Assessment, 24(2), 444-454. doi:10.1037/a0025831
59. Yao L.,Boughton K. A. 2007 A multidimensional item response modeling approach for improving subscale proficiency estimation and classification Applied Psycholog ical Measurement 31 83-105. 10.1177/0146621606291559
60. Yuan, K.-H., & Bentler, P. M. (2000). Three likelihood-based methods for mean and covariance structure analysis with nonnormal missing data. S ociological methodology, 30, 165-200. doi:10.1111/0081-1750.00078
61. Zimmer-Gembeck, M. J., Geiger, T. C., & Crick, N. R. (2005). Relational and physical aggression, prosocial behavior, and peer relations. Journal of Early Adolescence, 25, 421-452. doi:10.1177/0272431605279841
62. Zumbo, B. D. (1999). A handbook on the theory an d methods of differential item functioning (DIF): Logistic regression modeling as a unitary framework for binary and Likert-type (ordinal) item scores. Ottawa, Canada: Directorate of Human Resources Research and Evaluation, Department of National Def ense.