Modeling competence development in the presence of selection bias

Behavior Research Methods

Volumn 50, Issue 6, 2018, Pages 2426-2441

  Zinn, Sabine ^a   Gnambs, Timo ^a  

^a Leibniz Institute for Educational Trajectories (LIfBi)   (Germany)

Author keywords

Competence development; Longitudinal design; Selection bias; Selectivity analysis

Indexed keywords

HUMAN; LONGITUDINAL STUDY; MEASUREMENT ACCURACY; PROBABILITY; SELECTION BIAS; STATISTICAL ANALYSIS; STATISTICAL MODEL;

DATA ACCURACY; DATA INTERPRETATION, STATISTICAL; HUMANS; LIKELIHOOD FUNCTIONS; LONGITUDINAL STUDIES; MODELS, STATISTICAL; PROBABILITY; SELECTION BIAS;

EID: 85042067152     PISSN: 1554351X     EISSN: 15543528     Source Type: Journal    
DOI: 10.3758/s13428-018-1021-z     Document Type: Article

References (75)

1. Adams, R. J. (2005). Reliability as a measurement design effect. Studies in Educational Evaluation, 31, 162–172. 10.1016/j.stueduc.2005.05.008
2. Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19, 716–723. 10.1109/TAC.1974.1100705
3. Allison, P. D. (2001). Missing data. Thousand Oaks: Sage
4. Allison, P. D. (2003). Missing data techniques for structural equation models. Journal of Abnormal Psychology, 112, 545–557. 10.1037/0021-843X.112.4.545
5. Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67, 1–48. 10.18637/jss.v067.i01
6. Blossfeld, H.-P., Roßbach, H.-G, & von Maurice, J. (Eds.) (2011). Education as a lifelong process—The German National Educational Panel Study (NEPS) [Special issue]. Zeitschrift für Erziehungswissenschaft, 14
7. Brunner, M., Lang, F. R., & Lüdtke, O. (2014). Erfassung der fluiden kognitiven Leistungsfähigkeit über die Lebensspanne im Rahmen der National Educational Panel Study: Expertise [Assessment of fluid cognitive skills over the life span in the National Educational Panel Study: Expertise]. (NEPS Working Paper 42). Bamberg, Germany: Otto-Friedrich-Universität, Nationales Bildungspanel
8. Carpenter, J., & Kenward, M. (2012). Multiple imputation and its application. Hoboken: Wiley
9. Chatfield, M. D., Brayne, C. E., & Matthews, F. E. (2005). A systematic literature review of attrition between waves in longitudinal studies in the elderly shows a consistent pattern of dropout between differing studies. Journal of Clinical Epidemiology, 58, 13–19. 10.1016/j.jclinepi.2004.05.006
10. Chou, C. P., Bentler, P. M., & Pentz, M. A. (1998). Comparisons of two statistical approaches to study growth curves: The multilevel model and the latent curve analysis. Structural Equation Modeling, 5, 247–266. 10.1080/10705519809540104
11. Curran, P. J., Obeidat, K., & Losardo, D. (2010). Twelve frequently asked questions about growth curve modeling. Journal of Cognition and Development, 11, 121–136. 10.1080/15248371003699969
12. Deary, I. J., Strand, S., Smith, P., & Fernandes, C. (2007). Intelligence and educational achievement. Intelligence, 35, 13–21.10.1016/j.intell.2006.02.001
13. Diggle, P., & Kenward, M. G. (1994). Informative drop-out in longitudinal data analysis. Journal of the Royal Statistical Society: Series C, 43, 49–93. 10.2307/2986113
14. Duchhardt, C., & Gerdes, A. (2013). NEPS Technical Report for Mathematics—Scaling results of Starting Cohort 4 in ninth grade (NEPS Working Paper No. 22). Bamberg: University of Bamberg, National Educational Panel Study
15. Enders, C. K. (2010). Applied missing data analysis. Guilford Press. New York. U.S
16. Enders, C. K. (2011). Missing not at random models for latent growth curve analyses. Psychological Methods, 16, 1–16. 10.1037/a0022640
17. Enders, C. K., & Bandalos, D. L. (2001). The relative performance of full information maximum likelihood estimation for missing data in structural equation models. Structural Equation Modeling, 8, 430–457. 10.1207/S15328007SEM0803_5
18. Enders, C. K., Keller, B. T., & Levy, R. (2017). A fully conditional specification approach to multilevel imputation of categorical and continuous variables. Psychological Methods. Advance online publication. 10.1037/met0000148
19. Enders, C. K., & Mansolf, M. (2016). Assessing the fit of structural equation models with multiply imputed data. Psychological Methods. Advance online publication. 10.1037/met0000102
20. Fischer, L., Rohm, T., & Gnambs, T. (2017). NEPS Technical Report for Mathematics—Scaling results of Starting Cohort 4 for Grade 12 (NEPS Working Paper No. 12). Bamberg: University of Bamberg, National Educational Panel Study
21. Fischer, L., Rohm, T., Gnambs, T., & Carstensen, C. H. (2016). Linking the data of the competence tests (NEPS Survey Paper No. 1). Bamberg: Leibniz Institute for Educational Trajectories, National Educational Panel Study
22. Galimard, J. E., Chevret, S., Protopopescu, C., & Resche-Rigon, M. (2016). A multiple imputation approach for MNAR mechanisms compatible with Heckman’s model. Statistics in Medicine, 35, 2907–2920. 10.1002/sim.6902
23. Goldberger, A. S. (1981). Linear regression after selection. Journal of Econometrics, 15, 357–366. 10.1016/0304-4076(81)90100-7
24. Gottfredson, N. C., Bauer, D. J., & Baldwin, S. A. (2014). Modeling change in the presence of nonrandomly missing data: Evaluating a shared parameter mixture model. Structural Equation Modeling, 21, 196–209. 10.1080/10705511.2014.882666
25. Graham, J. W. (2003). Adding missing-data-relevant variables to FIML-based structural equation models. Structural Equation Modeling, 10, 80–100. 10.1207/S15328007SEM1001_4
26. Graham, J. W. (2009). Missing data analysis: Making it work in the real world. Annual Review of Psychology, 60, 549–576. 10.1146/annurev.psych.58.110405.085530
27. Green, S. B., & Yang, Y. (2009). Reliability of summed item scores using structural equation modeling: An alternative to coefficient alpha. Psychometrika, 74, 155–167. 10.1007/s11336-008-9099-3
28. Grund, S., Lüdtke, O., & Robitzsch, A. (2016). Multiple imputation of multilevel missing data: An introduction to the R Package pan. SAGE Open, 6(4), 2158244016668220
29. Helmschrott, S., & Martin, S. (2014). Nonresponse in PIAAC Germany. Methods, Data, Analyses, 8, 243–266. 10.12758/mda.2014.010
30. Henrich, J., Heine, S. J., & Norenzayan, A. (2010). Most people are not WEIRD. Nature, 466, 29. 10.1038/466029a
31. Jansen, M., Scherer, R., & Schroeders, U. (2015). Students’ self-concept and self-efficacy in the sciences: Differential relations to antecedents and educational outcomes. Contemporary Educational Psychology, 41, 13–24. 10.1016/j.cedpsych.2014.11.002
32. Kalton, G., & Flores-Cervantes, I. (2003). Weighting methods. Journal of Official Statistics, 19, 81–97
33. Kriegbaum, K., Jansen, M., & Spinath, B. (2015). Motivation: A predictor of PISA’s mathematical competence beyond intelligence and prior test achievement. Learning and Individual Differences, 43, 140–148. 10.1016/j.lindif.2015.08.026
34. Kuncel, N. R., Hezlett, S. A., & Ones, D. S. (2004). Academic performance, career potential, creativity, and job performance: Can one construct predict them all? Journal of Personality and Social Psychology, 86, 148–161. 10.1037/0022-3514.86.1.148
35. Leon, A. C., Demirtas, H., & Hedeker, D. (2007). Bias reduction with an adjustment for participants’ intent to dropout of a randomized controlled clinical trial. Clinical Trials, 4, 540–547. 10.1177/1740774507083871
36. Li, M., Chen, N., Cui, Y., & Liu, H. (2017). Comparison of different LGM-based methods with MAR and MNAR dropout data. Frontiers in Psychology, 8, 722. 10.3389/fpsyg.2017.00722
37. Little, R. J. (1988). A test of missing completely at random for multivariate data with missing values. Journal of the American Statistical Association, 83, 1198–1202. 10.2307/2290157
38. Little, R. J. (1995). Modeling the drop-out mechanism in repeated-measures studies. Journal of the American Statistical Association, 90, 1112–1121. 10.2307/2291350
39. Little, R. J. (2008). Selection and pattern-mixture models. In G. Fitzmaurice, M. Davidian, G. Verbeke & G. Molenberghs (Eds.), Longitudinal data analysis (pp. 409–431). Boca Raton: CRC Press
40. Little, R. J., D’Agostino, R., Cohen, M. L., Dickersin, K., Emerson, S. S., Farrar, J. T., … Neaton, J. D. (2012). The prevention and treatment of missing data in clinical trials. New England Journal of Medicine, 367, 1355–1360. 10.1056/NEJMsr1203730
41. Little, R. J., & Wang, Y. (1996). Pattern-mixture models for multivariate incomplete data with covariates. Biometrics, 52, 98–111. 10.2307/2533148
42. Liu, S., Rovine, M. J., & Molenaar, P. (2012). Selecting a linear mixed model for longitudinal data: Repeated measures analysis of variance, covariance pattern model, and growth curve approaches. Psychological Methods, 17, 15–30. 10.1037/a0026971
43. MacCallum, R. C., Kim, C., Malarkey, W. B., & Kiecolt-Glaser, J. K. (1997). Studying multivariate change using multilevel models and latent curve models. Multivariate Behavioral Research, 32, 215–253. 10.1207/s15327906mbr3203_1
44. McNeish, D. (2017). Thanks coefficient alpha, we’ll take it from here. Psychological Methods. Advance online publication. 10.1037/met0000144
45. Molenberghs, G., Beunckens, C., Sotto, C., & Kenward, M. G. (2008). Every missingness not at random model has a missingness at random counterpart with equal fit. Journal of the Royal Statistical Society: Series B, 70, 371–388. 10.1111/j.1467-9868.2007.00640.x
46. Molenberghs, G., & Kenward, M. (2007). Missing data in clinical studies. Chichester: Wiley
47. Molenberghs, G., Verbeke, G., & Kenward, M. G. (2008). Sensitivity analysis for incomplete data. In G. Fitzmaurice, M. Davidian, G. Verbeke, & G. Molenberghs (Eds.), Longitudinal data analysis (pp. 501–552). Boca Raton: CRC Press
48. Muthén, B., Asparouhov, T., Hunter, A. M., & Leuchter, A. F. (2011). Growth modeling with nonignorable dropout: Alternative analyses of the STAR*D antidepressant trial. Psychological Methods, 16, 17–33. 10.1037/a0022634
49. Muthén, L. K., & Muthén, B. O. (1998–2017). MPlus user’s guide (8th ed.). Los Angeles: Muthén & Muthén
50. Neumann, I., Duchhardt, C., Grüßing, M., Heinze, A., Knopp, E., & Ehmke, T. (2013). Modeling and assessing mathematical competence over the lifespan. Journal for Educational Research Online, 5, 80–109
51. Pohl, S., & Carstensen, C. H. (2013). Scaling the competence tests in the National Educational Panel Study. Journal for Educational Research Online, 5, 189–216
52. Prenzel, M., Carstensen, C. H., Schöps, K., & Maurischat, C. (2006). Die Anlage des Längsschnitts bei PISA 2003 [The longitudinal design in PISA 2003]. In PISA Konsortium Deutschland (Ed.), PISA 2003: Untersuchungen zur Kompetenzentwicklungim Verlauf eines Schuljahres [PISA 2003: Investigations of the development of competencies across one school year] (pp. 29–62). Münster, Germany: Waxmann
53. R Core Team. (2017). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from https://www.R-project.org/
54. Rammstedt, B., Martin, S., Zabal, A., Carstensen, C., & Schupp, J. (2017). The PIAAC longitudinal study in Germany: Rationale and design. Large-Scale Assessments in Education, 5(1). 10.1186/s40536-017-0040-z
55. Rogelberg, S. G., Fisher, G. G., Maynard, D. C., Hakel, M. D., & Horvath, M. (2001). Attitudes toward surveys: Development of a measure and its relationship to respondent behavior. Organizational Research Methods, 4, 3–25. 10.1177/109442810141001
56. Rohrer, J. M., Egloff, B., & Schmukle, S. C. (2017). Probing birth-order effects on narrow traits using specification-curve analysis. Psychological Science, 28, 1821–1832. 10.1177/0956797617723726
57. Rubin, D. B. (1976). Inference and missing data. Biometrika, 63, 581–592. 10.1093/biomet/63.3.581
58. Rubin, D. B. (1987). Multiple imputation for nonresponse in surveys. Hoboken: Wiley
59. Schwarz, G. E. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461–464. 10.1214/aos/1176344136
60. Simons, D. J., Shoda, Y., & Lindsay, S. D. (2017). Constraints on generality (COG): A proposed addition to all empirical papers. Perspectives on Psychological Science, 12, 1123–1128. 10.1177/1745691617708630
61. Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2015). Specification curve: Descriptive and inferential statistics on all reasonable specifications. SSRN Electronic Journal, 1–18. 10.2139/ssrn.2694998
62. Singer, J. D., & Willett, J. B. (2003). Applied longitudinal data analysis: Modeling change and event occurrence. New York: Oxford University Press. 10.1093/acprof:oso/9780195152968.003.0003
63. Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11, 702–712. 10.1177/1745691616658637
64. Steinhauer, H. W., Aßmann, C., Zinn, S., Goßmann, S., & Rässler, S. (2015). Sampling and weighting cohort samples in institutional contexts. AStA Wirtschafts- und Sozialstatistisches Archiv, 9, 131–157. 10.1007/s11943-015-0162-0
65. Steinhauer, H. W., & Zinn, S. (2016). NEPS Technical Report for Weighting: Weighting the sample of Starting Cohort 4 of the National Educational Panel Study (Waves 1 to 6) (NEPS Survey Paper No. 2). Bamberg: Leibniz Institute for Educational Trajectories, National Educational Panel Study
66. Trautwein, U., Marsh, H. W., Nagengast, B., Lüdtke, O., Nagy, G., & Jonkmann, K. (2012). Probing for the multiplicative term in modern expectancy–value theory: A latent interaction modeling study. Journal of Educational Psychology, 104, 763–777. 10.1037/a0027470
67. Valliant, R., Dever, J. A., & Kreuter, F. (2013). Practical tools for designing and weighting survey samples. New York: Springer
68. Van Buuren, S. (2011). Multiple imputation of multilevel data. In J. J. Hox & J. K. Roberts (Eds.), Handbook of advanced multilevel analysis (pp. 173–196). New York: Routledge
69. Van Buuren, S. (2012). Flexible imputation of missing data. Boca Raton: CRC Press
70. Van Buuren, S., & Groothuis-Oudshoorn, K. (2011). mice: Multivariate imputation by chained equations in R. Journal of Statistical Software, 45, 1–67. 10.18637/jss.v045.i03
71. Warm, T. A. (1989). Weighted likelihood estimation of ability in item response theory. Psychometrika, 54, 427–450. 10.1007/BF02294627
72. Weinert, S., Artelt, C., Prenzel, M., Senkbeil, M., Ehmke, T., & Carstensen, C. H. (2011). Development of competencies across the life span. Zeitschrift für Erziehungswissenschaft, 14, 67–86
73. Wilhelm, O. (2005). Measuring reasoning ability. In O. Wilhelm & R. W. Engle (Eds.), Handbook of understanding and measuring intelligence (pp. 373–392). Thousand Oaks: Sage
74. Wooldridge, J. M. (2007). Inverse probability weighted estimation for general missing data problems. Journal of Econometrics, 141, 1281–1301. 10.1016/j.jeconom.2007.02.002
75. Wu, M. C., & Carroll, R. J. (1988). Estimation and comparison of changes in the presence of informative right censoring by modeling the censoring process. Biometrics, 44, 175–188. 10.2307/2531905