Character building or subversive consequences of employment during high school: Causal effects based on propensity score models for categorical treatments

Journal of Educational Psychology

Volumn 106, Issue 2, 2014, Pages 584-603

  Nagengast, Benjamin ^a   Chiorri, Carlo ^e   Marsh, Herbert W ^b,c,d   Hau, Kit Tai ^f  

^a UNIVERSITY OF TÜBINGEN   (Germany)

^b UNIVERSITY OF WESTERN SYDNEY   (Australia)

^c UNIVERSITY OF OXFORD   (United Kingdom)

^d KING SAUD UNIVERSITY   (Saudi Arabia)

^e UNIVERSITY OF GENOA   (Italy)

^f CHINESE UNIVERSITY OF HONG KONG   (Hong Kong)

Author keywords

Achievement; Educational longitudinal study of 2002; Marginal mean weighting through stratification; Occupational aspirations; Part time working

Indexed keywords

EID: 84925688382     PISSN: 00220663     EISSN: 19392176     Source Type: Journal    
DOI: 10.1037/a0035615     Document Type: Article

References (93)

1. Arnett, J. J. (2000). Emerging adulthood. A theory of development from the late teens through the twenties. American Psychologist, 55, 469-480. doi:10.1037/0003-066X.55.5.469
2. Bachman, J. G., & Schulenberg, J. (1993). How part-time work intensity relates to drug use, problem behavior, time use, and satisfaction among high school seniors: Are these consequences or merely correlates? Developmental Psychology, 29, 220-235. doi:10.1037/0012-1649.29.2.220
3. Bachman, J. G., Staff, J., O'Malley, P. M., Schulenberg, J. E., & Freedman-Doan, P. (2011). Twelfth-grade student work intensity linked to later educational attainment and substance use: New longitudinal evidence. Developmental Psychology, 47, 344-363. doi:10.1037/a0021027
4. Barling, J., Rogers, K.-A., & Kelloway, E. K. (1995). Some effects of teenagers' part-time employment: The quantity and quality of work make the difference. Journal of Organizational Behavior, 16, 143-154. doi:10.1002/job.4030160205
5. Brooks, L., Cornelius, A., Greenfield, E., & Joseph, R. (1995). The relation of career-related work or internship experiences to the career development of college seniors. Journal of Vocational Behavior, 46, 332-349. doi:10.1006/jvbe.1995.1024
6. Coleman, J. S. (1959). Academic achievement and the structure of competition. Harvard Educational Review, 29, 330-351.
7. Coleman, J. S. (1961). The adolescent society: The social life of the teenager and its impact on education. New York, NY: Free Press of Glencoe.
8. Cook, T. D., Steiner, P., & Pohl, S. (2009). How bias reduction is affected by covariate choice, unreliability, and mode of data analysis: Results from two types of within-study comparisons. Multivariate Behavioral Research, 44, 828-847. doi:10.1080/00273170903333673
9. Creed, P. A., Muller, J., & Patton, W. (2003). Leaving high school: The influence and consequences for psychological well-being and careerrelated confidence. Journal of Adolescence, 26, 295-311. doi:10.1016/S0140-1971(03)00015-0
10. Finn, J. D. (1989). Withdrawing from school. Review of Educational Research, 59, 117-142. doi:10.3102/00346543059002117
11. Gardiner, J. C., Luo, Z., & Roman, L. A. (2009). Fixed effects, random effects and GEE: What are the differences? Statistics and Medicine, 28, 221-239. doi:10.1002/sim.3478
12. Greenberger, E., & Steinberg, L. (1986). When teenagers work: The psychological and social costs of adolescent employment. New York, NY: Basic Books.
13. Hirano, K., & Imbens, G. W. (2004). The propensity score with continuous treatments. In A. Gelman & X.-L. Meng (Eds.), Applied Bayesian modeling and causal inference from incomplete-data perspectives: An essential journey with Donald Rubin's statistical family (pp. 73-84). doi:10.1002/0470090456.ch7
14. Ho, D. E., Imai, K., King, G., & Stuart, E. A. (2007). Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Political Analysis, 15, 199-236. doi:10.1093/pan/mpl013
15. Holland, A., & Andre, T. (1987). Participation in extracurricular activities in secondary school: What is known, what needs to be known? Review of Educational Research, 57, 437-466. doi:10.3102/00346543057004437
16. Hong, G. (2010). Marginal mean weighting through stratification: Adjustment for selection bias in multilevel data. Journal of Educational and Behavioral Statistics, 35, 499-531. doi:10.3102/1076998609359785
17. Hong, G. (2012). Marginal mean weighting through stratification: A generalized method for evaluating multivalued and multiple treatments with nonexperimental data. Psychological Methods, 17, 44-60. doi:10.1037/a0024918
18. Hong, G., Corter, C., Hong, Y., & Pelletier, J. (2012). Differential effects of literacy instruction time and homogeneous ability grouping in kindergarten classrooms: Who will benefit? Who will suffer? Educational Evaluation and Policy Analysis, 34, 69-88. doi:10.3102/0162373711424206
19. Howe, N., & Strauss, W. (2000). Millennials rising: The next great generation. New York, NY: Vintage.
20. Imai, K., & van Dyk, D. A. (2004). Causal inference with general treatment regimes. Journal of the American Statistical Association, 99, 854-866. doi:10.1198/016214504000001187
21. Imbens, G. W. (2000). The role of the propensity score in estimating dose-response functions. Biometrika, 87, 706-710. doi:10.1093/biomet/87.3.706
22. Ingels, S. J., Pratt, D. J., Rogers, J. E., Siegel, P. H., & Stutts, E. S. (2005). Education Longitudinal Study of 2002: Base-year to first follow-up data file documentation (Report No. NCES 2006-344). Washington, DC: U.S. Department of Education, National Center for Education Statistics.
23. Ingels, S. J., Pratt, D. J., Wilson, D., Burns, L. J., Currivan, D., Rogers, J. E., & Hubbard-Bednasz, S. (2007). Education Longitudinal Study of 2002 (ELS:2002): Base-year to second follow-up data file documentation (including field test report) (Report No. NCES 2008-347). Washington, DC: U.S. Department of Education, National Center for Education Statistics.
24. Joffe, M. M., & Rosenbaum, P. R. (1999). Invited commentary: Propensity scores. American Journal of Epidemiology, 150, 327-333. doi:10.1093/oxfordjournals.aje.a010011
25. Kang, J. D. L., & Schafer, J. L. (2007). Demystifying double robustness: A comparison of alternative strategies for estimating a population mean from incomplete data. Statistical Science, 22, 523-539. doi:10.1214/07-STS227
26. Kluve, J., Schneider, H., Uhlendorff, A., & Zhao, Z. (in press). Evaluating continuous training programmes by using the generalized propensity score. Journal of the Royal Statistical Society: Series A. Statistics in Society. doi:10.1111/j.1467-985X.2011.01000.x
27. Larson, R. W., & Verma, S. (1999). How children and adolescents spend time across the world: Work, play, and developmental opportunities. Psychological Bulletin, 125, 701-736. doi:10.1037/0033-2909.125.6.701
28. Lee, C., & Orazem, P. F. (2010). High school employment, school performance, and college entry. Economics of Education Review, 29, 29-39. doi:10.1016/j.econedurev.2009.03.004
29. Lee, J. C., & Staff, J. (2007). When work matters: The varying impact of work intensity on high school dropout. Sociology of Education, 80, 158-178. doi:10.1177/003804070708000204
30. Leventhal, T., Graber, J. A., & Brooks-Gunn, J. (2001). Adolescent transitions to young adulthood. Journal of Research on Adolescence, 11, 297-323. doi:10.1111/1532-7795.00014
31. Li, K.-H., Raghunathan, T. E., & Rubin, D. B. (1991). Large-sample significance levels from multiply imputed data using moment-based statistics and an F reference distribution. Journal of the American Statistical Association, 86, 1065-1073.
32. Lu, B., Zanutto, E., Hornik, R., & Rosenbaum, P. R. (2001). Matching with doses in an observational study of a media campaign against drug abuse. Journal of the American Statistical Association, 96, 1245-1253. doi:10.1198/016214501753381896
33. Lumley, T. (2004). Analysis of complex survey samples. Journal of Statistical Software, 9, 1-19.
34. Lumley, T. (2011). Survey: Analysis of complex survey samples (R Package Version 3.24-1) [Computer software]. Available at http://cran.rproject.org
35. MacCallum, R. C., Zhang, S., Preacher, K. J., & Rucker, D. D. (2002). On the practice of dichotomization of quantitative variables. Psychological Methods, 7, 19-40. doi:10.1037/1082-989X.7.1.19
36. Mael, F. A., Morath, R. A., & McLellan, J. A. (1997). Dimensions of adolescent employment. Career Development Quarterly, 45, 351-368. doi:10.1002/j.2161-0045.1997.tb00539.x
37. Marsh, H. W. (1991). Employment during high school: Character building or a subversion of academic goals? Sociology of Education, 64, 172-189. doi:10.2307/2112850
38. Marsh, H. W., & Hau, K.-T. (2007). Applications of latent-variable models in educational psychology: The need for methodological-substantive synergies. Contemporary Educational Psychology, 32, 151-170. doi:10.1016/j.cedpsych.2006.10.008
39. Marsh, H. W., & Kleitman, S. (2002). Extra-curricular school activities: The good, the bad, and the nonlinear. Harvard Educational Review, 72, 464-502.
40. Marsh, H. W., & Kleitman, S. (2005). Consequences of employment during high school: Character building, subversion of academic goals, or a threshold? American Educational Research Journal, 42, 331-369. doi:10.3102/00028312042002331
41. McCullagh, P. (1980). Regression models for ordinal data. Journal of the Royal Statistical Society: Series B, 42, 109-142.
42. Monahan, K. C., Lee, J. M., & Steinberg, L. (2011). Revisiting the impact of part-time work on adolescent adjustment: Distinguishing between selection and socialization using propensity score matching. Child Development, 82, 96-112. doi:10.1111/j.1467-8624.2010.01543.x
43. Morgan, S. L., & Winship, C. (2007). Counterfactuals and causal inference: Methods and principles for social research. doi:10.1017/CBO9780511804564
44. Mortimer, J. T. (2003). Working and growing up in America. Cambridge, MA: Harvard University Press.
45. Mortimer, J. T., & Finch, M. D. (1986). The effects of part-time work on self-concept and achievement. In K. Borman & J. Reisman (Eds.), Becoming a worker (pp. 66-89). Norwood, NJ:Ablex.
46. Mortimer, J. T., Finch, M. D., Ryu, S., Shanahan, M. J., & Call, K. T. (1996). The effects of work intensity on adolescent mental health, achievement, and behavioral adjustment: New evidence from a prospective study. Child Development, 67, 1243-1261.
47. Mortimer, J. T., Staff, J., & Oesterle, S. (2003). Adolescent work and the early socioeconomic career. In J. T. Mortimer & M. J. Shanahan (Eds.), Handbook of the life course (pp. 437-459). doi:10.1007/978-0-306-48247-2_20
48. Nakao, K., & Treas, J. (1992). The 1989 Socioeconomic Index of Occupations: Construction from the 1989 Occupational Prestige Scores (General Social Survey Methodological Report No. 74). Chicago, IL: National Opinion Research Center.
49. National Center for Education Statistics. (1986). High School and Beyond, 1980: Sophomore cohort second follow-up (1984)-Data file user's manual. Ann Arbor, MI: Inter-University Consortium for Political and Social Research.
50. National Center for Education Statistics. (1996). NELS:88 manual (NCES Technical Report 96-174). Washington, DC: Author.
51. Panel on Youth of the President's Science Advisory Committee. (1974). Youth: Transition to adulthood. Chicago, IL: University of Chicago Press.
52. Payne, J. (2003). The impact of part-time jobs in Years 12 and 13 on qualification achievement. British Educational Research Journal, 29, 599-611. doi:10.1080/01411920301844
53. Pohl, S., Steiner, P. M., Eisermann, J., Soellner, R., & Cook, T. D. (2009). Unbiased causal inference from an observational study: Results of a within-study comparison. Educational Evaluation and Policy Analysis, 31, 463-479. doi:10.3102/0162373709343964
54. Post, D., & Pong, S. (2000). Employment during middle school: The effects on academic achievement in the U.S. and abroad. Educational Evaluation and Policy Analysis, 22, 273-298. doi:10.3102/01623737022003273
55. Post. D., & Pong, S. (2009). Student labour and academic proficiency in international perspective. International Labour Review, 148, 93-122. doi:10.1111/j.1564-913X.2009.00050.x
56. Raghunathan, T., & Dong, Q. (2012). Analysis of variance from multiply imputed data sets. Retrieved from http://www-personal.umich.edu/~teraghu/Raghunathan-Dong.pdf
57. Revelle, W. (2013). psych: Procedures for personality and psychological research [Computer software]. Retrieved from http://CRAN.R-project.org/package=psychVersion=1.3.2
58. Robins, J. M. (1999). Marginal structural models versus structural nested models as tools for causal inference. In M. E. Halloran & D. Berry (Eds.), Statistical models in epidemiology, the environment, and clinical trials (pp. 95-134). New York, NY: Springer.
59. Rosenbaum, P. R. (1984). The consequences of adjustment for a concomitant variable that has been affected by the treatment. Journal of the Royal Statistical Society: Series A, 147, 656-666. doi:10.2307/2981697
60. Rosenbaum, P. R. (1987). Model-based direct adjustment. Journal of the American Statistical Association, 82, 387-394. doi:10.1080/01621459.1987.10478441
61. Rosenbaum, P. R. (2002). Covariance adjustment in randomized experiments and observational studies. Statistical Science, 17, 286-327. doi:10.1214/ss/1042727942
62. Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41-55. doi:10.1093/biomet/70.1.41
63. Rosenbaum, P. R., & Rubin, D. B. (1984). Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association, 79, 516-524. doi:10.1080/01621459.1984.10478078
64. Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66, 688-701. doi:10.1037/h0037350
65. Rubin, D. B. (1977). Assignment to treatment group on the basis of a covariate. Journal of Educational Statistics, 2, 1-26. doi:10.2307/1164933
66. Rubin, D. B. (1978). Bayesian inference for causal effects: The role of randomization. Annals of Statistics, 6, 34-58. doi:10.1214/aos/1176344064
67. Rubin, D. B. (2001). Using propensity scores to help design observational studies: Application to the tobacco litigation. Health Services & Outcomes Research Methodology, 2, 169-188. doi:10.1023/A:1020363010465
68. Rubin, D. B. (2005). Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100, 322-331. doi:10.1198/016214504000001880
69. Rubin, D. B., & Thomas, N. (2000). Combining propensity score matching with additional adjustments for prognostic covariates. Journal of the American Statistical Association, 95, 573-585. doi:10.1080/01621459.2000.10474233
70. Sabia, J. J. (2009). School-year employment and academic performance of young adolescents. Economics of Education Review, 28, 268-276. doi:10.1016/j.econedurev.2008.05.001
71. Schafer, J. L., & Kang, J. D. Y. (2008). Average causal effects from nonrandomized studies: A practical guide and simulated example. Psychological Methods, 13, 279-313. doi:10.1037/a0014268
72. Schoenhals, M., Tienda, M., & Schneider, B. (1998). The educational and personal consequences of adolescent employment. Social Forces, 77, 723-761. doi:10.1093/sf/77.2.723
73. Shadish, W. R., Clark, M. H., & Steiner, P. M. (2008). Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random and nonrandom assignments. Journal of the American Statistical Association, 103, 1334-1344. doi:10.1198/016214508000000733
74. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton-Mifflin.
75. Shanahan, M. J., & Flaherty, B. P. (2001). Dynamic patterns of time use in adolescence. Child Development, 72, 385-401. doi:10.1111/1467-8624.00285
76. Singh, K., Chang, M., & Dika, S. (2007). Effects of part-time work on school achievement during high school. Journal of Educational Research, 101, 12-23. doi:10.3200/JOER.101.1.12-23
77. Spreeuwenberg, M. D., Bartak, A., Croon, M. A., Hagenaars, J. A., Busschbach, J. J. V., Andrea, H.,... Stijnen, T. (2010). The multiple propensity score as control for bias in the comparison of more than two treatment arms: An introduction from a case study in mental health. Medical Care, 48, 166-174. doi:10.1097/MLR.0b013e3181c1328f
78. Staff, J., Osgood, D. W., Schulenberg, J. E., Bachman, J. G., & Messersmith, E. E. (2010). Explaining the relationship between employment and juvenile delinquency. Criminology, 48, 1101-1131. doi:10.1111/j.1745-9125.2010.00213.x
79. Staff, J., Schulenberg, J. E., & Bachman, J. G. (2010). Adolescent work intensity, school performance, and academic engagement. Sociology of Education, 83, 183-200. doi:10.1177/0038040710374585
80. Steinberg, L., & Dornbusch, S. M. (1991). Negative correlates of part-time employment during adolescence: Replication and elaboration. Developmental Psychology, 27, 304-313. doi:10.1037/0012-1649.27.2.304
81. Stern, D., Stone, J. R., Hopkins, C., & McMillion, M. (1990). Quality of students' work experience and orientation toward work. Youth & Society, 22, 263-282. doi:10.1177/0044118X90022002008
82. Steyer, R., von Davier, A. A., Nachtigall, C., & Buhl, T. (2000). Causal regression models I: Individual and average causal effects. Methods of Psychological Research, 5, 39-71.
83. Stuart, E. A. (2010). Matching methods for causal inference. A review and a look forward. Statistical Science, 25, 1-21. doi:10.1214/09-STS313
84. Subramanian, S. V., & O'Malley, A. J. (2010). Modeling neighborhood effects: The futility of comparing mixed and marginal approaches. Epidemiology, 21, 475-478. doi:10.1097/EDE.0b013e3181d74a71
85. Thoemmes, F. J., & Kim, E. S. (2011). A systematic review of propensity score methods in the social sciences. Multivariate Behavioral Research, 46, 90-118. doi:10.1080/00273171.2011.540475
86. VanderWeele, T. J., & Onyebuchi, A. A. (2011). Unmeasured confounders for general outcomes, treatments and confounders: Bias formulas for sensitivity analysis. Epidemiology, 22, 42-52. doi:10.1097/EDE.0b013e3181f74493
87. Venables, W. N., & Ripley, B. D. (2002). Modern applied statistics with S (4th ed.). doi:10.1007/978-0-387-21706-2
88. Warren, J. R. (2002). Reconsidering the relationship between student employment and academic outcomes: A new theory and better data. Youth & Society, 33, 366-393. doi:10.1177/0044118X02033003002
89. Warren, J. R., & Lee, J. C. (2003). The impact of adolescent employment on high school dropout: Differences by individual and labormarket characteristics. Social Science Research, 32, 98-128. doi:10.1016/S0049089X(02)00021-2
90. Warren, J. R., LePore, P. C., & Mare, R. D. (2000). Employment during high school: Consequences for students' grades in academic courses. American Educational Research Journal, 37, 943-969. doi:10.3102/00028312037004943
91. Zanutto, E. L. (2006). A comparison of propensity score and linear regression analysis of complex survey data. Journal of Data Science, 4, 67-91.
92. Zanutto, E. L., Lu, B., & Hornik, R. (2005). Using propensity score subclassification for multiple treatment doses to evaluate a national antidrug media campaign. Journal of Educational and Behavioral Statistics, 30, 59-73. doi:10.3102/10769986030001059
93. Zimmer-Gembeck, M. J., & Mortimer, J. T. (2006). Adolescent work, vocational development, and education. Review of Educational Research, 76, 537-566. doi:10.3102/00346543076004537