An inconvenient dataset: Bias and inappropriate inference with the multilevel model

Quality and Quantity

Volumn 48, Issue 3, 2014, Pages 1619-1649

  Lucas, Samuel R ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

FMP sample; Multilevel modeling; Non probability sample; Probability sample; Sample design

Indexed keywords

EID: 84897412183     PISSN: 00335177     EISSN: 15737845     Source Type: Journal    
DOI: 10.1007/s11135-013-9865-x     Document Type: Article

References (83)

1. Aitkin, M., Longford, N.: Statistical modeling issues in school effectiveness studies. J. R. Stat. Soc. Ser. A 149, 1-43 (1986).
2. Allison, P. D.: Discrete-time methods for the analysis of event histories. Sociol. Methodol. 13, 61-98 (1982).
3. Arum, R.: Judging School Discipline: The Crisis of Moral Authority. Harvard University Press, Cambridge (2003).
4. Banyard, V. L., Williams, L. M., Siegel, J. A.: The long-term mental health consequences of child sexual abuse: an exploratory study of the impact of multiple traumas in a sample of women. J. Trauma. Stress 14, 697-715 (2001).
5. Başoǧlu, M., Şalcioǧlu, E., Livanou, M.: Traumatic stress responses in earthquake survivors in Turkey. J. Traum. Stress 15, 269-276 (2002).
6. Beise, J., Voland, E.: A multilevel event history analysis of the effects of grandmothers on child mortality in a historical German population, Krummhörn, Ostfriesland, 1720-1874. Demogr. Res. 7, 469-498 (2002).
7. Berk, R. A.: An introduction to sample selection bias in sociological data. Am. Sociol. Rev. 48, 386-398 (1983).
8. Berkman, M. B., Plutzer, E.: Scientific expertise and the culture war: public opinion and the teaching of evolution in the American states. Perspect. Politics 7, 485-499 (2009).
9. Bickel, R.: Multilevel Analysis for Applied Research: It's Just Regression!. The Guilford Press, New York (2007).
10. Bluthenthal, R. N., Kral, A. H., Erringer, E. A., Edlin, B. R.: Use of an illegal syringe exchange and injection-related risk behaviors among street-recruited injection drug users in Oakland, California, 1992 to 1995. J. Acquir. Immune Defic. Syndr. Hum. Retrovirology 18, 505-511 (1998).
11. Boomsma, A.: The robustness of maximum likelihood estimation in structural equation models. In: Cuttance, P., Ecob, R. (eds.) Structural Modeling by Example: Applications in Educational, Sociological, and Behavioral Research, pp. 160-188. Cambridge University Press, New York (1987).
12. Braun, H. I., Jones, D. H., Rubin, D. B., Thayer, D. T.: Empirical Bayes Estimation of coefficients in the general linear model from data of deficient rank. Psychometrika 489(2), 171-181 (1983).
13. Breen, R.: Regression Models: Censored, Sample Selected, or Truncated Data. Sage Publications, Newbury Park (1996).
14. Bryk, A. S., Raudenbush, S.: Toward a more appropriate conceptualization of research on school effects: a three-level hierarchical linear model. Am. J. Educ. 97, 65-108 (1988).
15. Bryk, A. S., Raudenbush, S.: Hierarchical Linear Models: Applications and Data Analysis Methods. Sage Publications, Thousand Oaks (1992).
16. Centers for Disease Control: Pneumocystis pneumonia-Los Angeles. Morb. Mortal. Week. Rep. 30, 250-252 (1981).
17. Clogg, C. C., Eliason, S. R.: Some common problems in log-linear analysis. Sociol. Methods Res. 16, 8-44 (1987).
18. de Leeuw, J., Kreft, I. G. G.: Questioning multilevel models. J. Educ. Behav. Stat. 20, 171-189 (1995).
19. Dempster, A. P., Laird, N. M., Rubin, D. B.: Maximum likelihood from incomplete data via the EM algorithm. J. R. Stat. Soc. Ser. B (Methodol.) 39, 1-38 (1977).
20. Engram, E.: Science, Myth, and Reality: The Black Family in One-Half Century of Research. Greenwood Press, Westport (1982).
21. Flaherty, J., Brown, R. B.: A multilevel systemic model of community attachment: assessing the relative importance of the community and individual levels. Am. J. Sociol. 116, 503-542 (2010).
22. Freedman, D., Thornton, A., Camburn, D., Alwin, D., Young-DeMarco, L.: The life history calendar: a technique for collecting retrospective data. Sociol. Methodol. 18, 37-68 (1988).
23. Froot, K. A.: Consistent covariance matrix estimation with cross-sectional dependence and heteroskedasticity in financial data. J. Financ. Quant. Anal. 24, 333-335 (1989).
24. Fuchs, I. H.: Prospects and possibilities of the digital age. Proc. Am. Philos. Soc. 145, 45-53 (2001).
25. Fuller, S.: Job mobility and wage trajectories for men and women in the United States. Am. Sociol. Rev. 73, 158-183 (2008).
26. Gamoran, A.: The variable effects of high school tracking. Am. Sociol. Rev. 57, 812-828 (1992).
27. Gelman, A., Hill, J.: Data Analysis Using Regression and Multilevel/Hierarchical Models. Cambridge University Press, New York (2007).
28. Goldstein, H.: Multilevel Statistical Models. 3rd edn. Arnold, London (2003).
29. Goldstein, H., Healy, M. J. R., Rasbash, J.: Multilevel time series models with application to repeated measures data. Stat. Med. 13, 1643-1655 (1994).
30. Hank, K., Kreyenfeld, M.: A multilevel analysis of child care and women's fertility decisions in western Germany. J. Marriage Fam. 65, 584-596 (2003).
31. Harding, D. J.: Cultural context, sexual behavior, and romantic relationships in disadvantaged neighborhoods. Am. Sociol. Rev. 72, 341-364 (2007).
32. Harding, D. J.: Neighborhood violence and adolescent friendships. Int. J. Confl. Violence 2(1), 28-55 (2008).
33. Harding, D. J.: Collateral consequences of violence in disadvantaged neighborhoods. Soc. Forces 88, 757-782 (2009).
34. Harding, D. J.: Rethinking the cultural context of schooling decisions in disadvantaged neighborhoods: from deviant subculture to cultural heterogeneity. Sociol. Educ. 84, 322-339 (2011).
35. Hauser, R. M.: Context and consex: a cautionary tale. Am. J. Soc. 75, 645-664 (1969).
36. Hauser, R. M.: Contextual analysis revisited. Sociol. Methods Res. 2, 365-375 (1974).
37. Hausman, J. A., Wise, D. A.: Social experimentation, truncated distributions and efficient estimation. Econometrica 45, 919-939 (1977).
38. Heck, R. H., Thomas, S. L.: An Introduction to Multilevel Modeling Techniques, 2nd edn. Routledge, New York (2009).
39. Heck, R. H., Thomas, S. L., Tabata, L. N.: Multilevel and Longitudinal Modeling with IBM SPSS. Routledge, New York (2010).
40. Heckman, J. J.: Sample selection bias as a specification error. Econometrica 47, 153-162 (1979).
41. Hoerl, A. E., Kennard, R. W.: Ridge regression: biased estimation for nonorthogonal problems. Technometrics 12, 55-67 (1970).
42. Horney, J., Osgood, D. W., Marshall, I. H.: Criminal careers in the short-term: intra-individual variability in crime and its relation to local life circumstances. Am. Sociol. Rev. 60, 655-673 (1995).
43. Hox, J.: Applied Multilevel Analysis. TT-Publikaties, Amsterdam (1995).
44. Hox, J.: Multilevel Analysis: Techniques and Applications, 2nd edn. Routledge, New York (2010).
45. Hubbard, R.: The probable consequences of violating the normality assumption in parametric statistical analysis. Area 10, 393-398 (1978).
46. Ingels, S. J., Dowd, K. L., Baldridge, J. D., Stipe, J. L., Bartot, V. H., Frankel, M. R., Quinn, P.: National Education Longitudinal Study of 1988, Second Follow-up: Student Component Data File User's Manual. U. S. Department of Education, Washington (1994).
47. Kreft, I., De Leeuw, J.: Introducing Multilevel Modeling. Sage Publications, Thousand Oaks (1998).
48. Lee, V. E., Smith, J. B.: Effects of high school restructuring and size on early gains in achievement and engagement. Sociol. Educ. 68, 241-270 (1995).
49. Little, R. J. A., Rubin, D. B.: Statistical Analysis with Missing Data, 2nd edn. Wiley, Hoboken (2002).
50. Longford, N. T.: A fast-scoring algorithm for maximum likelihood estimation in unbalanced mixed models with nested random effects. Biometrika 74, 817-827 (1987).
51. Longford, N. T.: Random Coefficient Models. Oxford University Press, Oxford (1993).
52. Lucas, S. R.: Beyond the existence proof: ontological conditions, epistemological implications, and in-depth interview research. Qual. Quant. (2013a). doi: 10. 1007/s11135-012-9775-3.
53. Lucas, S. R.: Just Who Loses? Discrimination in the United States, vol. 2. Temple University Press, Philadelphia (2013b).
54. Lucas, S. R., Berends, M.: Race and track location in U. S. public schools. Res. Soc. Stratif. Mobil. 25, 169-187 (2007).
55. Lucas, S. R., Fucella, P. N., Berends, M.: A neo-classical education transitions approach: a corrected tale for three cohorts. Res. Soc. Stratif. Mobil. 29, 263-285 (2011).
56. Luke, D. A.: Multilevel Modeling. Sage Publications, Thousand Oaks (2004).
57. Manski, C. F.: Identification Problems in the Social Sciences. Harvard University Press, Cambridge (1995).
58. Mason, W. M., Wong, G. Y., Entwisle, B.: Contextual analysis through the multilevel linear model. Sociol. Methodol. 14, 72-103 (1983).
59. McCall, L.: Gender and the new inequality: explaining the college/non-college wage gap. Am. Sociol. Rev. 65, 234-255 (2000).
60. McCulloch, C. E., Searle, S. R., Neuhaus, J. M.: Generalized, Linear, and Mixed Models, 2nd edn. Wiley, Hoboken (2008).
61. Moerbeek M., van Breukelen G. J. P., Berger M. P. F.: Optimal designs for multilevel studies. In: de Leeuw, J., Meijer, E. (eds.) Handbook of Multilevel Analysis, pp. 177-205. Springer, New York (2008).
62. Morgan, S. P., Liao, T. F.: A cautionary note on the analysis of life-cycle events: comments on Smith and Meitz. J. Marriage Fam. 47, 233-236 (1985).
63. Pellegrino, J., Jones, L. R., Mitchell, K. J.: Grading the Nation's Report Card: Evaluating NAEP and Transforming the Assessment of Educational Progress. National Academy Press, Washington (1999).
64. Pinheiro, J. C., Bates, D. M.: Mixed-Effects Models in S and S-PLUS. Springer, New York (2004).
65. Rabe-Hesketh, S., Skrondal, A.: Multilevel and Longitudinal Modeling Using Stata, 2nd edn. Stata Press, College Station (2008).
66. Raudenbush, S. W., Bryk, A. S.: Hierarchical Linear Models: Applications and Data Analysis Methods, 2nd edn. Sage Publications, Thousand Oaks (2002).
67. Reise S. P., Duan N.: Design issues in multilevel studies. In: Reise, S. P., Duan, N. (eds.) Multilevel Modeling: Methodological Advances, Issues, and Applications, pp. 285-298. Lawrence Erlbaum Associates Publishers, Mahwah (2003).
68. Roscigno, V. J.: Race and the reproduction of educational disadvantage. Soc. Forces 76, 1033-1061 (1998).
69. Salganik, M. J., Heckathorn, D. D.: Sampling and estimation in hidden populations using respondent-driven sampling. Sociol. Methodol. 34, 193-239 (2004).
70. Schiller, K. S.: Effects of feeder patterns on students' transition to high school. Sociol. Educ. 72, 216-233 (1999).
71. Scott, L. A., Ingels, S. J., Pulliam, P., Sehra, S., Taylor, J. R., Jergovic, D.: National Education Longitudinal Study of 1988: High School Effectiveness Study, Data File User's Manual. (DRAFT) National Center for Education Statistics, U. S. Department of Education, Washington, DC (1996).
72. Snijders, T., Bosker, R.: Multilevel Analysis: An Introduction to Basic and Advanced Multilevel Modeling. Sage Publications, Thousand Oaks (1999).
73. Steele, F., Goldstein, H., Browne, W.: A general multistate competing risks model for event history data, with an application to a study of contraceptive use dynamics. J. Stat. Model. 4, 145-159 (2004).
74. Stolzenberg, R. M., Relles, D.: Theory testing in a world of constrained research design: the significance of Heckman's censored sampling bias correction for nonexperimental research. Sociol. Methods Res. 18, 395-415 (1990).
75. Stolzenberg, R. M., Relles, D.: Tools for intuition about sample selection bias and its correction. Am. Sociol. Rev. 62, 494-507 (1997).
76. Swaminathan, H., Rogers, H. J.: Estimation procedures for hierarchical linear models. In: O'Connell, A. A., McCoach, D. B. (eds.) Multilevel Modeling of Educational Data. Information Age Publishing Inc., Charlotte, pp. 469-519 (2008).
77. Tam, T.: Accounting for dynamic selection bias in educational transitions: the Cameron-Heckman latent class estimator and its generalizations. Res. Soc. Stratif Mobil. 29, 287-310 (2011).
78. Tobin, J.: Estimation of relationships for limited dependent variables. Econometrica 26, 24-36 (1958).
79. Turnbull, B. W.: The empirical distribution function with arbitrarily grouped, censored and truncated data. J. R. Stat. Soc. Ser. B (Methodol.) 38, 290-295 (1976).
80. Twisk, J. W. R.: Applied Multilevel Analysis: A Practical Guide for Medical Researchers. Cambridge University Press, New York (2006).
81. Winship, C., Mare, R. D.: Models for sample selection bias. Annu. Rev. Sociol. 18, 327-350 (1992).
82. Winship, C., Radbill, L.: Sampling weights and regression models. Sociol. Methods Res. 23, 230-257 (1994).
83. Yates, F.: A review of recent statistical developments in sampling and sampling surveys. J. R. Stat. Soc. 109, 12-43 (1946).