Avoiding 'data snooping' in multilevel and mixed effects models

Journal of the Royal Statistical Society. Series A: Statistics in Society

Volumn 170, Issue 4, 2007, Pages 1035-1059

  Afshartous, David ^a   Wolf, Michael ^b  

^a UNIVERSITY OF MIAMI   (United States)

^b UNIVERSITY OF ZURICH   (Switzerland)

Author keywords

Data snooping; Hierarchical linear models; Hypothesis testing; Pairwise comparisons; Random effects; Rankings

Indexed keywords

EID: 34848904577     PISSN: 09641998     EISSN: 1467985X     Source Type: Journal    
DOI: 10.1111/j.1467-985X.2007.00494.x     Document Type: Article

References (34)

1. Afshartous, D. de Leeuw, J. (2004) An application of multilevel model prediction to NELS:88. Behaviormetrika, 31, 43 66.
2. Benjamini, Y. Hochberg, Y. (1995) Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Statist. Soc. B, 57, 289 300.
3. Benjamini, Y. Yekutieli, D. (2001) The control of the false discovery rate in multiple testing under dependency. Ann. Statist., 29, 1165 1188.
4. Browne, W. (2003) MCMC Estimation in MLwiN (Version 2.0). London : Institute of Education.
5. Bryk, A. Raudenbush, S. (1992) Hierarchical Linear Models. Newbury Park : Sage.
6. Carpenter, J. R., Goldstein, H. Rasbash, J. (2003) A novel bootstrap procedure for assessing the relationship between class size and achievement. Appl. Statist., 52, 431 443.
7. Dempster, A. P., Laird, N. M. Rubin, D. B. (1977) Maximum likelihood from incomplete data via the EM algorithm (with discussion). J. R. Statist. Soc. B, 39, 1 38.
8. Efron, B. Tibshirani, R. J. (1993) An Introduction to the Bootstrap. New York : Chapman and Hall.
9. Goldstein, H. (2003) Multilevel Statistical Models, 3rd edn. London : Hodder Arnold.
10. Goldstein, H. Healy, M. J. R. (1995) The graphical presentation of a collection of means. J. R. Statist. Soc. A, 158, 175 177.
11. Goldstein, H., Rasbash, J., Yang, M., Woodhouse, G., Pan, H., Nuttall, D. Thomas, S. (1993) A multilevel analysis of school examination results. Oxf. Rev. Educ., 19, 425 433.
12. Goldstein, H. Spiegelhalter, D. J. (1996) League tables and their limitations: statistical issues in comparisons of institutional performance (with discussion). J. R. Statist. Soc. A, 159, 385 443.
13. Harville, D. (1976) Extension of the Gauss-Markov theorem to include estimation of random effects. Ann. Statist., 4, 384 395.
14. Holm, S. (1979) A simple sequentially rejective multiple test procedure. Scand. J. Statist., 6, 65 70.
15. James, W. Stein, C. (1961) Estimation with quadratic loss. In Proc. 4th Berkeley Symp. Mathematical Statistics and Probability, vol. 1, pp. 361 380. Berkeley : University of California Press.
16. Korn, E. L., Troendle, J. F., McShane, L. M. Simon, R. (2004) Controlling the number of false discoveries: application to high-dimensional genomic data. J. Statist. Planng Inf., 124, 379 398.
17. Laird, N. M. Ware, J. W. (1982) Random-effects models for longitudinal data. Biometrics, 38, 963 974.
18. van der Leeden, R., Meijer, E. Busing, F. M. T. A. (2007) Resampling multilevel models. In Handbook of Multilevel Analysis (eds J. de Leeuw E. Meijer). New York : Springer.
19. de Leeuw, J. Kreft, I. (1986) Random coefficient models for multilevel analysis. J. Educ. Statist., 11, 57 85.
20. de Leeuw, J. Kreft, I. (1995) Questioning multilevel models. J. Educ. Behav. Statist., 20, 171 189.
21. Lehmann, E. L. Romano, J. P. (2005a) Testing Statistical Hypotheses, 3rd edn. New York : Springer.
22. Lehmann, E. L. Romano, J. P. (2005b) Generalizations of the familywise error rate. Ann. Statist., 33, 1138 1154.
23. Longford, N. (1987) A fast scoring algorithm for maximum likelihood estimation in unbalanced mixed models with random effects. Biometrika, 74, 817 827.
24. National Center for Educational Statistics (2006) National educational longitudinal study of 1988. National Center for Educational Statistics, Washington DC. (Available from http://nces.ed.gov/surveys/nels88/.)
25. Pinheiro, J. Bates, D. (2000) Mixed Effects Models in S and S-Plus. New York : Springer.
26. Rasbash, J., Steele, F., Browne, W. Prosser, B. (2004) A User's Guide to MLwiN Version 2.0. London : Institute of Education. (Available from http://multilevel.ioe.ac.uk/download/userman20.pdf.)
27. Raudenbush, S. Bryk, A. (2002) Hierarchical Linear Models. Newbury Park : Sage.
28. Robinson, G. (1991) That BLUP is a good thing: the estimation of random effects. Statist. Sci., 6, 15 51.
29. Romano, J. P. Wolf, M. (2005) Stepwise multiple testing as formalized data snooping. Econometrica, 73, 1237 1282.
30. Romano, J. P. Wolf, M. (2007) Control of generalized error rates in multiple testing. Working Paper 245. Institut für Empirische Wirtschaftsforschung, University of Zurich, Zurich. (Available from http://www.iew.unizh.ch/wp/index.php.)
31. R Project (2006) R - Version 2.2.1. Vienna : R Project. (Available from http://www.r-project.org/.)
32. Searle, S. R., Casella, G. McCulloch, C. E. (1992) Variance Components. New York : Wiley.
33. Williams, V. S. L., Jones, L. V. Tukey, J. W. (1999) Controlling error in multiple comparisons, with examples from state-to-state differences in educational achievement. J. Educ. Behav. Statist., 24, 42 69.
34. Yekutieli, D. (2002) A false discovery rate procedure for pairwise comparisons. Technical Report. Department of Statistics and Operations Research, Tel Aviv University, Tel Aviv. (Available from http://www.math.tau.ac.il/ %7Eyekutiel/work.html.)