An approach for addressing the multiple testing problem in social policy impact evaluations

Evaluation Review

Volumn 33, Issue 6, 2009, Pages 539-567

  Schochet, Peter Z ^a  

^a MATHEMATICA POLICY RESEARCH   (United States)

Author keywords

Design of experiments; Impact evaluations; Multiple comparisons

Indexed keywords

ANALYSIS OF VARIANCE; ARTICLE; EPIDEMIOLOGY; EVALUATION; HUMAN; METHODOLOGY; POLICY; REGRESSION ANALYSIS; STATISTICAL ANALYSIS;

ANALYSIS OF VARIANCE; DATA INTERPRETATION, STATISTICAL; EVALUATION STUDIES AS TOPIC; HUMANS; PUBLIC POLICY; REGRESSION ANALYSIS; RESEARCH DESIGN; SELECTION BIAS;

EID: 70449629102     PISSN: 0193841X     EISSN: 15523926     Source Type: Journal    
DOI: 10.1177/0193841X09350590     Document Type: Article

References (39)

1. Altman, D.G., K.F. Schulz, and D. Moher. 2001. The revised CONSORT statement for reporting randomized trials: Explanation and elaboration. Annals of Internal Medicine 134:663-94.
2. Bechhofer, R., and C. Dunnett. 1982. Multiple comparisons for orthogonal contrasts: Examples and tables. Technometrics 24:213-22.
3. Benjamini, Y., and Y. Hochberg. 1995. Controlling the false discovery rate: A new and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B 57:1289-300.
4. Benjamini, Y., and D. Yekutieli. 2001. The control of the false discovery rate in multiple testing under dependency. The Annals of Statistics 29:1165-88.
5. Bobko, P., P. Roth, and M. Buster. 2007. The usefulness of unit weights in creating composite scores. Organizational Research Methods 109:689-709.
6. Brookes, S.T., E. Whitley, T.J. Peters, P.A. Mulheran, M. Egger, and G. Smith. 2001. Subgroup analyses in randomized controlled trials: Quantifying the risks of false-positives and false-negatives. Health Technology Assessment 5:1-49.
7. Committee for Proprietary Medicinal Products (CPMP). 2002. Points to consider on multiplicity issues in clinical trials. London: The European Agency for the Evaluation of Medicinal Products (EMEA).
8. Cook, R., and V. Farewell. 1996. Multiplicity considerations in the design and analysis of clinical trials. Journal of the Royal Statistical Society, Series A 159:93-110.
9. Curran-Everett, D. 2000. Multiplicity comparisons: Philosophies and illustration. American Journal of Physiology 279:R1-8.
10. Dunnett, C.W. 1955. A multiple comparison procedure for comparing several treatments with a control. Journal of the American Statistical Association 50:1096-121.
11. Freeman, M.F., and J.W. Tukey. 1950. Transformations related to the angular and square root. Annals of Mathematical Statistics 21:607-11.
12. Gelman, A., and H. Stern. 2006. The difference between "significant" and "not significant" is not itself statistically significant. The American Statistician 60:328-31.
13. Gordon, A., G. Glazko, Z. Qiu, and A. Yakovlev. 2007. Control of the mean number of false discoveries, Bonferroni and stability of multiple testing. The Annals of Applied Statistics 1:179-90.
14. Gulliksen, H. 1950. Theory of mental health. New York: John Wiley & Sons.
15. Harris, R.J. 1975. A primer of multivariate statistics. New York: Academic Press, Inc.
16. Hochberg, Y. 1988. A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75:800-2.
17. Holm, S. 1979. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6:65-70.
18. Hsu, J.C. 1996. Multiple comparisons: Theory and methods. London: Chapman and Hall.
19. Kane, M., and S. Case. 2004. The reliability and validity of weighted composite scores. Applied Measurement in Education 17:221-40.
20. Kirk, M. 1994. Experimental design: Procedures for the behavioral sciences. Pacific Grove, CA: Brooks/Cole.
21. Kramer, C.Y. 1956. Extension of the multiple range test to group means with unequal numbers of replications. Biometrics 12:307-10.
22. Landis, R., D. Beal, and P. Tesluk. 2000. A comparison of approaches to forming composite measures in structural equation models. Organizational Research Methods 3:186-207.
23. Lang, T., and M. Secic. 2007. How to report statistics in medicine. 2nd ed. Philadelphia, PA: American College of Physicians.
24. Lord, F.M. 1980. Applications of item response theory to practical testing problems. New Jersey: Lawrence Erlbaum Associates, Inc.
25. Raju, N., R. Bilgic, J. Edwards, and P. Fleer. 1997. Methodology review: Estimation of population and cross validity and the use of equal weights in prediction. Applied Psychological Measurement 21:291-305.
26. Rom, D.M. 1990. A sequentially rejective test procedure based on a modified Bonferroni inequality. Biometrika 77:663-5.
27. Rothwell, P.M. 2005. Subgroup analyses in randomized controlled trials: Importance, indications, and interpretation. The Lancet 365:176-86.
28. Saville, D.J. 1990. Multiple comparison procedures: The practical solution. The American Statistician 44:174-80.
29. Scheffé, H. 1959. The analysis of variance. New York: John Wiley & Sons.
30. Schochet, P. 2008. Guidelines for multiple testing in impact evaluations of educational interventions. Technical Methods Report, The U.S. Department of Education, Institute of Education Sciences.
31. Storey, J.D. 2002. A direct approach to false discovery rates. Journal of the Royal Statistical Society Series B 64:479-98.
32. Tukey, J.W. 1953. The problem of multiple comparisons. In Mimeographed notes. Princeton, NJ: Princeton University.
33. Wainer, H., and D. Thissen. 1993. Combining multiple-choice and constructed-response test scores: Toward a Marxist theory of test construction. Applied Measurement in Education 6:103-18.
34. Wang, M., and J. Stanley. 1970. Differential weighting: A review of methods and empirical studies. Review of Educational Research 40:663-705.
35. Westfall, P.H., Y. Lin, and S. Young. 1990. Resampling-based multiple testing. Proceedings of the fifteenth annual SAS users group international, 1359-64. Cary, NC: SAS Institute, Inc.
36. Westfall, P.H., R. Tobias, D. Rom, R. Wolfinger, and Y. Hochberg. 1999. Multiple comparisons and multiple tests using SAS. Cary, NC: SAS Institute, Inc.
37. Westfall, P.H., and R.D. Wolfinger. 1997. Multiple tests with discrete distributions. The American Statistician 51:3-8.
38. Westfall, P.H., and S.S. Young. 1993. Resampling-based multiple testing: Examples and methods for p-value adjustment. New York: John Wiley & Sons.
39. Wilks, S. 1938. Weighting systems for linear functions of correlated variables when there is no dependent variable. Psychometrika 3:23-40.