Consequences of sequential sampling for meta-analysis

Behavior Research Methods

Volumn 46, Issue 4, 2014, Pages 1167-1183

  Braschi, Lorenzo ^a   Botella, Juan ^b   Suero, Manuel ^b  

^a UNIVERSIDAD ALFONSO X EL SABIO   (Spain)

^b UNIVERSIDAD AUTÓNOMA DE MADRID   (Spain)

Author keywords

CLAST rule; Meta analysis; Sequential analysis; Sequential sampling

Indexed keywords

ARTICLE; EFFECT SIZE; HUMAN; META ANALYSIS; PUBLISHING; SAMPLE SIZE; SEQUENTIAL ANALYSIS; COMPUTER SIMULATION; CONFIDENCE INTERVAL; EVALUATION STUDY; META ANALYSIS (TOPIC); METHODOLOGY; PSYCHOLOGICAL MODEL; STATISTICAL ANALYSIS; STATISTICAL BIAS;

BIAS (EPIDEMIOLOGY); COMPUTER SIMULATION; CONFIDENCE INTERVALS; DATA INTERPRETATION, STATISTICAL; META-ANALYSIS AS TOPIC; MODELS, PSYCHOLOGICAL; RESEARCH DESIGN; SAMPLE SIZE;

EID: 84942117436     PISSN: 1554351X     EISSN: 15543528     Source Type: Journal    
DOI: 10.3758/s13428-013-0433-z     Document Type: Article

References (57)

1. Abelson, R. P. (1997). A retrospective on the significance test ban of 1999 (if there were no significance tests, they would be invented). In L. L. Harlow, S. A. Mulaik, & J. H. Steiger (Eds.), What if there were no significance tests? (pp. 117–141). Mahwah, NJ: Erlbaum.
2. Allison, D. B., Silverstein, J. M., & Gorman, B. S. (1996). Power, sample size estimation, and early stopping rules. In R. D. Franklin, D. B. Allison, & B. S. Gorman (Eds.), Design and analysis of single case research (pp. 335–371). Mahwah, NJ: Erlbaum.
3. Bassler, D., Briel, M., Montori, V. M., Lane, M., Glasziou, P., Zhou, Q., & Ramsay, T. (2010). Stopping randomized trials early for benefit and estimation of treatment effects: Systematic review and meta-regression analysis. JAMA, 303, 1180–1187. doi:10.1001/jama.2010.310 DOI: 10.1001/jama.2010.310
4. Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Chichester, UK: Wiley. DOI: 10.1002/9780470743386
5. Botella, J., Suero, M., & Gambara, H. (2010). Psychometric inferences from meta-analysis of reliability and internal consistency coefficients. Psychological Methods, 15, 386–397. doi:10.1037/a0019626 DOI: 10.1037/a0019626
6. Botella, J., Ximénez, C., Revuelta, J., & Suero, M. (2006). Optimization of sample size in controlled experiments: The CLAST rule. Behavior Research Methods, 38, 65–76. doi:10.3758/BF03192751 DOI: 10.3758/BF03192751
7. Burton, A., Altman, D. G., Royston, P., & Holder, R. L. (2006). The design of simulation studies in medical statistics. Statistics in Medicine, 25, 4279–4292. DOI: 10.1002/sim.2673
8. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
9. Cohen, J. (1992a). A power primer. Psychological Bulletin, 112, 155–159. doi:10.1037/0033-2909.112.1.155 DOI: 10.1037/0033-2909.112.1.155
10. Cohen, J. (1992b). Statistical power analysis. Current Directions in Psychological Science, 1, 98–101. doi:10.1111/1467-8721.ep10768783 DOI: 10.1111/1467-8721.ep10768783
11. Cooper, H., Hedges, L. V., & Valentine, J. C. (Eds.). (2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York, NY: Russell Sage Foundation.
12. Cowles, M., & Davis, C. (1982). On the origins of the.05 level of statistical significance. American Psychologist, 37, 553–558. DOI: 10.1037/0003-066X.37.5.553
13. Cumming, G. (2012). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. New York, NY: Routledge, Taylor & Francis.
14. Duval, S., & Tweedie, R. (2000). Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 455–463. DOI: 10.1111/j.0006-341X.2000.00455.x
15. Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple graphical test. British Medical Journal, 315, 629–634. DOI: 10.1136/bmj.315.7109.629
16. Fitts, D. A. (2010). Improved stopping rules for the design of efficient small-sample experiments in biomedical and biobehavioral research. Behavior Research Methods, 42, 3–22. doi:10.3758/BRM.42.1.3 DOI: 10.3758/BRM.42.1.3
17. Fitts, D. A. (2011). Minimizing animal numbers: The variable-criteria sequential stopping rule. Comparative Medicine, 61, 206–218.
18. Francis, G. (2012a). The psychology of replication and replication in psychology. Perspectives on Psychological Science, 7, 585–594. doi:10.1177/1745691612459520 DOI: 10.1177/1745691612459520
19. Francis, G. (2012b). Publication bias and the failure of replication in experimental psychology. Psychonomic Bulletin & Review, 19, 975–991. doi:10.3758/s13423-012-0322-y DOI: 10.3758/s13423-012-0322-y
20. Freidlin, B., & Korn, E. L. (2009). Stopping clinical trials early for benefit: Impact on estimation. Clinical Trials, 6, 119–125. DOI: 10.1177/1740774509102310
21. Frick, R. W. (1998). A better stopping rule for conventional statistical tests. Behavior Research Methods, Instruments, & Computers, 30, 690–697. doi:10.3758/BF03209488 DOI: 10.3758/BF03209488
22. Ghosh, B. K., & Sen, P. K. (Eds.). (1991). Handbook of sequential analysis. New York, NY: Dekker.
23. Gigerenzer, G. (1993). The superego, the ego, and the statistical reasoning. In G. Keren & C. Lewis (Eds.), A handbook for data analysis in the behavioral sciences: Methodological issues (pp. 311–339). Hillsdale, NJ: Erlbaum.
24. Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3–8. DOI: 10.3102/0013189X005010003
25. Harlow, L. L., Mulaik, S. A., & Steiger, J. H. (1997). What if there were no significance tests? Hillsdale, NJ: Erlbaum.
26. Hays, W. J. (1988). Statistics (4th ed.). San Diego, CA: Harcourt Brace Jovanovich.
27. Hedges, L. V. (1981). Distribution theory for Glass’s estimator of effect size and related estimators. Journal of Educational Statistics, 6, 107–128. DOI: 10.2307/1164588
28. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
29. Howard, G. S., Maxwell, S. E., & Fleming, K. J. (2000). The proof of the pudding: An illustration of the relative strengths of null hypothesis, meta-analysis, and Bayesian analysis. Psychological Methods, 3, 315–332. DOI: 10.1037/1082-989X.5.3.315
30. Hunter, J. E. (1997). Needed: A ban on the significance test. Psychological Science, 8, 3–7. DOI: 10.1111/j.1467-9280.1997.tb00534.x
31. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524–532. doi:10.1177/0956797611430953 DOI: 10.1177/0956797611430953
32. Kelley, K., & Preacher, K. J. (2012). On effect size. Psychological Methods, 17, 137–152. DOI: 10.1037/a0028086
33. Konstantopoulos, S., & Hedges, L. V. (2009). Analyzing effect sizes: Fixed-effects models. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis (2nd ed., pp. 279–293). New York, NY: Russell Sage Foundation.
34. Lachin, J. M. (1981). Introduction to sample size determination and power analysis for clinical trials. Controlled Clinical Trials, 2, 93–113. DOI: 10.1016/0197-2456(81)90001-5
35. Lai, T. L. (2001). Sequential analysis: Some classical problems and new challenges. Statistica Sinica, 11, 303–408.
36. Lan, K. K. G., & DeMets, D. L. (1983). Discrete sequential boundaries for clinical trials. Biometrika, 70, 659–663. DOI: 10.2307/2336502
37. Lenth, R. V. (2001). Some practical guidelines for effective sample size determination. American Statistician, 55, 187–193. DOI: 10.1198/000313001317098149
38. Morris, S. B. (2000). Distribution of the standardized mean change effect size for meta-analysis on repeated measures. British Journal of Mathematical and Statistical Psychology, 53, 17–29. DOI: 10.1348/000711000159150
39. Nickerson, R. S. (2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5, 241–301. doi:10.1037/1082-989X.5.2.241 DOI: 10.1037/1082-989X.5.2.241
40. Oakes, M. (1986). Statistical inference: A commentary for the social and behavioral sciences. New York, NY: Wiley.
41. O’Brien, P. C., & Fleming, T. R. (1979). A multiple testing procedure for clinical trials. Biometrics, 35, 549–556. DOI: 10.2307/2530245
42. Orwin, R. G. (1983). A fail-safe N for effect size in meta-analysis. Journal of Educational Statistics, 8, 157–159. DOI: 10.2307/1164923
43. Raudenbush, S. W. (2009). Analyzing effect sizes: random-effects models. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis (2nd ed., pp. 295–315). New York, NY: Russell Sage Foundation.
44. R Development Core Team. (2008). R: A language and environment for statistical computing (Version 2.14). Vienna, Austria: R Foundation for Statistical Computing. Retrieved from www.R-project.org.
45. Rosenberg, M. S. (2005). The file-drawer problem revisited: A general weighted method for calculating fail-safe numbers in Meta-analysis. Evolution, 59, 464–468. DOI: 10.1111/j.0014-3820.2005.tb01004.x
46. Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638–641. doi:10.1037/0033-2909.86.3.638 DOI: 10.1037/0033-2909.86.3.638
47. Rothstein, H. R., Sutton, A. J., & Borenstein, M. (Eds.). (2005). Publication bias in meta-analysis: Prevention, assessment, and adjustments. New York, NY: Wiley.
48. Shadish, W. R., & Haddock, C. K. (2009). Combining estimates of effect size. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis (2nd ed., pp. 257–277). New York, NY: Russell Sage Foundation.
49. Shrout, P. E. (1997). Should significance tests be banned? Psychological Science, 8, 1–2. DOI: 10.1111/j.1467-9280.1997.tb00533.x
50. Siegmund, D. (1985). Sequential analysis: Tests and confidence intervals. New York, NY: Springer. DOI: 10.1007/978-1-4757-1862-1
51. Siegmund, D. (1994). A retrospective of Wald’s sequential analysis: Its relation to challenge-point detection and sequential clinical trials. In S. S. Gupta & J. O. Berger (Eds.), Statistical decision theory and related topics (pp. 9–33). New York, NY: Springer.
52. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. doi:10.1177/0956797611417632 DOI: 10.1177/0956797611417632
53. Tryon, W. W. (2001). Evaluating statistical difference, equivalence, and indeterminacy using inferential confidence intervals: An integrated alternative method of conducting null hypothesis statistical tests. Psychological Methods, 6, 371–386. doi:10.1037/1082-989X.6.4.371 DOI: 10.1037/1082-989X.6.4.371
54. Wald, W. (1947). Sequential analysis. New York, NY: Dover.
55. Wetherill, G. B., & Glazebrook, K. D. (1986). Sequential methods in statistics. London, UK: Chapman & Hall.
56. Whitehead, A. (1997). The design and analysis of sequential clinical trials (Rev (2nd ed.). Chichester, UK: Wiley.
57. Ximénez, C., & Revuelta, J. (2007). Extending the CLAST sequential rule to one-way ANOVA under group sampling. Behavior Research Methods, 39, 86–100. doi:10.3758/BF03192847 DOI: 10.3758/BF03192847