Statistical conclusion validity: Some common threats and simple remedies

Frontiers in Psychology

Volumn 3, Issue AUG, 2012, Pages

  García Pérez, Miguel A ^a  

^a UNIVERSIDAD COMPLUTENSE DE MADRID   (Spain)

Author keywords

Data analysis; Preliminary tests; Regression; Stopping rules; Validity of research

Indexed keywords

EID: 84867037235     PISSN: None     EISSN: 16641078     Source Type: Journal    
DOI: 10.3389/fpsyg.2012.00325     Document Type: Article

References (130)

1. Agresti, A., and Min, Y. (2005). Frequentist performance of Bayesian confidence intervals for comparing proportions in 2 × 2 contingency tables. Biometrics 61, 515-523.
2. Ahn, C., Overall, J. E., and Tonidandel, S. (2001). Sample size and power calculations in repeated measurement analysis. Comput. Methods Programs Biomed. 64, 121-124.
3. Aickin, M. (2011). Test ban: policy of the Journal of Alternative and Complementary Medicine with regard to an increasingly common statistical error. J. Altern. Complement. Med. 17, 1093-1094.
4. Aiken, L. S., West, S. G., and Millsap, R. E. (2008). Doctoral training in statistics, measurement, and methodology in psychology: replication and extension of Aiken, West, Sechrest, and Reno's (1990) survey of PhD programs in North America. Am. Psychol. 63, 32-50.
5. Aiken, L. S., West, S. G., Sechrest, L., and Reno, R. R. (1990). Graduate training in statistics, methodology, and measurement in psychology: a survey of PhD programs in North America. Am. Psychol. 45, 721-734.
6. Albers, W., Boon, P. C., and Kallenberg, W. C. M. (2000). The asymptotic behavior of tests for normal means based on a variance pre-test. J. Stat. Plan. Inference 88, 47-57.
7. Alcalá-Quintana, R., and García-Pérez, M. A. (2004). The role of parametric assumptions in adaptive Bayesian estimation. Psychol. Methods 9, 250-271.
8. Alcalá-Quintana, R., and García-érez, M. A. (2005). Stopping rules in Bayesian adaptive threshold estimation. Spat. Vis. 18, 347-374.
9. Anscombe, F. J. (1953). Sequential estimation. J. R. Stat. Soc. Series B 15, 1-29.
10. Anscombe, F. J. (1954). Fixedsample-size analysis of sequential observations. Biometrics 10, 89-100.
11. Armitage, P., McPherson, C. K., and Rowe, B. C. (1969). Repeated significance tests on accumulating data. J. R. Stat. Soc. Ser. A 132, 235-244.
12. Armstrong, L., and Marks, L. E. (1997). Differential effect of stimulus context on perceived length: implications for the horizontal-vertical illusion. Percept. Psychophys. 59, 1200-1213.
13. Austin, J. T., Boyle, K. A., and Lualhati, J. C. (1998). Statistical conclusion validity for organizational science researchers: a review. Organ. Res. Methods 1, 164-208.
14. Baddeley, A., and Wilson, B. A. (2002). Prose recall and amnesia: implications for the structure of working memory. Neuropsychologia 40, 1737-1743.
15. Bakker, M., and Wicherts, J. M. (2011). The (mis) reporting of statistical results in psychology journals. Behav. Res. Methods 43, 666-678.
16. Bauer, P., and Köhne, K. (1994). Evaluation of experiments with adaptive interim analyses. Biometrics 50, 1029-1041.
17. Beins, B. C. (2009). Research Methods. A Tool for Life, 2nd Edn. Boston, MA: Pearson Education.
18. Bennett, C. M., Wolford, G. L., and Miller, M. B. (2009). The principled control of false positives in neuroimaging. Soc. Cogn. Affect. Neurosci. 4, 417-422.
19. Bland, J. M., and Altman, D. G. (2011). Comparisons against baseline within randomised groups are often used and can be highly misleading. Trials 12, 264.
20. Bollen, K. A. (2002). Latent variables in psychology and the social sciences. Annu. Rev. Psychol. 53, 605-634.
21. Boos, D. D., and Hughes-Oliver, J. M. (2000). How large does n have to be for Z and t intervals? Am. Stat. 54, 121-128.
22. Borsboom, D. (2006). The attack of the psychometricians. Psychometrika 71, 425-440.
23. Botella, J., Ximenez, C., Revuelta, J., and Suero, M. (2006). Optimization of sample size in controlled experiments: the CLAST rule. Behav. Res. Methods Instrum. Comput. 38, 65-76.
24. Campbell, D. T., and Stanley, J. C. (1966). Experimental and Quasi-Experimental Designs for Research. Chicago, IL: Rand McNally.
25. Caudill, S. B. (1988). Type I errors after preliminary tests for heteroscedasticity. Statistician 37, 65-68.
26. Chen, Y. H. J., DeMets, D. L., and Lang, K. K. G. (2004). Increasing sample size when the unblinded interim result is promising. Stat. Med. 23, 1023-1038.
27. Cheng, C. L., and Van Ness, J. W. (1994). On estimating linear relationships when both variables are subject to errors. J. R. Stat. Soc. Series B 56, 167-183.
28. Cook, T. D., and Campbell, D. T. (1979). Quasi-Experimentation: Design and Analysis Issues for Field Settings. Boston, MA: Houghton Mifflin.
29. Crawford, E. D., Blumenstein, B., and Thompson, I. (1998). Type III statistical error. Urology 51, 675.
30. Cronbach, L. J. (1957). The two disciplines of scientific psychology. Am. Psychol. 12, 671-684.
31. DeCarlo, L. T. (1998). Signal detection theory and generalized linear models. Psychol. Methods 3, 186-205.
32. Dell, R. B., Holleran, S., and Ramakrishnan, R. (2002). Sample size determination. ILAR J. 43, 207-213.
33. Draine, S. C., and Greenwald, A. G. (1998). Replicable unconscious semantic priming. J. Exp. Psychol. Gen. 127, 286-303.
34. Draper, N. R., and Smith, H. (1998). Applied Regression Analysis, 3rd Edn. New York: Wiley.
35. Dunn, G. (2007). Regression models for method comparison data. J. Biopharm. Stat. 17, 739-756.
36. Dunn, G., and Roberts, C. (1999). Modellingmethod comparisondata. Stat. Methods Med. Res. 8, 161-179.
37. Easterling, R. G., and Anderson, H. E. (1978). The effect of preliminary normality goodness of fit tests on subsequent inference. J. Stat. Comput. Simul. 8, 1-11.
38. Elvik, R. (1998). Evaluating the statistical conclusion validity of weighted mean results in meta-analysis by analysing funnel graph diagrams. Accid. Anal. Prev. 30, 255-266.
39. Erceg-Hurn, C. M., and Mirosevich, V. M. (2008). Modern robust statistical methods: an easy way to maximize the accuracy and power of your research. Am. Psychol. 63, 591-601.
40. Faul, F., Erdfelder, E., Lang, A.-G., and Buchner, A. (2007). G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav. Res. Methods 39, 175-191.
41. Fitts, D. A. (2010a). Improved stopping rules for the design of efficient smallsample experiments in biomedical and biobehavioral research. Behav. Res. Methods 42, 3-22.
42. Fitts, D. A. (2010b). The variablecriteria sequential stopping rule: generality to unequal sample sizes, unequal variances, or to large ANOVAs. Behav. Res. Methods 42, 918-929.
43. Fitts, D. A. (2011a). Ethics and animal numbers: Informal analyses, uncertain sample sizes, inefficient replications, and Type I errors. J. Am. Assoc. Lab. Anim. Sci. 50, 445-453.
44. Fitts, D. A. (2011b). Minimizing animal numbers: the variable-criteria sequential stopping rule. Comp. Med. 61, 206-218.
45. Frick, R. W. (1998). A better stopping rule for conventional statistical tests. Behav. Res. Methods Instrum. Comput. 30, 690-697.
46. Fried, R., and Dehling, H. (2011). Robust nonparametric tests for the two-sample location problem. Stat. Methods Appl. 20, 409-422.
47. Friedrich, J., Buday, E., and Kerr, D. (2000). Statistical training in psychology: a national survey and commentary on undergraduate programs. Teach. Psychol. 27, 248-257.
48. Fuller, W. A. (1987). Measurement Error Models. New York: Wiley.
49. Gans, D. J. (1981). Use of a preliminary test in comparing two sample means. Commun. Stat. Simul. Comput. 10, 163-174.
50. García-Pérez, M. A. (2005). On the confidence interval for the binomial parameter. Qual. Quant. 39, 467-481.
51. García-Pérez, M. A., and Alcalá-Quintana, R. (2007). Bayesian adaptive estimation of arbitrary points on a psychometric function. Br. J. Math. Stat. Psychol. 60, 147-174.
52. García-Pérez, M. A., and Alcalá-Quintana, R. (2011). Testing equivalence with repeated measures: tests of the difference model of two-alternative forced-choice performance. Span. J. Psychol. 14, 1023-1049.
53. García-Pérez, M. A., and Alcalá-Quintana, R. (2012). On the discrepant results in synchrony judgment and temporal-order judgment tasks: a quantitative model. Psychon. Bull. Rev. (in press). doi:10.3758/s13423-012-0278-y
54. García-Pérez, M. A., Alcalá-Quintana, R., and García-Cueto, M. A. (2010). A comparison of anchor-item designs for the concurrent calibration of large banks of Likert-type items. Appl. Psychol. Meas. 34, 580-599.
55. García-Pérez, M. A., Alcalá-Quintana, R., Woods, R. L., and Peli, E. (2011). Psychometric functions for detection and discrimination with and without flankers. Atten. Percept. Psychophys. 73, 829-853.
56. García-Pérez, M. A., and Núñez-Antón, V. (2009). Statistical inference involving binomial and negative binomial parameters. Span. J. Psychol. 12, 288-307.
57. Girden, E. R., and Kabacoff, R. I. (2011). Evaluating Research Articles. From Start to Finish, 3rd Edn. Thousand Oaks, CA: Sage.
58. Goodwin, C. J. (2010). Research in Psychology. Methods and Design, 6th Edn. Hoboken, NJ: Wiley.
59. Graybill, F. A. (1958). Determining sample size for a specified width confidence interval. Ann. Math. Stat. 29, 282-287.
60. Green, B. G. (1982). The perception of distance and location for dual tactile figures. Percept. Psychophys. 31, 315-323.
61. Greenwald, A. G., Klinger, M. R., and Schuh, E. S. (1995). Activation by marginally perceptible ("subliminal") stimuli: dissociation of unconscious from conscious cognition. J. Exp. Psychol. Gen. 124, 22-42.
62. Hawkins, D. M. (2002). Diagnostics for conformity of paired quantitative measurements. Stat. Med. 21, 1913-1935.
63. Hayes, A. F., and Cai, L. (2007). Further evaluating the conditional decision rule for comparing two independent means. Br. J. Math. Stat. Psychol. 60, 217-244.
64. Henson, R. K., Hull, D. M., and Williams, C. S. (2010). Methodology in our education research culture: toward a stronger collective quantitative proficiency. Educ. Res. 39, 229-240.
65. Hoekstra, R., Kiers, H., and Johnson, A. (2012). Are assumptions of well-known statistical techniques checked, and why (not)? Front. Psychol. 3:137. doi:10.3389/fpsyg.2012.00137
66. Howard, G. S., Obledo, F. H., Cole, D. A., and Maxwell, S. E. (1983). Linked raters' judgments: combating problems of statistical conclusion validity. Appl. Psychol. Meas. 7, 57-62.
67. Isaac, P. D. (1970). Linear regression, structural relations, and measurement error. Psychol. Bull. 74, 213-218.
68. Jan, S.-L., and Shieh, G. (2011). Optimal sample sizes for Welch's test under various allocation and cost considerations. Behav. Res. Methods 43, 1014-1022.
69. Jennison, C., and Turnbull, B. W. (1990). Statistical approaches to interim monitoring of clinical trials: a review and commentary. Stat. Sci. 5, 299-317.
70. John, L. K., Loewenstein, G., and Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 23, 524-532.
71. Kalantar, A. H., Gelb, R. I., and Alper, J. S. (1995). Biases in summary statistics of slopes and intercepts in linear regression with errors in both variables. Talanta 42, 597-603.
72. Keselman, H. J., Othman, A. R., Wilcox, R. R., and Fradette, K. (2004). The new and improved two-sample t test. Psychol. Sci. 15, 47-51.
73. Ketellapper, R. H. (1983). On estimating parameters in a simple linear errors-in-variables model. Technometrics 25, 43-47.
74. Lee, B. (1985). Statistical conclusion validity in ex post facto designs: practicality in evaluation. Educ. Eval. Policy Anal. 7, 35-45.
75. Lippa, R. A. (2007). The relation between sex drive and sexual attraction to men and women: a cross-national study of heterosexual, bisexual, and homosexual men and women. Arch. Sex. Behav. 36, 209-222.
76. Lumley, T., Diehr, P., Emerson, S., and Chen, L. (2002). The importance of the normality assumption in large public health data sets. Annu. Rev. Public Health 23, 151-169.
77. Malakoff, D. (1999). Bayes offers a"new" way to make sense of numbers. Science 286, 1460-1464.
78. Mandansky, A. (1959). The fitting of straight lines when both variables are subject to error. J. Am. Stat. Assoc. 54, 173-205.
79. Matthews, W. J. (2011). What might judgment and decision making research be like if we took a Bayesian approach to hypothesis testing? Judgm. Decis. Mak. 6, 843-856.
80. Maxwell, S. E., Kelley, K., and Rausch, J. R. (2008). Sample size planning for statistical power and accuracy in parameter estimation. Annu. Rev. Psychol. 59, 537-563.
81. Maylor, E. A., and Rabbitt, P. M. A. (1993). Alcohol, reaction time and memory: a meta-analysis. Br. J. Psychol. 84, 301-317.
82. McCarroll, D., Crays, N., and Dunlap, W. P. (1992). Sequential ANOVAs and type I error rates. Educ. Psychol. Meas. 52, 387-393.
83. Mehta, C. R., and Pocock, S. J. (2011). Adaptive increase in sample size when interim results are promising: a practical guide with examples. Stat. Med. 30, 3267-3284.
84. Milligan, G. W., and McFillen, J. M. (1984). Statistical conclusion validity in experimental designs used in business research. J. Bus. Res. 12, 437-462.
85. Morse, D. T. (1998). MINSIZE: a computer program for obtaining minimum sample size as an indicator of effect size. Educ. Psychol. Meas. 58, 142-153.
86. Morse, D. T. (1999). MINSIZE2: a computer program for determining effect size and minimum sample size for statistical significance for univariate, multivariate, and nonparametric tests. Educ. Psychol. Meas. 59, 518-531.
87. Moser, B. K., and Stevens, G. R. (1992). Homogeneity of variance in the twosample means test. Am. Stat. 46, 19-21.
88. Ng, M., and Wilcox, R. R. (2011). A comparison of two-stage procedures for testing least-squares coefficients under heteroscedasticity. Br. J. Math. Stat. Psychol. 64, 244-258.
89. Nickerson, R. S. (2000). Null hypothesis significance testing: a review of an old and continuing controversy. Psychol. Methods 5, 241-301.
90. Nickerson, R. S. (2005). What authors want from journal reviewers and editors. Am. Psychol. 60, 661-662.
91. Nieuwenhuis, S., Forstmann, B. U., and Wagenmakers, E.-J. (2011). Erroneous analyses of interactions in neuroscience: a problem of significance. Nat. Neurosci. 14, 1105-1107.
92. Nisen, J. A., and Schwertman, N. C. (2008). A simple method of computing the sample size for chi-square test for the equality of multinomial distributions. Comput. Stat. Data Anal. 52, 4903-4908.
93. Orme, J. G. (1991). Statistical conclusion validity for single-system designs. Soc. Serv. Rev. 65, 468-491.
94. Ottenbacher, K. J. (1989). Statistical conclusion validity of early intervention research with handicapped children. Except. Child. 55, 534-540.
95. Ottenbacher, K. J., and Maas, F. (1999). How to detect effects: statistical power and evidence-based practice in occupational therapy research. Am. J. Occup. Ther. 53, 181-188.
96. Rankupalli, B., and Tandon, R. (2010). Practicing evidence-based psychiatry: 1. Applying a study's findings: the threats to validity approach. Asian J. Psychiatr. 3, 35-40.
97. Rasch, D., Kubinger, K. D., and Moder, K. (2011). The two-sample t test: pre-testing its assumptions does not pay off. Stat. Pap. 52, 219-231.
98. Riggs, D. S., Guarnieri, J. A., and Addelman, S. (1978). Fitting straight lines when both variables are subject to error. Life Sci. 22, 1305-1360.
99. Rochon, J., and Kieser, M. (2011). A closer look at the effect of preliminary goodness-of-fit testing for normality for the one-sample ttest. Br. J. Math. Stat. Psychol. 64, 410-426.
100. Saberi, K., and Petrosyan, A. (2004). A detection-theoretic model of echo inhibition. Psychol. Rev. 111, 52-66.
101. Schucany, W. R., and Ng, H. K. T. (2006). Preliminary goodness-of-fit tests for normality do not validate the onesample Student t. Commun. Stat. Theory Methods 35, 2275-2286.
102. Shadish, W. R., Cook, T. D., and Campbell, D. T. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston, MA: Houghton Mifflin.
103. Shieh, G., and Jan, S.-L. (2012). Optimal sample sizes for precise interval estimation of Welch's procedure under various allocation and cost considerations. Behav. Res. Methods 44, 202-212.
104. Shun, Z. M., Yuan, W., Brady, W. E., and Hsu, H. (2001). Type I error in sample size re-estimations based on observed treatment difference. Stat. Med. 20, 497-513.
105. Simmons, J. P., Nelson, L. D., and Simoshohn, U. (2011). Falsepositive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359-1366.
106. Smith, P. L., Wolfgang, B. F., and Sinclair, A. J. (2004). Mask-dependent attentional cuing effects in visual signal detection: the psychometric function for contrast. Percept. Psychophys. 66, 1056-1075.
107. Sternberg, R. J. (2002). On civility in reviewing. APS Obs. 15, 34.
108. Stevens, W. L. (1950). Fiducial limits of the parameter of a discontinuous distribution. Biometrika 37, 117-129.
109. Strube, M. J. (2006). SNOOP: a program for demonstrating the consequences of premature and repeated null hypothesis testing. Behav. Res. Methods 38, 24-27.
110. Sullivan, L. M., and D'Agostino, R. B. (1992). Robustness of the t test applied to data distorted from normality by floor effects. J. Dent. Res. 71, 1938-1943.
111. Treisman, M., and Watts, T. R. (1966). Relation between signal detectability theory and the traditional procedures for measuring sensory thresholds: estimating d'from results given by the method of constant stimuli. Psychol. Bull. 66, 438-454.
112. Vecchiato, G., Fallani, F. V., Astolfi, L., Toppi, J., Cincotti, F., Mattia, D., Salinari, S., and Babiloni, F. (2010). The issue of multiple univariate comparisons in the context of neuroelectric brain mapping: an application in a neuromarketing experiment. J. Neurosci. Methods 191, 283-289.
113. Vul, E., Harris, C., Winkielman, P., and Pashler, H. (2009a). Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspect. Psychol. Sci. 4, 274-290.
114. Vul, E., Harris, C., Winkielman, P., and Pashler, H. (2009b). Reply to comments on "Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition." Perspect. Psychol. Sci. 4, 319-324.
115. Wagenmakers, E.-J. (2007). A practical solution to the pervasive problems of p values. Psychon. Bull. Rev. 14, 779-804.
116. Wald, A. (1940). The fitting of straight lines if both variables are subject to error. Ann. Math. Stat. 11, 284-300.
117. Wald, A. (1947). Sequential Analysis. New York: Wiley.
118. Wells, C. S., and Hintze, J. M. (2007). Dealing with assumptions underlying statistical tests. Psychol. Sch. 44, 495-502.
119. Wetherill, G. B. (1966). Sequential Methods in Statistics. London: Chapman and Hall.
120. Wicherts, J. M., Kievit, R. A., Bakker, M., and Borsboom, D. (2012). Letting the daylight in: reviewing the reviewers and other ways to maximize transparency in science. Front. Comput. Psychol. 6:20. doi:10.3389/fncom.2012.00020
121. Wilcox, R. R. (2006). New methods for comparing groups: strategies for increasing the probability of detecting true differences. Curr. Dir. Psychol. Sci. 14, 272-275.
122. Wilcox, R. R., and Keselman, H. J. (2003). Modern robust data analysis methods: measures of central tendency. Psychol. Methods 8, 254-274.
123. Wilkinson, L., and The Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: guidelines and explanations. Am. Psychol. 54, 594-604.
124. Ximenez, C., and Revuelta, J. (2007). Extending the CLAST sequential rule to one-way ANOVA under group sampling. Behav. Res. Methods Instrum. Comput. 39, 86-100.
125. Xu, E. R., Knight, E. J., and Kralik, J. D. (2011). Rhesus monkeys lack a consistent peak-end effect. Q. J. Exp. Psychol. 64, 2301-2315.
126. Yeshurun, Y., Carrasco, M., and Maloney, L. T. (2008). Bias and sensitivity in two-interval forced choice procedures: tests of the difference model. Vision Res. 48, 1837-1851.
127. Zaslavsky, B. G. (2010). Bayesian versus frequentist hypotheses testing in clinical trials with dichotomous and countable outcomes. J. Biopharm. Stat. 20, 985-997.
128. Zimmerman, D. W. (1996). Some properties of preliminary tests of equality of variances in the two-sample location problem. J. Gen. Psychol. 123, 217-231.
129. Zimmerman, D. W. (2004). A note on preliminary tests of equality of variances. Br. J. Math. Stat. Psychol. 57, 173-181.
130. Zimmerman, D. W. (2011). A simple and effective decision rule for choosing a significance test to protect against non-normality. Br. J. Math. Stat. Psychol. 64, 388-409.