Violating the normality assumption may be the lesser of two evils

Behavior Research Methods

Volumn 53, Issue 6, 2021, Pages 2576-2590

  Knief, Ulrich ^a   Forstmeier, Wolfgang ^b  

^a UNIVERSITY OF MUNICH   (Germany)

^b MAX PLANCK INSTITUTE FOR ORNITHOLOGY   (Germany)

Author keywords

Hypothesis testing; Linear model; Normality; Regression

Indexed keywords

ARTICLE; BEAR; CONTROLLED STUDY; HUMAN; MONTE CARLO METHOD; NONHUMAN; PEER REVIEW; RANDOMIZATION; RANDOMIZED CONTROLLED TRIAL; RELIABILITY; RISK ASSESSMENT; SAMPLE SIZE; NORMAL DISTRIBUTION; REPRODUCIBILITY; STATISTICAL ANALYSIS;

DATA INTERPRETATION, STATISTICAL; HUMANS; MONTE CARLO METHOD; NORMAL DISTRIBUTION; REPRODUCIBILITY OF RESULTS; SAMPLE SIZE;

EID: 85105490979     PISSN: 1554351X     EISSN: 15543528     Source Type: Journal    
DOI: 10.3758/s13428-021-01587-5     Document Type: Article

References (72)

1. Ali MM, Sharma SC (1996) Robustness to nonnormality of regression F-tests. J Econom 71, 175–205.
2. Arnqvist G (2020) Mixed models offer no freedom from degrees of freedom. Trends Ecol Evol 35, 329–335.
3. Barr DJ, Levy R, Scheepers C, Tily HJ (2013) Random effects structure for confirmatory hypothesis testing: keep it maximal. J Mem Lang 68, 255–278. DOI: 10.1016/j.jml.2012.11.001
4. Bates D, Mächler M, Bolker BM, Walker SC (2015) Fitting linear mixed-effects models using lme4. J Stat Softw 67, 1–48. DOI: 10.18637/jss.v067.i01
5. Bishara AJ, Hittner JB (2012) Testing the significance of a correlation with nonnormal data: comparison of Pearson, Spearman, transformation, and resampling approaches. Psychol Methods 17, 399–417. DOI: 10.1037/a0028087
6. Blair RC, Lawson SB (1982) Another look at the robustness of the product-moment correlation coefficient to population non-normality. Florida J Educ Res 24, 11–15.
7. Bliss CI (1967) Statistics in biology. McGraw-Hill.
8. Bolker BM, Brooks ME, Clark CJ, Geange SW, Poulsen JR, Stevens MHH, White JSS (2009) Generalized linear mixed models: a practical guide for ecology and evolution. Trends Ecol Evol 24, 127–135. DOI: 10.1016/j.tree.2008.10.008
9. Box GEP (1953) Non-normality and tests on variances. Biometrika 40, 318–335. DOI: 10.1093/biomet/40.3-4.318
10. Box GEP, Watson GS (1962) Robustness to non-normality of regression tests. Biometrika 49, 93–106. DOI: 10.1093/biomet/49.1-2.93
11. Brooks ME, Kristensen K, van Benthem KJ, Magnusson A, Berg CW, Nielsen A, … Bolker BM (2017) Modeling zero-inflated count data with glmmTMB. bioRxiv, e132753.
12. Brunner J, Austin PC (2009) Inflation of type I error rate in multiple regression when independent variables are measured with error. Can J Stat 37, 33–46.
13. Camerer CF, Dreber A, Holzmeister F, Ho TH, Huber J, Johannesson M, … Wu H (2018) Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nat Hum Behav 2, 637–644. DOI: 10.1038/s41562-018-0399-z
14. Casella G, Berger RL (2002) Statistical inference. Duxbury Press.
15. Cochran WG (1950) The comparison of percentages in matched samples. Biometrika 37, 256–266. DOI: 10.1093/biomet/37.3-4.256
16. Dupont WD, Plummer WD (1998) Power and sample size calculations for studies involving linear regression. Control Clin Trials 19, 589–601. DOI: 10.1016/S0197-2456(98)00037-3
17. Ebersole CR, Atherton OE, Belanger AL, Skulborstad HM, Allen JM, Banks JB, … Nosek BA (2016) Many labs 3: evaluating participant pool quality across the academic semester via replication. J Exp Soc Psychol 67, 68–82. DOI: 10.1016/j.jesp.2015.10.012
18. Fordyce JA, Gompert Z, Forister ML, Nice CC (2011) A hierarchical Bayesian approach to ecological count data: a flexible tool for ecologists. PLOS ONE 6, e26785.
19. Forstmeier W, Wagenmakers EJ, Parker TH (2017) Detecting and avoiding likely false-positive findings – a practical guide. Biol Rev 92, 1941–1968. DOI: 10.1111/brv.12315
20. Frank SA (2009) The common patterns of nature. J Evol Biol 22, 1563–1585.
21. Freckleton RP (2011) Dealing with collinearity in behavioural and ecological data: model averaging and the problems of measurement error. Behav Ecol Sociobiol 65, 91–101. DOI: 10.1007/s00265-010-1045-6
22. Gelman A, Hill J (2007) Data analysis using regression and multilevel/hierarchical models. Cambridge University Press.
23. Ghasemi A, Zahediasl S (2012) Normality tests for statistical analysis: a guide for non-statisticians. Int J Endocrinol Metab 10, 486–489. DOI: 10.5812/ijem.3505
24. Glass GV, Peckham PD, Sanders JR (1972) Consequences of failure to meet assumptions underlying the fixed effects analysis of variance and covariance. Rev Educ Res 42, 237–288.
25. Good PI (2005) Permutation, parametric, and bootstrap tests of hypotheses. Springer.
26. Hack HRB (1958) An empirical investigation into the distribution of the F-ratio in samples from two non-normal populations. Biometrika 45, 260–265. DOI: 10.1093/biomet/45.1-2.260
27. Harrison XA (2014) Using observation-level random effects to model overdispersion in count data in ecology and evolution. PeerJ 2, e616.
28. Harrison XA (2015) A comparison of observation-level random effect and Beta-Binomial models for modelling overdispersion in Binomial data in ecology & evolution. PeerJ 3, e1114.
29. Harrison XA, Donaldson L, Correa-Cano ME, Evans J, Fisher DN, Goodwin CE, … Inger R (2018) A brief introduction to mixed effects modelling and multi-model inference in ecology. PeerJ 6, e4794.
30. Hayes AF, Cai L (2007) Using heteroskedasticity-consistent standard error estimators in OLS regression: an introduction and software implementation. Behav Res Methods 39, 709–722. DOI: 10.3758/BF03192961
31. Huber PJ (1967) The behavior of maximum likelihood estimates under nonstandard conditions. Berkeley Symp on Math Statist and Prob 5.1, 221–233.
32. Ihle M, Pick JL, Winney IS, Nakagawa S, Burke T (2019) Measuring up to reality: null models and analysis simulations to study parental coordination over provisioning offspring. Front Ecol Evol 7, e142. DOI: 10.3389/fevo.2019.00142
33. Ives AR (2015) For testing the significance of regression coefficients, go ahead and log-transform count data. Methods Ecol Evol 6, 828–835. DOI: 10.1111/2041-210X.12386
34. Ives AR, Garland T (2014) Phylogenetic regression for binary dependent variables. In: Modern phylogenetic comparative methods and their application in evolutionary biology (ed. Garamszegi LZ), pp. 231–261. Springer, Berlin, Heidelberg.
35. Kass RE, Caffo BS, Davidian M, Meng XL, Yu B, Reid N (2016) Ten simple rules for effective statistical practice. PLOS Comput Biol 12, e1004961.
36. Knief U, Schielzeth H, Backström N, Hemmrich-Stanisak G, Wittig M, Franke A, … Forstmeier W (2017) Association mapping of morphological traits in wild and captive zebra finches: reliable within, but not between populations. Mol Ecol 26, 1285–1305. DOI: 10.1111/mec.14009
37. Komsta L, Novomestky F (2015) moments: Moments, cumulants, skewness, kurtosis and related tests. R package version 0.14.
38. Lin LI (1989) A concordance correlation-coefficient to evaluate reproducibility. Biometrics 45, 255–268. DOI: 10.2307/2532051
39. Lumley T, Diehr P, Emerson S, Chen L (2002) The importance of the normality assumption in large public health data sets. Annu Rev Public Health 23, 151–169.
40. Mardia KV (1971) The effect of nonnormality on some multivariate tests and robustness to nonnormality in the linear model. Biometrika 58, 105–121. DOI: 10.1093/biomet/58.1.105
41. McGuinness KA (2002) Of rowing boats, ocean liners and tests of the ANOVA homogeneity of variance assumption. Austral Ecol 27, 681–688. DOI: 10.1046/j.1442-9993.2002.01233.x
42. Miller RG (1986) Beyond ANOVA: basics of applied statistics. John Wiley & Sons, Inc.
43. O'Hara RB (2009) How to make models add up—a primer on GLMMs. Ann Zool Fenn 46, 124–137. DOI: 10.5735/086.046.0205
44. O'Hara RB, Kotze DJ (2010) Do not log-transform count data. Methods Ecol Evol 1, 118–122. DOI: 10.1111/j.2041-210X.2010.00021.x
45. Önöz B, Bayazit M (2012) Block bootstrap for Mann–Kendall trend test of serially dependent data. Hydrol Process 26, 3552–3560. DOI: 10.1002/hyp.8438
46. Open Science Collaboration (2015) Estimating the reproducibility of psychological science. Science 349, aac4716. DOI: 10.1126/science.aac4716
47. Osborne JW, Overbay A (2004) The power of outliers (and why researchers should ALWAYS check for them). Pract Assess Res Evaluation 9, art6.
48. Osborne JW, Waters E (2002) Four assumptions of multiple regression that researchers should always test. Pract Assess Res Evaluation 8, art2.
49. Pinheiro JC, Bates DM (2000) Mixed-effects models in S and S-PLUS. Springer.
50. Plaschke S, Bulla M, Cruz-López M, Gómez del Ángel S, Küpper C (2019) Nest initiation and flooding in response to season and semi-lunar spring tides in a ground-nesting shorebird. Front Zool 16, e15. DOI: 10.1186/s12983-019-0313-1
51. Puth MT, Neuhauser M, Ruxton GD (2014) Effective use of Pearson's product-moment correlation coefficient. Anim Behav 93, 183–189. DOI: 10.1016/j.anbehav.2014.05.003
52. Qiu W (2018) powerMediation: Power/Sample Size Calculation for Mediation Analysis. R package version 0.2.9.
53. Quinn GP, Keough MJ (2002) Experimental design and data analysis for biologists. Cambridge University Press.
54. Ramsey F, Schafer DW (2013) The statistical sleuth: a course in methods of data analysis. Brooks/Cole.
55. Ruscio J, Kaczetow W (2008) Simulating multivariate nonnormal data using an iterative algorithm. Multivar Behav Res 43, 355–381. DOI: 10.1080/00273170802285693
56. Santema P, Schlicht E, Kempenaers B (2019) Testing the conditional cooperation model: what can we learn from parents taking turns when feeding offspring? Front Ecol Evol 7, e94. DOI: 10.3389/fevo.2019.00094
57. Saravanan V, Berman GJ, Sober SJ (2020) Application of the hierarchical bootstrap to multi-level data in neuroscience. bioRxiv, e819334.
58. Schielzeth H, Dingemanse NJ, Nakagawa S, Westneat DF, Allegue H, Teplitsky C, … Araya-Ajoy YG (2020) Robustness of linear mixed-effects models to violations of distributional assumptions. Methods Ecol Evol 11, 1141–1152. DOI: 10.1111/2041-210X.13434
59. Schönbrodt F (2012) Ruscio - Code for generating correlating variables with arbitrary distributions. https://gist.github.com/nicebread/4045717.
60. Silberzahn R, Uhlmann EL, Martin DP, Anselmi P, Aust F, Awtrey E, … Nosek BA (2018) Many analysts, one data set: making transparent how variations in analytic choices affect results. Adv Methods Pract Psychol Sci 1, 337–356.
61. Simmons JP, Nelson LD, Simonsohn U (2011) False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci 22, 1359–1366. DOI: 10.1177/0956797611417632
62. Sokal RR, Rohlf FJ (1995) Biometry. W. H. Freeman.
63. Szöcs E, Schäfer RB (2015) Ecotoxicology is not normal. Environ Sci Pollut Res 22, 13990–13999.
64. Warton DI, Hui FKC (2011) The arcsine is asinine: the analysis of proportions in ecology. Ecology 92, 3–10. DOI: 10.1890/10-0340.1
65. Warton DI, Lyons M, Stoklosa J, Ives AR (2016) Three points to consider when choosing a LM or GLM test for count data. Methods Ecol Evol 7, 882–890. DOI: 10.1111/2041-210X.12552
66. Wedderburn RWM (1974) Quasi-likelihood functions, generalized linear models, and the Gauss-Newton method. Biometrika 61, 439–447.
67. White H (1980) A Heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica 48, 817–838. DOI: 10.2307/1912934
68. Williams MN, Grajales CAG, Kurkiewicz D (2013) Assumptions of multiple regression: correcting two misconceptions. Pract Assess Res Evaluation 18, art11.
69. Young LJ, Campbell NL, Capuano GA (1999) Analysis of overdispersed count data from single-factor experiments: a comparative study. J Agric Biol Environ Stat 4, 258–275.
70. Zuur A, Ieno EN, Walker N, Saveliev AA, Smith GM (2009) Mixed effects models and extensions in ecology with R. Springer.
71. Zuur AF, Ieno EN, Elphick CS (2010) A protocol for data exploration to avoid common statistical problems. Methods Ecol Evol 1, 3–14. DOI: 10.1111/j.2041-210X.2009.00001.x
72. Zuur AK, Ieno EN, Smith GM (2007) Analysing ecological data. Springer Science + Business Media, LLC.