Confidence intervals for correlations when data are not normal

Behavior Research Methods

Volumn 49, Issue 1, 2017, Pages 294-309

  Bishara, Anthony J ^a   Hittner, James B ^a  

^a Department of Teacher Education and Department of Sociology and Anthropology   (United States)

Author keywords

Confidence interval; Correlation; Fisher z; Fisher z'; Normal; r to z; Robust

Indexed keywords

AVOIDANCE BEHAVIOR; BOOTSTRAPPING; CONFIDENCE INTERVAL; SAMPLE SIZE; MONTE CARLO METHOD; NORMAL DISTRIBUTION; PROCEDURES; STATISTICS;

CONFIDENCE INTERVALS; MONTE CARLO METHOD; NORMAL DISTRIBUTION; SAMPLE SIZE; STATISTICS AS TOPIC;

EID: 84955573875     PISSN: 1554351X     EISSN: 15543528     Source Type: Journal    
DOI: 10.3758/s13428-016-0702-8     Document Type: Article

References (78)

1. American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington: Author.
2. Beasley, W. H., DeShea, L., Toothaker, L. E., Mendoza, J. L., Bard, D. E., & Rodgers, J. (2007). Bootstrapping to test for nonzero population correlation coefficients using univariate sampling. Psychological Methods, 12, 414–433. doi:10.1037/1082-989X.12.4.414
3. Beasley, T., Erickson, S., & Allison, D. (2009). Rank-based inverse normal transformations are increasingly used, but are they merited? Behavior Genetics, 39, 580–595. doi:10.1007/s10519-009-9281-0
4. Beasley, W. H., & Rodgers, J. L. (2012). Bootstrapping and Monte Carlo methods. In H. Cooper, P. M. Camic, D. L. Long, A. T. Panter, D. Rindskopf, & K. J. Sher (Eds.), APA handbook of research methods in psychology: Vol. 2. Research designs: Quantitative, qualitative, neuropsychological, and biological (pp. 407–425). Washington: American Psychological Association.
5. Bishara, A. J., & Hittner, J. B. (2012). Testing the significance of a correlation with non-normal data: Comparison of Pearson, Spearman, transformation, and resampling approaches. Psychological Methods, 17, 399–417. doi:10.1037/a0028087
6. Bishara, A. J., & Hittner, J. B. (2015). Reducing bias and error in the correlation coefficient due to nonnormality. Educational and Psychological Measurement, 75, 785–804.
7. Blanca, M. J., Arnau, J., López-Montiel, D., Bono, R., & Bendayan, R. (2013). Skewness and kurtosis in real data samples. Methodology, 9, 78–84.
8. Blanton, H., & Jaccard, J. (2006). Arbitrary metrics in psychology. American Psychologist, 61, 27–41.
9. Bliss, C. I. (1967). Statistics in biology. New York: McGraw-Hill.
10. Blom, G. (1958). Statistical estimates and transformed beta-variables. New York: Wiley.
11. Bonett, D. G., & Wright, T. A. (2000). Sample size requirements for estimating Pearson, Kendall and Spearman correlations. Psychometrika, 65, 23–28.
12. Boos, D. D. (2003). Introduction to the bootstrap world. Statistical Science, 18, 168–174.
13. Box, G. E. P., & Cox, D. R. (1964). An analysis of transformations. Journal of the Royal Statistical Society, Series B, 26, 211–252.
14. Cohen, J. (1994). The earth is round (p <.05). American Psychologist, 49, 997–1003. doi:10.1037/0003-066X.49.12.997
15. Coulson, M., Healey, M., Fidler, F., & Cumming, G. (2010). Confidence intervals permit, but do not guarantee, better inference than statistical significance testing. Frontiers in Quantitative Psychology and Measurement, 1, 26. doi:10.3389/fpsyg.2010.00026
16. Cox, N. J. (2010). Speaking Stata: The limits of sample skewness and kurtosis. Stata Journal, 10, 482–495.
17. Cumming, G. (2012). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. New York: Routledge.
18. Cumming, G., & Finch, S. (2005). Inference by eye: Confidence intervals and how to read pictures of data. American Psychologist, 60, 170–180. doi:10.1037/0003-066X.60.2.170
19. D’Agostino, R. B., Belanger, A., & D’Agostino, R. B., Jr. (1990). A suggestion for using powerful and informative tests of normality. American Statistician, 44, 316–321.
20. DeCarlo, L. T. (1997). On the meaning and use of kurtosis. Psychological Methods, 2, 292–307. doi:10.1037/1082-989X.2.3.292
21. Development Core Team, R. (2014). R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing. Retrieved from www.R-project.org
22. Duncan, G. T., & Layard, M. W. J. (1973). A Monte-Carlo study of asymptotically robust tests for correlation coefficients. Biometrika, 60, 551–558.
23. Edgell, S. E., & Noon, S. M. (1984). Effect of violation of normality on the t test of the correlation coefficient. Psychological Bulletin, 95, 576–583. doi:10.1037/0033-2909.95.3.576
24. Efron, B. (1979). Bootstrap methods: Another look at the jackknife. Annals of Statistics, 7, 1–26.
25. Efron, B. (1982). The jackknife, the bootstrap, and other resampling plans. SIAM CBMS-NSF Monograph, 38. doi:10.1137/1.9781611970319
26. Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. Boca Raton: CRC Press.
27. Fidler, F., & Loftus, G. R. (2009). Why figures with error bars should replace p values. Journal of Psychology, 217, 27–37.
28. Field, A. (2000). Discovering statistics using SPSS for Windows. Thousand Oaks: Sage.
29. Fieller, E. C., Hartley, H. O., & Pearson, E. S. (1957). Tests for rank correlation coefficients: I. Biometrika, 44, 470–481.
30. Filliben, J. J. (1975). The probability plot correlation coefficient test for normality. Technometrics, 17, 111–117.
31. Fisher, R. A. (1915). Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population. Biometrika, 10, 507–521.
32. Fisher, R. A. (1921). On the “probable error” of a coefficient of correlation deduced from a small sample. Metron, 1, 3–32.
33. Fisher, R. A., & Yates, F. (1938). Statistical tables for biological, agricultural and medical research. Oxford: Oliver & Boyd.
34. Fouladi, R. T., & Steiger, J. H. (2008). The Fisher transform of the Pearson product moment correlation coefficient and its square: Cumulants, moments, and applications. Communications in Statistics—Simulation and Computation, 37, 928–944.
35. Gayen, A. K. (1951). The frequency distribution of the product-moment correlation coefficient in random samples of any size drawn from non-normal universes. Biometrika, 38, 219–247. doi:10.1093/biomet/38.1-2.219
36. Haldane, J. B. S. (1949). A note on non-normal correlation. Biometrika, 36, 467–468.
37. Havlicek, L., & Peterson, N. (1977). Effect of the violation of assumptions upon significance levels of the Pearson r. Psychological Bulletin, 84, 373–377.
38. Hawkins, D. L. (1989). Using U statistics to derive the asymptotic distribution of Fisher’s Z statistic. American Statistician, 43, 235–237.
39. 0: ρ = 0. Psychological Methods, 1, 184–198.
40. Hays, W. L. (1994). Statistics (5th ed.). Fort Worth: Harcourt Brace.
41. Headrick, T. (2002). Fast fifth-order polynomial transforms for generating univariate and multivariate nonnormal distributions. Computational Statistics and Data Analysis, 40, 685–711.
42. Headrick, T. C. (2010). Statistical simulation: Power method polynomials and other transformations. Boca Raton: Chapman & Hall.
43. Headrick, T. C., Sheng, Y., & Hodis, F. A. (2007). Numerical computing and graphics for the power method transformation using Mathematica. Journal of Statistical Software, 19, 1–17.
44. Hoekstra, R., Johnson, A., & Kiers, H. A. L. (2012). Confidence intervals make a difference: Effects of presentation mode on inferential reasoning. Educational and Psychological Measurement, 72, 1039–1052.
45. Joanes, D. N., & Gill, C. A. (1998). Comparing measures of sample skewness and kurtosis. Statistician, 47, 183–189.
46. Kaiser, H. F., & Dickman, K. (1962). Sample and population score matrices and sample correlation matrices from an arbitrary population correlation matrix. Psychometrika, 27, 179–182.
47. Klaassen, C. A. J., & Wellner, J. A. (1997). Efficient estimation in the bivariate normal copula model: Normal margins are least favourable. Bernoulli, 3, 55–77.
48. Kowalski, C. J. (1972). On the effects of non-normality on the distribution of the sample product moment correlation coefficient. Applied Statistics, 21, 1–12.
49. Lee, W.-C., & Rodgers, J. L. (1998). Bootstrapping correlation coefficients using univariate and bivariate sampling. Psychological Methods, 3, 91–103. doi:10.1037/1082-989X.3.1.91
50. Lindsay, D. S. (2015). Replication in Psychological Science. Psychological Science, 26, 1827–1832. doi:10.1177/0956797615616374
51. Loftus, G. R. (1996). Psychology will be a much better science when we change the way we analyze data. Current Directions in Psychological Science, 5, 161–171. doi:10.1111/1467-8721.ep11512376
52. Lunneborg, C. E. (1985). Estimating the correlation coefficient: The bootstrap approach. Psychological Bulletin, 98, 209–215. doi:10.1037/0033-2909.98.1.209
53. Mardia, K. V. (1970). Measures of multivariate skewness and kurtosis with applications. Biometrika, 57, 519–530.
54. Mardia, K. V. (1974). Applications of some measures of multivariate skewness and kurtosis in testing normality and robustness studies. Sankhyã, Series B, 36, 115–128.
55. Mardia, K. V., Kent, J. T., & Bibby, J. M. (1979). Multivariate analysis. London: Academic Press.
56. Micceri, T. (1989). The unicorn, the normal curve, and other improbable creatures. Psychological Bulletin, 105, 156–166. doi:10.1037/0033-2909.105.1.156
57. Olvera Astivia, O. L., & Zumbo, B. D. (2015). A cautionary note on the use of the Vale and Maurelli method to generate multivariate, nonnormal data for simulation purposes. Educational and Psychological Measurement, 75, 541–567.
58. Pagano, M., & Gauvreau, K. (2000). Principles of biostatistics (2nd ed.). Pacific Grove: Duxbury Press.
59. Psychological Science. (2014). Submission guidelines. Retrieved September 3, 2014, from www.psychologicalscience.org/index.php/publications/journals/psychological_science/ps-submissions
60. Psychonomic Society. (2012). Psychonomic society guidelines on statistical issues. Retrieved from www.psychonomic.org/statistical-guidelines
61. Puth, M., Neuhäuser, M., & Ruxton, G. D. (2014). Effective use of Pearson’s product-moment correlation coefficient. Animal Behaviour, 93, 183–189.
62. Puth, M., Neuhäuser, M., & Ruxton, G. D. (2015). Effective use of Spearman’s and Kendall’s correlation coefficients for association between two measured traits. Animal Behaviour, 102, 77–84.
63. Rasmussen, J. (1987). Estimating correlation coefficients: Bootstrap and parametric approaches. Psychological Bulletin, 101, 136–139. doi:10.1037/0033-2909.101.1.136
64. Ratcliff, R. (1979). Group reaction time distributions and an analysis of distribution statistics. Psychological Bulletin, 86, 446–461. doi:10.1037/0033-2909.86.3.446
65. Rosner, B. (1995). Fundamentals of biostatistics (4th ed.). Belmont: Wadsworth.
66. Rosner, B., & Glynn, R. J. (2007). Interval estimation for rank correlation coefficients based on the probit transformation with extension to measurement error correction of correlated ranked data. Statistics in Medicine, 26, 633–646.
67. Ruscio, J. (2008). Constructing confidence intervals for Spearman’s rank correlation with ordinal data: A simulation study comparing analytic and bootstrap methods. Journal of Modern Applied Statistical Methods, 7, 416–434.
68. Shapiro, S. S., & Wilk, M. B. (1965). Analysis of variance test for normality (complete samples). Biometrika, 52, 591–611.
69. Shapiro, S. S., Wilk, M. B., & Chen, H. J. (1968). A comparative study of various tests for normality. Journal of the American Statistical Association, 63, 1343–1372.
70. Solomon, S. R., & Sawilowsky, S. S. (2009). Impact of rank-based normalizing transformations on the accuracy of test scores. Journal of Modern Applied Statistical Methods, 8, 448–462.
71. Strube, M. (1988). Bootstrap Type I error rates for the correlation coefficient: An examination of alternate procedures. Psychological Bulletin, 104, 290–292. doi:10.1037/0033-2909.104.2.290
72. Triola, M. (2010). Elementary statistics (11th ed.). Boston: Addison-Wesley/Pearson Education.
73. Tryon, W. W. (2001). Evaluating statistical difference, equivalence, and indeterminacy using inferential confidence intervals: An integrated alternative method of conducting null hypothesis statistical tests. Psychological Methods, 6, 371–386. doi:10.1037/1082-989X.6.4.371
74. van der Waerden, B. L. (1952). Order tests for the two-sample problem and their power. Indagationes Mathematicae, 14, 453–458.
75. Westfall, P. H. (2014). Kurtosis as peakedness, 1905–2014. RIP. American Statistician, 68, 191–195.
76. Woods, C. M. (2007). Confidence intervals for gamma-family measures of ordinal association. Psychological Methods, 12, 185–204. doi:10.1080/00031305.2014.917055
77. Zimmerman, D., Zumbo, B., & Williams, R. (2003). Bias in estimation and hypothesis testing of correlation. Psicológica, 24, 133–158.
78. Zou, K. H., & Hall, W. J. (2002). On estimating a transformation correlation coefficient. Journal of Applied Statistics, 29, 745–760. doi:10.1080/02664760120098801