Testing the significance of a correlation with nonnormal data: Comparison of Pearson, Spearman, transformation, and resampling approaches

Psychological Methods

Volumn 17, Issue 3, 2012, Pages 399-417

  Bishara, Anthony J ^a   Hittner, James B ^a  

^a School of Business   (United States)

Author keywords

Correlation; Nonnormal; Pearson; Spearman; Transformation

Indexed keywords

METHODOLOGY; MONTE CARLO METHOD; NONLINEAR SYSTEM; NONPARAMETRIC TEST; REVIEW; SAMPLE SIZE; STATISTICAL ANALYSIS; STATISTICAL DISTRIBUTION; STATISTICAL MODEL; STATISTICS;

DATA INTERPRETATION, STATISTICAL; LINEAR MODELS; MONTE CARLO METHOD; NONLINEAR DYNAMICS; SAMPLE SIZE; STATISTICAL DISTRIBUTIONS; STATISTICS AS TOPIC; STATISTICS, NONPARAMETRIC;

EID: 84870065807     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0028087     Document Type: Article

References (97)

1. Allison, D. B., Neale, M. C., Zannolli, R., Schork, N. J., Amos, C. I., & Blangero, J. (1999). Testing the robustness of the likelihood-ratio test in a variance-component quantitative-trait loci-mapping procedure. American Journal of Human Genetics, 65, 531-544. doi:10.1086/302487.
2. American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: Author.
3. Anderson, D., Sweeney, D., & Williams, T. (1997). Essentials of statistics for business and economics. Minneapolis/St. Paul, MN: West/Thomson Learning.
4. Beasley, T. M., Erickson, S., & Allison, D. B. (2009). Rank-based inverse normal transformations are increasingly used, but are they merited? Behavior Genetics, 39, 580-595. doi:10.1007/s10519-009-9281-0.
5. Beasley, W. H., DeShea, L., Toothaker, L. E., Mendoza, J. L., Bard, D. E., & Rodgers, J. (2007). Bootstrapping to test for nonzero population correlation coefficients using univariate sampling. Psychological Methods, 12, 414-433. doi:10.1037/1082-989X.12.4.414.
6. Beasley, W. H., & Rodgers, J. L. (2009). Resampling methods. In R. E. Millsap and A. Maydeu-Olivares (Eds.), The SAGE handbook of quantitative methods in psychology (pp. 362-386). London, England: Sage.
7. Berry, G. L. (1981). The Weibull distribution as a human performance descriptor. IEEE Transactions on Systems, Man, & Cybernetics, 11, 501-504. doi:10.1109/TSMC.1981.4308727.
8. Best, D. J., & Roberts, D. E. (1975). Algorithm AS 89: The upper tail probabilities of Spearman's rho. Applied Statistics, 24, 377-379. doi: 10.2307/2347111.
9. Bishara, A. J., Pleskac, T. J., Fridberg, D. J., Yechiam, E., Lucas, J., Busemeyer, J. R., Stout, J. C. (2009). Similar processes despite divergent behavior in two commonly used measures of risky decision making. Journal of Behavioral Decision Making, 22, 435-454. doi: 10.1002/bdm.641.
10. Blair, R. C., & Karniski, W. (1993). An alternative method for significance testing of waveform difference potentials. Psychophysiology, 30, 518-524. doi:10.1111/j.1469-8986.1993.tb02075.x.
11. Blair, R., & Lawson, S. (1982). Another look at the robustness of the product-moment correlation coefficient to population non-normality. Florida Journal of Educational Research, 24, 11-15.
12. Blair, R. C., Sawilowsky, S. S., & Higgins, J. J. (1987). Limitations of the rank transform statistic in tests for interactions. Communications in Statistics: Simulation and Computation, 16, 1133-1145. doi:10.1080/03610918708812642.
13. Bliss, C. I. (1967). Statistics in biology. New York, NY: McGraw-Hill.
14. Blom, G. (1958). Statistical estimates and transformed beta variables. New York, NY: Wiley.
15. Borkowf, C. B. (2002). Computing the nonnull asymptotic variance and the asymptotic relative efficiency of Spearman's rank correlation. Computational Statistics and Data Analysis, 39, 271-286. doi:10.1016/S0167-9473(01)00081-0.
16. Box, G. E. P., & Cox, D. R. (1964). An analysis of transformations. Journal of the Royal Statistical Society Series B (Methodological), 26, 211-252.
17. Bradley, J. (1977). A common situation conducive to bizarre distribution shapes. American Statistician, 31, 147-150. doi:10.2307/2683535.
18. Bray, J. H. (2010). The future of psychology practice and science. American Psychologist, 65, 355-369. doi:10.1037/a0020273.
19. Bullmore, E. T., Suckling, J., Overmeyer, S., Rabe-Hesketh, S., Taylor, R., & Brammer, M. J. (1999). Global, voxel, and cluster tests, by theory and permutation, for a difference between two groups of structural MR images of the brain. IEEE Transactions on Medical Imaging, 18, 32-42. doi:10.1109/42.750253.
20. Calkins, D. S. (1974). Some effects of non-normal distribution shape on the magnitude of the Pearson product moment correlation coefficient. Revista Interamericana de Psicología, 8, 261-288.
21. Cervone, D. (1985). Randomization tests to determine significance levels for microanalytic congruences between self-efficacy and behavior. Cognitive Therapy and Research, 9, 357-365. doi:10.1007/BF01173085.
22. Cohen, J. (1977). Statistical power analysis for the behavioral sciences (Rev. ed.). New York, NY: Academic Press.
23. Cohen, J., & Cohen, P. (1983). Applied multiple regression/correlation for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
24. Cohen, J., Cohen, P., West, S., & Aiken, L. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah, NJ: Erlbaum.
25. Conneely, K. N., & Boehnke, M. (2007). So many correlated tests, so little time! Rapid adjustment of P values for multiple correlated tests. American Journal of Human Genetics, 81, 1158-1168. doi:10.1086/522036.
26. Daniel, W. (1983). Biostatistics: A foundation for analysis in the health sciences (3rd ed.). New York, NY: Wiley.
27. Darlington, R. B. (1990). Regression and linear models. New York, NY: McGraw-Hill.
28. Duncan, G. T., & Layard, M. W. J. (1973). A Monte-Carlo study of asymptotically robust tests for correlation coefficients. Biometrika, 60, 551-558. doi:10.1093/biomet/60.3.551.
29. Dunlap, W., Burke, M., & Greer, T. (1995). The effect of skew on the magnitude of product-moment correlations. Journal of General Psychology, 122, 365-377. doi:10.1080/00221309.1995.9921248.
30. Edgell, S., & Noon, S. (1984). Effect of violation of normality on the t test of the correlation coefficient. Psychological Bulletin, 95, 576-583. doi:10.1037/0033-2909.95.3.576.
31. Efron, B. (1979). Bootstrap methods: Another look at the jackknife. Annals of Statistics, 7, 1-26. doi:10.1214/aos/1176344552.
32. Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. New York, NY: Chapman & Hall.
33. Field, A. (2000). Discovering statistics using SPSS for Windows. Thousand Oaks, CA: Sage.
34. Filliben, J. J. (1975). The probability plot correlation coefficient test for normality. Technometrics, 17, 111-117. doi:10.2307/1268008.
35. Fisher, R. A. (1928). Statistical methods for research workers (2nd ed.). London, England: Oliver & Boyd.
36. Fisher, R. A. (1935). The design and analysis of experiments. Edinburgh, Scotland: Oliver & Boyd.
37. Fisher, R. A., & Yates, F. (1938). Statistical tables for biological, agricultural and medical research. Oxford, England: Oliver & Boyd.
38. Fowler, R. (1987). Power and robustness in product-moment correlation. Applied Psychological Measurement, 11, 419-428. doi:10.1177/014662168701100407.
39. Gauthier, T. (2001). Detecting trends using Spearman's rank correlation coefficient. Environmental Forensics, 2, 359-362. doi:10.1006/enfo.2001.0061.
40. Gay, L., Mills, G., & Airasian, P. (2009). Educational research: Competencies for analysis and applications (9th ed.). Upper Saddle River, NJ: Merrill/Pearson.
41. Good, P. (2005). Permutation, parametric and bootstrap tests of hypotheses (3rd ed.). New York, NY: Springer-Verlag.
42. Good, P. (2009). Robustness of Pearson correlation. InterStat, 15(5), 1-6.
43. Gravetter, F., & Wallnau, L. (2004). Statistics for the behavioral sciences: A short course and student manual. Lanham, MD: University Press of America.
44. Havlicek, L., & Peterson, N. (1977). Effect of the violation of assumptions upon significance levels of the Pearson r. Psychological Bulletin, 84, 373-377. doi:10.1037/0033-2909.84.2.373.
45. Hayes, A. (1996). Permutation test is not distribution-free: Testing H0: ρ= 0. Psychological Methods, 1, 184-198. doi:10.1037/1082-989X.1.2.184.
46. Hays, W. L. (1988). Statistics (4th ed.). Fort Worth, TX: Holt, Rinehart & Winston.
47. Headrick, T. C. (2002). Fast fifth-order polynomial transforms for generating univariate and multivariate nonnormal distributions. Computational Statistics & Data Analysis, 40, 685-711. doi:10.1016/S0167-9473(02)00072-5.
48. Headrick, T. C., & Sawilowsky, S. (1999). Simulating correlated multivariate nonnormal distributions: Extending the Fleishman power method. Psychometrika, 64, 25-35. doi:10.1007/BF02294317.
49. Headrick, T. C., & Sawilowsky, S. (2000). Weighted simplex procedures for determining boundary points and constants for the univariate and multivariate power methods. Journal of Educational and Behavioral Statistics, 25, 417-436. doi:10.2307/1165223.
50. Hesterberg, T., Monaghan, S., Moore, D. S., Clipson, A., & Epstein, R. (2003). Bootstrap methods and permutation tests. In D. S. Moore, G. P. McCabe, W. M. Duckworth II, & S. L. Sclove (Eds.), The practice of business statistics (pp. 18-1-18-73). New York, NY: Freeman.
51. Hittner, J. B., May, K., & Silver, N. (2003). A Monte Carlo evaluation of tests for comparing dependent correlations. Journal of General Psychology, 130, 149-168. doi:10.1080/00221300309601282.
52. Hurlburt, R. (1994). Comprehending behavioral statistics. Belmont, CA: Thomson Brooks/Cole.
53. Keller-McNulty, S., & Higgins, J. J. (1987). Effect of tail weight and outliers on power and Type-I error of robust permutation tests for location. Communications in Statistics: Simulation and Computation, 16, 17-35. doi:10.1080/03610918708812575.
54. Kowalski, C. J., & Tarter, M. E. (1969). Co-ordinate transformations to normality and the power of normal tests for independence. Biometrika, 56, 139-148. doi:10.1093/biomet/56.1.139.
55. Lancaster, H. O. (1957). Some properties of the bivariate normal distribution considered in the form of a contingency table. Biometrika, 44, 289-292. doi:10.1093/biomet/44.1-2.289.
56. Lee, W., & Rodgers, J. L. (1998). Bootstrapping correlation coefficients using univariate and bivariate sampling. Psychological Methods, 3, 91-103. doi:10.1037/1082-989X.3.1.91.
57. Logan, G. D. (1992). Shapes of reaction-time distributions and shapes of learning curves: A test of the instance theory of automaticity. Journal of Experimental Psychology: Learning, Memory, and Cognition, 18, 883-914. doi:10.1037/0278-7393.18.5.883.
58. Lunneborg, C. E. (1985). Estimating the correlation coefficient: The bootstrap approach. Psychological Bulletin, 98, 209-215. doi:10.1037/0033-2909.98.1.209.
59. Manly, B. F. J. (1976). Exponential data transformations. Statistician, 25, 37-42. doi:10.2307/2988129.
60. Manly, B. F. J. (1997). Randomization, bootstrap and Monte Carlo methods in biology (2nd ed.). Boca Raton, FL: Chapman-Hall/CRC.
61. May, R., & Hunter, M. (1993). Some advantages of permutation tests. Canadian Psychology, 34, 401-407. doi:10.1037/h0078862.
62. McGrath, R. (1996). Understanding statistics: A research perspective. New York, NY: Longman.
63. Micceri, T. (1989). The unicorn, the normal curve, and other improbable creatures. Psychological Bulletin, 105, 156-166. doi:10.1037/0033-2909.105.1.156.
64. Mielke, P. W., & Berry, K. J. (2007). Permutation methods: A distance function approach (2nd ed.). New York, NY: Springer.
65. Munro, B. (2005). Statistical methods for health care research (5th ed.). Philadelphia, PA: Lippincott, Williams & Wilkins.
66. Osborne, J. W. (2002). Notes on the use of data transformations. Practical Assessment, Research & Evaluation, 8(6). Available online at http://pareonline.net/getvn.asp?v=8&n=6.
67. Osborne, J. W. (2010). Improving your data transformations: Applying the Box-Cox transformation. Practical Assessment, Research, & Evaluation, 15(12), 1-9.
68. Pagano, M., & Gauvreau, K. (2000). Principles of biostatistics (2nd ed.). Pacific Grove, CA: Duxbury Press.
69. Penfield, D. (1994). Choosing a two-sample location test. Journal of Experimental Education, 62, 343-360. doi:10.1080/00220973.1994.9944139.
70. Pitman, E. J. G. (1937). Significance tests which may be applied to samples from any populations. Journal of the Royal Statistical Society (Series B), 4, 119-130.
71. Rasmussen, J. (1987). Estimating correlation coefficients: Bootstrap and parametric approaches. Psychological Bulletin, 101, 136-139. doi: 10.1037/0033-2909.101.1.136.
72. Rasmussen, J. (1989). Data transformation, Type I error rate and power. British Journal of Mathematical and Statistical Psychology, 42, 203-213. doi:10.1111/j.2044-8317.1989.tb00910.x.
73. Rasmussen, J., & Dunlap, W. (1991). Dealing with nonnormal data: Parametric analysis of transformed data vs. nonparametric analysis. Educational and Psychological Measurement, 51, 809-820. doi: 10.1177/001316449105100402.
74. R Development Core Team. (2010). R: A language and environment for statistical computing. Retrieved from http://www.r-project.org.
75. Roberts, D., & Kunst, R. (1990). A case against continuing use of the Spearman formula for rank-order correlation. Psychological Reports, 66, 339-349.
76. Rosner, B. (1995). Fundamentals of biostatistics (4th ed.). Belmont, CA: Wadsworth.
77. Runyon, R. P., Haber, A., Pittenger, D. J., & Coleman, K. A. (1996). Fundamentals of behavioral statistics (8th ed.). New York, NY: McGraw-Hill.
78. Ruscio, J., & Kaczetow, W. (2008). Simulating multivariate nonnormal data using an iterative algorithm. Multivariate Behavioral Research, 43, 355-381. doi:10.1080/00273170802285693.
79. Salkind, N. (2008). Statistics for people who (think they) hate statistics (3rd ed.). Thousand Oaks, CA: Sage.
80. Salter, K. C., & Fawcett, R. F. (1993). The ART test of interaction: A robust and powerful test of interaction in factorial models. Communications in Statistics: Simulation and Computation, 22, 137-153. doi: 10.1080/03610919308813085.
81. Solomon, S. R., & Sawilowsky, S. S. (2009). Impact of rank-based normalizing transformations on the accuracy of test scores. Journal of Modern Applied Statistical Methods, 8, 448-462.
82. Strube, M. (1988). Bootstrap Type I error rates for the correlation coefficient: An examination of alternate procedures. Psychological Bulletin, 104, 290-292. doi:10.1037/0033-2909.104.2.290.
83. Tabachnick, B., & Fidell, L. (2007). Using multivariate statistics (5th ed.). Boston, MA: Allyn & Bacon/Pearson Education.
84. Tomarken, A., & Serlin, R. (1986). Comparison of ANOVA alternatives under variance heterogeneity and specific noncentrality structures. Psychological Bulletin, 99, 90-99. doi:10.1037/0033-2909.99.1.90.
85. Triola, M. (2010). Elementary statistics (11th ed.). Boston, MA: AddisonWesley/Pearson Education.
86. Tukey, J. W. (1962). The future of data analysis. Annals of Mathematical Statistics, 33, 1-67. doi:10.1214/aoms/1177704711.
87. van der Waerden, B. L. (1952). Order tests for the two-sample problem and their power. Indagationes Mathematicae, 14, 453-458.
88. Wampold, B., & Worsham, N. (1986). Randomization tests for multiplebaseline designs. Behavioral Assessment, 8, 135-143.
89. Wang, K., & Huang, J. (2002). A score-statistic approach for the mapping of quantitative-trait loci with sibships of arbitrary size. American Journal of Human Genetics, 70, 412-424. doi:10.1086/338659.
90. Warner, R. (2008). Applied statistics: From bivariate through multivariate techniques. Thousand Oaks, CA: Sage.
91. Witte, R., & Witte, J. (2010). Statistics (9th ed.). New York, NY: Wiley.
92. Yeo, I., & Johnson, R. A. (2000). A new family of power transformations to improve normality or symmetry. Biometrika, 87, 954-959. doi: 10.1093/biomet/87.4.954.
93. Zeller, R. A., & Levine, Z. H. (1974). The effects of violating the normality assumption underlying r. Sociological Methods & Research, 2, 511-519. doi:10.1177/004912417400200406.
94. Zimmerman, D. W. (1996). A note on homogeneity of variance of scores and ranks. Journal of Experimental Education, 64, 351-362.
95. Zimmerman, D. W. (2011). Inheritance of properties of normal and nonnormal distributions after transformation of scores to ranks. Psicológica, 32, 65-85.
96. Zimmerman, D. W., & Zumbo, B. D. (1993). Significance testing of correlation using scores, ranks, and modified ranks. Educational and Psychological Measurement, 53, 897-904. doi:10.1177/0013164493053004003.
97. Zimmerman, D., Zumbo, B. D., & Williams, R. H. (2003). Bias in estimation and hypothesis testing of correlation. Psicológica, 24(1), 133-158.