Small sample mediation testing: Misplaced confidence in bootstrapped confidence Intervals

Journal of Applied Psychology

Volumn 100, Issue 1, 2015, Pages 194-202

  Koopman, Joel ^a   Howe, Michael ^a   Hollenbeck, John R ^a   Sin, Hock Peng ^b  

^a MICHIGAN STATE UNIVERSITY   (United States)

^b FLORIDA INTERNATIONAL UNIVERSITY   (United States)

Author keywords

Bayes; Bootstrapping; Mediation; Permutation

Indexed keywords

BAYES THEOREM; CONFIDENCE INTERVAL; HUMAN; PROCEDURES; PSYCHOLOGY; SAMPLE SIZE; STATISTICAL ANALYSIS;

BAYES THEOREM; CONFIDENCE INTERVALS; DATA INTERPRETATION, STATISTICAL; HUMANS; PSYCHOLOGY, APPLIED; SAMPLE SIZE;

EID: 84925642741     PISSN: 00219010     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0036635     Document Type: Article

References (53)

1. Baron, R. M., & Kenny, D. A. (1986). The moderator mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51, 1173-1182. doi:10.1037/0022-3514.51.6.1173
2. Bayarri, M. J., & Berger, J. O. (2004). The interplay of Bayesian and frequentist analysis. Statistical Science, 19, 58 - 80. doi:10.1214/088342304000000116
3. Biesanz, J. C., Falk, C. F., & Savalei, V. (2010). Assessing mediational models: Testing and interval estimation for indirect effects. Multivariate Behavioral Research, 45, 661-701. doi:10.1080/00273171.2010.498292
4. Bishara, A. J., & Hittner, J. B. (2012). Testing the significance of a correlation with nonnormal data: Comparison of Pearson, Spearman, transformation, and resampling approaches. Psychological Methods, 17, 399-417. doi:10.1037/a0028087
5. Canty, A., & Ripley, B. (2013). Boot: Bootstrap R (S-plus) functions. (R Package Version 1.3-9). Vienna, Austria: R Foundation for Statistical Computing.
6. Cheung, G. W., & Lau, R. S. (2008). Testing mediation and suppression effects of latent variables: Bootstrapping with structural equation models. Organizational Research Methods, 11, 296 -325. doi:10.1177/1094428107300343
7. Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45, 1304-1312. doi:10.1037/0003-066X.45.12.1304
8. Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155-159. doi:10.1037/0033-2909.112.1.155
9. Colquitt, J. A., LePine, J. A., Zapata, C. P., & Wild, R. E. (2011). Trust in typical and high-reliability contexts: Building and reacting to trust among firefighters. Academy of Management Journal, 54, 999-1015. doi:10.5465/amj.2006.0241
10. Combs, J. G. (2010). Big samples and small effects: Let's not trade relevance and rigor for power. Academy of Management Journal, 53, 9-13. doi:10.5465/AMJ.2010.48036305
11. Edwards, J. R. (2008). To prosper, organizational psychology should . . . overcome methodological barriers to progress. Journal of Organizational Behavior, 29, 469-491. doi:10.1002/job.529
12. Efron, B. (1979). Bootstrap methods: Another look at the jackknife. Annals of Statistics, 7, 1-26. doi:10.1214/aos/1176344552
13. Efron, B. (1990). Six questions raised by the bootstrap (Technical Report No. 139). Stanford, CA: Stanford University, Division of Biostatistics. Efron, B. (2000). The bootstrap and modern statistics. Journal of the American Statistical Association, 95, 1293-1296. doi:10.1080/01621459 2000.10474333
14. Efron, B., & Tibshirani, R. (1993). An introduction to the bootstrap. New York, NY: Chapman & Hall/CRC. doi:10.1007/978-1-4899-4541-9
15. Fritz, M. S., Taylor, A. B., & MacKinnon, D. P. (2012). Explanation of two anomalous results in statistical mediation analysis. Multivariate Behavioral Research, 47, 61-87. doi:10.1080/00273171.2012.640596
16. Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2003). Bayesian data analysis (2nd ed.). New York, NY: Chapman & Hall/CRC.
17. Gleser, L. J. (1996). Comment on "bootstrap confidence intervals" by T. J. Diciccio and B. Efron. Statistical Science, 11, 219-221.
18. Good, P. (2005). Permutation, parametric, and bootstrap tests of hypotheses (3rd ed.). New York, NY: Springer-Verlag.
19. Hollenbeck, J. R., DeRue, D. S., & Mannor, M. (2006). Statistical power and parameter stability when subjects are few and tests are many: Comment on Peterson, Smith, Martorana, and Owens (2003). Journal of Applied Psychology, 91, 1-5. doi:10.1037/0021-9010.91.1.1
20. James, L. R., & Brett, J. M. (1984). Mediators, moderators, and tests for mediation. Journal of Applied Psychology, 69, 307-321. doi:10.1037/0021-9010.69.2.307
21. James, L. R., Mulaik, S. A., & Brett, J. M. (2006). A tale of two methods. Organizational Research Methods, 9, 233-244. doi:10.1177/1094428105285144
22. Koopman, J., Howe, M., & Hollenbeck, J. R. (in press). Pulling the Sobel test up by its bootstraps. In C. E. Lance & R. J. Vandenberg (Eds.), More statistical and methodological myths and urban legends. New York, NY: Routledge.
23. Kozlowski, S. W. J. (2009). Editorial. Journal of Applied Psychology, 94, 1-4. doi:10.1037/a0014990
24. Kruschke, J. K., Aguinis, H., & Joo, H. (2012). The time has come: Bayesian methods for data analysis in the organizational sciences. Organizational Research Methods, 15, 722-752. doi:10.1177/1094428112457829
25. Lance, C. E. (2011). More statistical and methodological myths and urban legends. Organizational Research Methods, 14, 279-286. doi:10.1177/1094428110391814
26. MacKinnon, D. P., Lockwood, C. M., & Williams, J. (2004). Confidence limits for the indirect effect: Distribution of the product and resampling methods. Multivariate Behavioral Research, 39, 99-128. doi:10.1207/s15327906mbr3901_4
27. Martin, A. D., Quinn, K. M., & Park, J. H. (2011). MCMCpack: Markov chain Monte Carlo in R. Journal of Statistical Software, 42, 1-21.
28. Maxwell, S. E. (2004). The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods, 9, 147-163. doi:10.1037/1082-989X.9.2.147
29. Newman, D. A., Jacobs, R. R., & Bartram, D. (2007). Choosing the best method for local validity estimation: Relative accuracy of meta-analysis versus a local study versus Bayes-analysis. Journal of Applied Psychology, 92, 1394-1413. doi:10.1037/0021-9010.92.5.1394
30. Pituch, K. A., & Stapleton, L. M. (2008). The performance of methods to test upper level mediation in the presence of nonnormal data. Multivariate Behavioral Research, 43, 237-267. doi:10.1080/00273170802034844
31. Pituch, K. A., Stapleton, L. M., & Kang, J. Y. (2006). A comparison of single sample and bootstrap methods to assess mediation in cluster randomized trials. Multivariate Behavioral Research, 41, 367-400. doi:10.1207/s15327906mbr4103_5
32. Preacher, K. J., & Hayes, A. F. (2004). SPSS and SAS procedures for estimating indirect effects in simple mediation models. Behavior Research Methods, Instruments & Computers, 36, 717-731. doi:10.3758/BF03206553
33. Preacher, K. J., & Hayes, A. F. (2008). Asymptotic and resampling strategies for assessing and comparing indirect effects in multiple mediator models. Behavior Research Methods, 40, 879-891. doi:10.3758/BRM.40.3.879
34. Preacher, K. J., Zyphur, M. J., & Zhang, Z. (2010). A general multilevel SEM framework for assessing multilevel mediation. Psychological Methods, 15, 209-233. doi:10.1037/a0020141
35. R Core Team. (2012). R: A language and environment for statistical computing (Version 2.15.1) [Computer software]. Vienna, Austria: R Foundation for Statistical Computing.
36. Schmidt, F. L., Hunter, J. E., & Pearlman, K. (1981). Task differences as moderators of aptitude-test validity in selection: A red herring. Journal of Applied Psychology, 66, 166-185. doi:10.1037/0021-9010.66.2.166
37. Shen, W. N., Kiger, T. B., Davies, S. E., Rasch, R. L., Simon, K. M., & Ones, D. S. (2011). Samples in applied psychology: Over a decade of research in review. Journal of Applied Psychology, 96, 1055-1064. doi:10.1037/a0023322
38. Shrout, P. E., & Bolger, N. (2002). Mediation in experimental and nonexperimental studies: New procedures and recommendations. Psychological Methods, 7, 422-445. doi:10.1037/1082-989X.7.4.422
39. Sobel, M. E. (1982). Asymptotic confidence intervals for indirect effects in structural equation models. Sociological Methodology, 13, 290-312. doi:10.2307/270723
40. Stone, C. A., & Sobel, M. E. (1990). The robustness of estimates of total indirect effects in covariance structure models estimated by maximum likelihood. Psychometrika, 55, 337-352. doi:10.1007/BF02295291
41. Stone-Romero, E. F., & Rosopa, P. J. (2008). The relative validity of inferences about mediation as a function of research design characteristics. Organizational Research Methods, 11, 326-352. doi:10.1177/1094428107300342
42. Taylor, A. B., & MacKinnon, D. P. (2012). Four applications of permutation methods to testing a single-mediator model. Behavior Research Methods, 44, 806-844. doi:10.3758/s13428-011-0181-x
43. Taylor, A. B., MacKinnon, D. P., & Tein, J. Y. (2008). Tests of the three-path mediated effect. Organizational Research Methods, 11, 241-269. doi:10.1177/1094428107300344
44. Tversky, A., & Kahneman, D. (1971). Belief in the law of small numbers. Psychological Bulletin, 76, 105-110. doi:10.1037/h0031322
45. Tversky, A., & Kahneman, D. (1974, September 27). Judgment under uncertainty: Heuristics and biases. Science, 185, 1124 -1131. doi:10.1126/science.185.4157.1124
46. Venables, W. N., & Ripley, B. D. (2002). Modern statistics with S (4th ed.). New York, NY: Springer. doi:10.1007/978-0-387-21706-2
47. Wang, J., Carpenter, J. R., & Kepler, M. A. (2006). Using SAS to conduct nonparametric residual bootstrap multilevel modeling with a small number of groups. Computer Methods and Programs in Biomedicine, 82, 130-143. doi:10.1016/j.cmpb.2006.02.006
48. Wang, M., Liu, S. Q., Liao, H., Gong, Y. P., Kammeyer-Mueller, J., & Shi, J. Q. (2013). Can't get it out of my mind: Employee rumination after customer mistreatment and negative mood in the next morning. Journal of Applied Psychology, 98, 989-1004. doi:10.1037/a0033656
49. Williams, L. J., Edwards, J. R., & Vandenberg, R. J. (2003). Recent advances in causal modeling methods for organizational and management research. Journal of Management, 29, 903-936. doi:10.1016/S0149-2063(03)00084-9
50. Young, M. J., Bauman, C. W., Chen, N., & Bastardi, A. (2012). The pursuit of missing information in negotiation. Organizational Behavior and Human Decision Processes, 117, 88-95. doi:10.1016/j.obhdp.2011.09.004
51. Yuan, Y., & MacKinnon, D. P. (2009). Bayesian mediation analysis. Psychological Methods, 14, 301-322. doi:10.1037/a0016972
52. Zickar, M. J., & Highhouse, S. (2001). Measuring prestige of journals in industrial- organizational psychology. Industrial-Organizational Psychologist, 38, 29-36.
53. Zyphur, M. J., & Oswald, F. L. (in press). Bayesian estimation and inference: A user's guide. Journal of Management