Performing Contrast Analysis in Factorial Designs: From NHST to Confidence Intervals and Beyond

Educational and Psychological Measurement

Volumn 77, Issue 4, 2017, Pages 690-715

  Wiens, Stefan ^a   Nilsson, Mats E ^a  

^a STOCKHOLM UNIVERSITY   (Sweden)

Author keywords

analysis of variance; Bayesian analysis; confidence interval; contrast analysis; null hypothesis significance testing

Indexed keywords

EID: 85025102909     PISSN: 00131644     EISSN: 15523888     Source Type: Journal    
DOI: 10.1177/0013164416668950     Document Type: Article

References (97)

1. Abelson R. P., (1996). Vulnerability of contrast tests to simpler interpretations: An addendum to Rosnow and Rosenthal. Psychological Science, 7, 242-246. doi:10.1111/j.1467-9280.1996.tb00367.x
2. Abelson R. P., Prentice D. A., (1997). Contrast tests of interaction hypotheses. Psychological Methods, 2, 315-328. doi:10.1037/1082-989x.2.4.315
3. American Psychological Association. (2009). Publication manual of the American Psychological Association (6th ed.). Washington, DC: Author.
4. Andraszewicz S., Scheibehenne B., Rieskamp J., Grasman R., Verhagen J., Wagenmakers E.-J., (2015). An introduction to Bayesian hypothesis testing for management research. Journal of Management, 41, 521-543. doi:10.1177/0149206314560412
5. Baguley T., (2009). Standardized or simple effect size: What should be reported? British Journal of Psychology, 100, 603-617. doi:10.1348/000712608x377117
6. Baguley T., (2012). Calculating and graphing within-subject confidence intervals for ANOVA. Behavior Research Methods, 44, 158-175. doi:10.3758/s13428-011-0123-7
7. Bird K. D., (2002). Confidence intervals for effect sizes in analysis of variance. Educational and Psychological Measurement, 62, 197-226. doi:10.1177/0013164402062002001
8. Bushman B. J., Anderson C. A., (2009). Comfortably numb: Desensitizing effects of violent media on helping others. Psychological Science, 20, 273-277. doi:10.1111/j.1467-9280.2009.02287.x
9. Chavalarias D., Wallach J., Li A., Ioannidis J. A., (2016). Evolution of reporting p values in the biomedical literature, 1990-2015. JAMA, 315, 1141-1148. doi:10.1001/jama.2016.1952
10. Colegrave N., Ruxton G. D., (2003). Confidence intervals are a more useful complement to nonsignificant tests than are power calculations. Behavioral Ecology, 14, 446-447. doi:10.1093/beheco/14.3.446
11. Cousineau D., O’Brien F., (2014). Error bars in within-subject designs: a comment on Baguley (2012). Behavior Research Methods, 46, 1-3. doi:10.3758/s13428-013-0441-z
12. Cumming G., (2008). Replication and p intervals p values predict the future only vaguely, but confidence intervals do much better. Perspectives on Psychological Science, 3, 286-300. doi:10.1111/j.1745-6924.2008.00079.x
13. Cumming G., (2012). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. New York, NY: Routledge.
14. Cumming G., (2014). The new statistics: Why and how. Psychological Science, 25, 7-29. doi:10.1177/0956797613504966
15. Cumming G., Fidler F., (2009). Confidence intervals: Better answers to better questions. Zeitschrift für Psychologie–Journal of Psychology, 217, 15-26. doi:10.1027/0044-3409.217.1.15
16. Cumming G., Fidler F., Kalinowski P., Lai J., (2012). The statistical recommendations of the American Psychological Association publication manual: Effect sizes, confidence intervals, and meta-analysis. Australian Journal of Psychology, 64, 138-146. doi:10.1111/j.1742-9536.2011.00037.x
17. Cumming G., Finch S., (2005). Inference by eye: Confidence intervals and how to read pictures of data. American Psychologist, 60, 170-180. doi:10.1037/0003-066x.60.2.170
18. Cumming G., Maillardet R., (2006). Confidence intervals and replication: Where will the next mean fall? Psychological Methods, 11, 217-227. doi:10.1037/1082-989X.11.3.217
19. Dienes Z., (2008). Understanding psychology as a science: An introduction to scientific and statistical inference. New York, NY: Palgrave Macmillan.
20. Dienes Z., (2011). Bayesian versus orthodox statistics: Which side are you on? Perspectives on Psychological Science, 6, 274-290. doi:10.1177/1745691611406920
21. Dienes Z., (2014). Using Bayes to get the most out of non-significant results. Frontiers in Psychology, 5:781. doi:10.3389/fpsyg.2014.00781
22. Dienes Z., (2016). How Bayes factors change scientific practice. Journal of Mathematical Psychology, 72, 78-89. doi:10.1016/j.jmp.2015.10.003
23. Efron B., Tibshirani R. J., (1994). An introduction to the bootstrap. Boca Raton, FL: Chapman & Hall.
24. Faulkner C., Fidler F., Cumming G., (2008). The value of RCT evidence depends on the quality of statistical analysis. Behaviour Research and Therapy, 46, 270-281. doi:10.1016/j.brat.2007.12.001
25. Feinstein A. R., (1998). P-values and confidence intervals: Two sides of the same unsatisfactory coin. Journal of Clinical Epidemiology, 51, 355-360. doi:10.1016/s0895-4356(97)00295-3
26. Fidler F., Loftus G. R., (2009). Why figures with error bars should replace p values. Zeitschrift für Psychologie–Journal of Psychology, 217, 27-37. doi:10.1027/0044-3409.217.1.27
27. Finch S., Cumming G., (2009). Putting research in context: Understanding confidence intervals from one or more studies. Journal of Pediatric Psychology, 34, 903-916. doi:10.1093/jpepsy/jsn118
28. Franz V. H., Loftus G. R., (2012). Standard errors and confidence intervals in within-subjects designs: generalizing Loftus and Masson (1994) and avoiding the biases of alternative accounts. Psychonomic Bulletin & Review, 19, 395-404. doi:10.3758/s13423-012-0230-1
29. Fritz C. O., Morris P. E., Richler J. J., (2012). Effect size estimates: Current use, calculations, and interpretation. Journal of Experimental Psychology. General, 141, 2-18. doi:10.1037/a0024338
30. Funder D. C., Levine J. M., Mackie D. M., Morf C. C., Sansone C., Vazire S., West S. G., (2014). Improving the dependability of research in personality and social psychology: Recommendations for research and educational practice. Personality and Social Psychology Review, 18, 3-12. doi:10.1177/1088868313507536
31. Furr R. M., Rosenthal R., (2003). Evaluating theories efficiently: The nuts and bolts of contrast analysis. Understanding Statistics, 2, 45-67.
32. Gallistel C. R., (2009). The importance of proving the null. Psychological Review, 116, 439-453. doi:10.1037/a0015251
33. Gigerenzer G., (2004). Mindless statistics. Journal of Socio-Economics, 33, 587-606. doi:10.1016/j.socec.2004.09.033
34. Glover S., Dixon P., (2004). Likelihood ratios: A simple and flexible statistic for empirical psychologists. Psychonomic Bulletin & Review, 11, 791-806. doi:10.3758/bf03196706
35. Goodman S. N., (1999a). Toward evidence-based medical statistics. 1: The p value fallacy. Annals of Internal Medicine, 130, 995-1004.
36. Goodman S. N., (1999b). Toward evidence-based medical statistics. 2: The Bayes factor. Annals of Internal Medicine, 130, 1005-1013.
37. Halsey L. G., Curran-Everett D., Vowler S. L., Drummond G. B., (2015). The fickle p value generates irreproducible results. Nature Methods, 12, 179-185. doi:10.1038/nmeth.3288
38. Hoekstra R., Johnson A., Kiers H. A. L., (2012). Confidence intervals make a difference: Effects of showing confidence intervals on inferential reasoning. Educational and Psychological Measurement, 72, 1039-1052. doi:10.1177/0013164412450297
39. Hoekstra R., Morey R. D., Rouder J. N., Wagenmakers E.-J., (2014). Robust misinterpretation of confidence intervals. Psychonomic Bulletin & Review, 21, 1157-1164. doi:10.3758/s13423-013-0572-3
40. Howell D. C., (2013). Statistical methods for psychology (8th ed.). Belmont, CA: Wadsworth/Cengage Learning.
41. Hubbard R., (2004). Alphabet soup: Blurring the distinctions between p’s and alpha’s in psychological research. Theory & Psychology, 14, 295-327.
42. Hunter J. E., (1997). Needed: A ban on the significance test. Psychological Science, 8, 3-7. doi:10.1111/j.1467-9280.1997.tb00534.x
43. Ivarsson A., Andersen M. B., Stenling A., Johnson U., Lindwall M., (2015). Things we still haven’t learned (so far). Journal of Sport & Exercise Psychology, 37, 449-461. doi:10.1123/jsep.2015-0015
44. Jaccard J., Guilamo-Ramos V., (2002). Analysis of variance frameworks in clinical child and adolescent psychology: Issues and recommendations. Journal of Clinical Child and Adolescent Psychology, 31, 130-146. doi:10.1207/153744202753441747
45. Johansson T., (2011). Hail the impossible: P-values, evidence, and likelihood. Scandinavian Journal of Psychology, 52, 113-125. doi:10.1111/j.1467-9450.2010.00852.x
46. Kalinowski P., Fidler F., (2010). Interpreting significance: The differences between statistical significance, effect size, and practical importance. Newborn and Infant Nursing Reviews, 10, 50-54. doi:10.1053/j.nainr.2009.12.007
47. Kass R. E., Raftery A. E., (1995). Bayes factors. Journal of the American Statistical Association, 90, 773-795. doi:10.1080/01621459.1995.10476572
48. Kirk R. E., (1996). Practical significance: A concept whose time has come. Educational And Psychological Measurement, 56, 746-759. doi:10.1177/0013164496056005002
49. Kline R. B., (2013). Beyond significance testing: Statistics reform in the behavioral science (2nd ed.). Washington, DC: American Psychological Association.
50. Kruschke J. K., (2010). Bayesian data analysis. Wiley Interdisciplinary Reviews: Cognitive Science, 1, 658-676. doi:10.1002/wcs.72
51. Lai J., Fidler F., Cumming G., (2012). Subjective p intervals: Researchers underestimate the variability of p values over replication. Methodology–European Journal of Research Methods for the Behavioral and Social Sciences, 8, 51-62. doi:10.1027/1614-2241/a000037
52. Lenth R. V., (2001). Some practical guidelines for effective sample size determination. The American Statistician, 55, 187-193. doi:10.1198/000313001317098149
53. Lew M. J., (2012). Bad statistical practice in pharmacology (and other basic biomedical disciplines): You probably don’t know p. British Journal of Pharmacology, 166, 1559-1567. doi:10.1111/j.1476-5381.2012.01931.x
54. Loftus G. R., Masson M. E. J., (1994). Using confidence-intervals in within-subject designs. Psychonomic Bulletin & Review, 1, 476-490. doi:10.3758/BF03210951
55. Love J., Selker R., Marsman M., Jamil T., Dropmann D., Verhagen A. J., … Wagenmakers E.-J., (2015). JASP (Version 0.7.5) [Computer software].
56. McCormack J., Vandermeer B., Allan G. M., (2013). How confidence intervals become confusion intervals. BMC Medical Research Methodology, 13, 134. doi:10.1186/1471-2288-13-134
57. Morey R. D., Hoekstra R., Rouder J. N., Lee M. D., Wagenmakers E.-J., (2016). The fallacy of placing confidence in confidence intervals. Psychonomic Bulletin & Review, 23, 103-123. doi:10.3758/s13423-015-0947-8
58. Morey R. D., Rouder J. N., Verhagen J., Wagenmakers E.-J., (2014). Why hypothesis tests are essential for psychological science: A comment on Cumming (2014). Psychological Science, 25, 1289-1290. doi:10.1177/0956797614525969
59. Nakagawa S., Cuthill I. C., (2007). Effect size, confidence interval and statistical significance: A practical guide for biologists. Biological Reviews, 82, 591-605. doi:10.1111/j.1469-185X.2007.00027.x
60. Neyman J., (1935). On the problem of confidence intervals. Annals of Mathematical Statistics, 6, 111-116.
61. Neyman J., (1937). Outline of a theory of statistical estimation based on the classical theory of probability. Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, 236, 333-380. doi:10.1098/rsta.1937.0005
62. Noguchi K., Marmolejo-Ramos F., (2016). Assessing equality of means using the overlap of range-preserving confidence intervals. The American Statistician. Advance online publication. doi:10.1080/00031305.2016.1200487
63. Nuzzo R., (2014). Statistical errors. Nature, 506, 150-152.
64. Nuzzo R., (2015). How scientists fool themselves-and how they can stop. Nature, 526, 182-185.
65. O’Brien F., Cousineau D., (2014). Representing error bars in within-subject designs in typical software packages. Quantitative Methods for Psychology, 10, 56-67.
66. Perezgonzalez J. D., (2015). Fisher, Neyman-Pearson or NHST? A tutorial for teaching data testing. Frontiers in Psychology, 6, 223. doi:10.3389/fpsyg.2015.00223
67. Petty R. E., Fabrigar L. R., Wegener D. T., Priester J. R., (1996). Understanding data when interactions are present or hypothesized. Psychological Science, 7, 247-252. doi:10.1111/j.1467-9280.1996.tb00368.x
68. Pfister R., Janczyk M., (2013). Confidence intervals for two sample means: Calculation, interpretation, and a few simple rules. Advances in Cognitive Psychology, 9, 74-80. doi:10.2478/v10053-008-0133-x
69. Richter M., (2016). Residual tests in the analysis of planned contrasts: Problems and solutions. Psychological Methods, 21, 112-120. doi:10.1037/met0000044
70. Rosenthal R., Rosnow R. L., (1991). Essentials of behavioral research: Methods and data analysis (2nd ed.). New York, NY: McGraw-Hill.
71. Rosenthal R., Rosnow R. L., Rubin D. B., (1999). Contrasts and effect sizes in behavioral research. Cambridge, England: Cambridge University Press.
72. Rosnow R. L., Rosenthal R., (1995). Some things you learn aren’t so—Cohen’s paradox, Asch’s paradigm, and the interpretation of interaction. Psychological Science, 6, 3-9. doi:10.1111/j.1467-9280.1995.tb00297.x
73. Rosnow R. L., Rosenthal R., (1996). Contrasts and interactions redux: Five easy pieces. Psychological Science, 7, 253-257. doi:10.1111/j.1467-9280.1996.tb00369.x
74. Rotteveel M., Gierholz A., Koch G., van Aalst C., Pinto Y., Matzke D., … Wagenmakers E.-J., (2015). On the automatic link between affect and tendencies to approach and avoid: Chen and Bargh (1999) revisited. Frontiers in Psychology, 6, 335. doi:10.3389/fpsyg.2015.00335
75. Rouder J. N., Morey R. D., (2005). Relational and arelational confidence intervals: A comment on Fidler, Thomasow, Cumming, Finch, and Leeman (2004). Psychological Science, 16, 77-79. doi:10.1111/j.0956-7976.2005.00783.x
76. Rouder J. N., Speckman P. L., Sun D., Morey R. D., Iverson G., (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16, 225-237. doi:10.3758/pbr.16.2.225
77. Savalei V., Dunn E., (2015). Is the call to abandon p-values the red herring of the replicability crisis? Frontiers in Psychology, 6, 245. doi:10.3389/fpsyg.2015.00245
78. Sharpe D., (2013). Why the resistance to statistical innovations? Bridging the communication gap. Psychological Methods, 18, 572-582. doi:10.1037/a0034177
79. Thompson B., (2002). “Statistical,” “practical,” and “clinical”: How many kinds of significance do counselors need to consider? Journal of Counseling and Development, 80, 64-71.
80. Torgerson W. S., (1965). Multidimensional-scaling of similarity. Psychometrika, 30, 379-393. doi:10.1007/bf02289530
81. Trafimow D., (2014). Editorial. Basic and Applied Social Psychology, 36, 1-2. doi:10.1080/01973533.2014.865505
82. Trafimow D., Marks M., (2015). Editorial. Basic and Applied Social Psychology, 37, 1-2. doi:10.1080/01973533.2015.1012991
83. Tryon W. W., (2001). Evaluating statistical difference, equivalence, and indeterminacy using inferential confidence intervals: An integrated alternative method of conducting null hypothesis statistical tests. Psychological Methods, 6, 371-386. doi:10.1037//1082-989x.6.4.371
84. Valentine J. C., Aloe A. M., Lau T. S., (2015). Life after NHST: How to describe your data without “p-ing” everywhere. Basic and Applied Social Psychology, 37, 260-273. doi:10.1080/01973533.2015.1060240
85. van de Schoot R., Kaplan D., Denissen J., Asendorpf J. B., Neyer F. J., van Aken M. A. G., (2014). A gentle introduction to Bayesian analysis: Applications to developmental research. Child Development, 85, 842-860. doi:10.1111/cdev.12169
86. Wagenmakers E.-J., (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14, 779-804. doi:10.3758/bf03194105
87. Wagenmakers E.-J., Beek T. F., Rotteveel M., Gierholz A., Matzke D., Steingroever H., … Pinto Y., (2015). Turning the hands of time again: A purely confirmatory replication study and a Bayesian analysis. Frontiers in Psychology, 6, 494. doi:10.3389/fpsyg.2015.00494
88. Wagenmakers E.-J., Lodewyckx T., Kuriyal H., Grasman R., (2010). Bayesian hypothesis testing for psychologists: A tutorial on the Savage-Dickey method. Cognitive Psychology, 60, 158-189. doi:10.1016/j.cogpsych.2009.12.001
89. Wagenmakers E.-J., Morey R. D., Lee M. D., (2016). Bayesian benefits for the pragmatic researcher. Current Directions in Psychological Science, 25, 169-176. doi:10.1177/0963721416643289
90. Wagenmakers E.-J., Verhagen J., Ly A., Bakker M., Lee M. D., Matzke D., … Morey R. D., (2015). A power fallacy. Behavior Research Methods, 47, 913-917. doi:10.3758/s13428-014-0517-4
91. Wagenmakers E.-J., Verhagen J., Ly A., Matzke D., Steingroever H., Rouder J. N., Morey R. D., (2014). The need for Bayesian hypothesis testing in psychological science. In Lilienfeld S. O., Waldman I., (Eds.), Psychological science under scrutiny: Recent challenges and proposed solutions. New York, NY: Wiley.
92. Wagenmakers E.-J., Wetzels R., Borsboom D., van der Maas H. L., (2011). Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of Personality and Social Psychology, 100, 426-432. doi:10.1037/a0022790
93. Walker E., Nowacki A. S., (2011). Understanding equivalence and noninferiority testing. Journal of General Internal Medicine, 26, 192-196. doi:10.1007/s11606-010-1513-8
94. Wasserstein R. L., Lazar N. A., (2016). The ASA’s statement on p-values: Context, process, and purpose. The American Statistician. doi:10.1080/00031305.2016.1154108
95. Wetzels R., Matzke D., Lee M. D., Rouder J. N., Iverson G. J., Wagenmakers E.-J., (2011). Statistical evidence in experimental psychology: An empirical comparison using 855 t tests. Perspectives on Psychological Science, 6, 291-298. doi:10.1177/1745691611406923
96. Wilkinson L., (1999). Statistical methods in psychology journals—Guidelines and explanations. American Psychologist, 54, 594-604.
97. Zyphur M. J., Oswald F. L., (2015). Bayesian estimation and inference: A user’s guide. Journal of Management, 41, 390-420. doi:10.1177/0149206313501200