Equivalence Testing for Psychological Research: A Tutorial

Advances in Methods and Practices in Psychological Science

Volumn 1, Issue 2, 2018, Pages 259-269

  Lakens, Daniël ^a   Scheel, Anne M ^a   Isager, Peder M ^a  

^a EINDHOVEN UNIVERSITY OF TECHNOLOGY   (Netherlands)

Author keywords

equivalence testing; falsification; frequentist; null hypothesis; null hypothesis significance test; open materials; power; TOST

Indexed keywords

EID: 85119407590     PISSN: 25152459     EISSN: 25152467     Source Type: Journal    
DOI: 10.1177/2515245918770963     Document Type: Article

References (52)

1. Allaire J. Xie Y. McPherson J. Luraschi J. Ushey K. Atkins A... Chang W. (2017). rmarkdown: Dynamic documents for R (Version 1.7) [Computer software]. Retrieved from https://CRAN.R-project.org/package=rmarkdown
2. Auguie B. (2017). gridExtra: Miscellaneous functions for “grid” graphics (Version 2.3) [Computer software]. Retrieved from https://CRAN.R-project.org/package=gridExtra
3. Aust F. Barth M. (2017). papaja: Create APA manuscripts with R Markdown (Version 0.1.0.9492) [Computer software]. Retrieved from https://github.com/crsh/papaja
4. Baguley T. (2009). Standardized or simple effect size: What should be reported? British Journal of Psychology, 100, 603–617. doi:10.1348/000712608X377117
5. Banerjee P. Chatterjee P. Sinha J. (2012). Is it light or dark? Recalling moral behavior changes perception of brightness. Psychological Science, 23, 407–409. doi:10.1177/0956797611432497
6. Brandt M. J. IJzerman H. Blanken I. (2014). Does recalling moral behavior change the perception of brightness? A replication and meta-analysis of Banerjee, Chatterjee, and Sinha (2012). Social Psychology, 45, 246–252. doi:10.1027/1864-9335/a000191
7. Brown M. Rodriguez D. N. Gretak A. P. Berry M. A. (2017). Preliminary evidence for how the behavioral immune system predicts juror decision-making. Evolutionary Psychological Science, 3, 325–334. doi:10.1007/s40806-017-0102-z
8. Burriss R. P. Troscianko J. Lovell P. G. Fulford A. J. C. Stevens M. Quigley R... Rowland H. M. (2015). Changes in women’s facial skin color over the ovulatory cycle are not detectable by the human visual system. PLOS ONE, 10(7), Article e0130093. doi:10.1371/journal.pone.0130093
9. Button K. S. Kounali D. Thomas L. Wiles N. J. Peters T. J. Welton N. J... Lewis G. (2015). Minimal clinically important difference on the Beck Depression Inventory - II according to the patient’s perspective. Psychological Medicine, 45, 3269–3279. doi:10.1017/S0033291715001270
10. Central Intelligence Agency. (2016). The CIA World Factbook 2017. New York, NY: Skyhorse.
11. Champely S. (2017). pwr: Basic functions for power analysis (Version 1.2.1) [Computer software]. Retrieved from https://CRAN.R-project.org/package=pwr
12. Cohen J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
13. Delacre M. Lakens D. Leys C. (2017). Why psychologists should by default use Welch’s t-test instead of Student’s t-test. International Review of Social Psychology, 30, 92–101. doi:10.5334/irsp.82
14. Goertzen J. R. Cribbie R. A. (2010). Detecting a lack of association: An equivalence testing approach. British Journal of Mathematical and Statistical Psychology, 63, 527–537. doi:10.1348/000711009X475853
15. Hemphill J. F. (2003). Interpreting the magnitudes of correlation coefficients. American Psychologist, 58, 78–80.
16. Henry L. Wickham H. (2017). purrr: Functional programming tools (Version 0.2.4) [Computer software]. Retrieved from https://CRAN.R-project.org/package=purrr
17. Hyde J. S. Lindberg S. M. Linn M. C. Ellis A. B. Williams C. C. (2008). Gender similarities characterize math performance. Science, 321, 494–495. doi:10.1126/science.1160364
18. jamovi project. (2017). Jamovi (Version 0.8) [Computer software]. Retrieved from https://www.jamovi.org/download.html
19. Kahane G. Everett J. A. Earp B. D. Farias M. Savulescu J. (2015). “Utilitarian” judgments in sacrificial moral dilemmas do not reflect impartial concern for the greater good. Cognition, 134, 193–209. doi:10.1016/j.cognition.2014.10.005
20. Kordsmeyer T. L. Penke L. (2017). The association of three indicators of developmental instability with mating success in humans. Evolution & Human Behavior, 38, 704–713. doi:10.1016/j.evolhumbehav.2017.08.002
21. Lakens D. (2017a). Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. Social Psychological & Personality Science, 8, 355–362. doi:10.1177/1948550617697177
22. Lakens D. (2017b). TOSTER: Two one-sided tests (TOST) equivalence testing (Version 0.3) [Computer software]. Retrieved from https://CRAN.R-project.org/package=TOSTER
23. Lakens D. Adolfi F. G. Albers C. J. Anvari F. Apps M. A. J. Argamon S. E... Zwaan R. A. (2018). Justify your alpha. Nature Human Behavior, 2, 168–171. doi:10.1038/s41562-018-0311-x
24. Lenth R. V. (2007). Post hoc power: Tables and commentary. Iowa City: University of Iowa, Department of Statistics and Actuarial Science.
25. Lynott D. Corker K. S. Wortman J. Connell L. Donnellan M. B. Lucas R. E. O’Brien K. (2014). Replication of “Experiencing physical warmth promotes interpersonal warmth” by Williams and Bargh (2008). Social Psychology, 45, 216–222. doi:10.1027/1864-9335/a000187
26. Maxwell S. E. Lau M. Y. Howard G. S. (2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean? American Psychologist, 70, 487–498. doi:10.1037/a0039400
27. Meehl P. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806–834. doi:10.1037/0022-006X.46.4.806
28. Meyners M. (2012). Equivalence tests – a review. Food Quality and Preference, 26, 231–245. doi:10.1016/j.foodqual.2012.05.003
29. Moon A. Roeder S. S. (2014). A secondary replication attempt of stereotype susceptibility (Shih, Pittinsky, & Ambady, 1999). Social Psychology, 45, 199–201. doi:10.1027/1864-9335/a000193
30. Morey R. Lakens D. (2016). Why most of psychology is statistically unfalsifiable. doi:10.5281/zenodo.838685
31. Murphy K. R. Myors B. Wolach A. (2014). Statistical power analysis: A simple and general model for traditional and modern hypothesis tests (4th ed.). New York, NY: Routledge.
32. Neyman J. Pearson E. S. (1933). On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, 231, 289–337. doi:10.1098/rsta.1933.0009
33. Norman G. R. Sloan J. A. Wyrwich K. W. (2003). Interpretation of changes in health-related quality of life: The remarkable universality of half a standard deviation. Medical Care, 41, 582–592.
34. Perugini M. Gallucci M. Costantini G. (2014). Safeguard power as a protection against imprecise power estimates. Perspectives on Psychological Science, 9, 319–332. doi:10.1177/1745691614528519
35. Piaggio G. Elbourne D. R. Altman D. G. Pocock S. J. Evans S. J. W. & the CONSORT Group. (2006). Reporting of noninferiority and equivalence randomized trials: An extension of the CONSORT Statement. Journal of the American Medical Association, 295, 1152–1160. doi:10.1001/jama.295.10.1152
36. Platt J. R. (1964). Strong inference. Science, 146, 347–353.
37. Quertemont E. (2011). How to statistically show the absence of an effect. Psychologica Belgica, 51, 109–127. doi:10.5334/pb-51-2-109
38. R Core Team. (2017). R: A language and environment for statistical computing (Version 3.4.2) [Computer software]. Retrieved from https://www.R-project.org/
39. Rogers J. L. Howard K. I. Vessey J. T. (1993). Using significance tests to evaluate equivalence between two ex-perimental groups. Psychological Bulletin, 113, 553–565.
40. RStudio Team. (2016). RStudio: Integrated development environment for R (Version 1.1.383) [Computer software]. Boston, MA: RStudio.
41. Sarkar D. (2008). Lattice: Multivariate data visualization with R (Version 0.20.35) [Computer software]. Retrieved from http://lmdvr.r-forge.r-project.org
42. Schumann S. Klein O. Douglas K. Hewstone M. (2017). When is computer-mediated intergroup contact most promising? Examining the effect of out-group members’ anonymity on prejudice. Computers in Human Behavior, 77, 198–210. doi:10.1016/j.chb.2017.08.006
43. Seaman M. A. Serlin R. C. (1998). Equivalence confidence intervals for two-group comparisons of means. Psychological Methods, 3, 403–411. doi:10.1037/1082-989X.3.4.403
44. Senn S. (2007). Statistical issues in drug development (2nd ed.). Chichester, England: John Wiley & Sons.
45. Shih M. Pittinsky T. L. Ambady N. (1999). Stereotype susceptibility: Identity salience and shifts in quantitative performance. Psychological Science, 10, 80–83. doi:10.1111/1467-9280.00111
46. Simonsohn U. (2015). Small telescopes: Detectability and the evaluation of replication results. Psychological Science, 26, 559–569. doi:10.1177/0956797614567341
47. Tryon W. W. (2001). Evaluating statistical difference, equivalence, and indeterminacy using inferential confidence intervals: An integrated alternative method of conducting null hypothesis statistical tests. Psychological Methods, 6, 371–386.
48. Weber R. Popova L. (2012). Testing equivalence in communication research: Theory and application. Communication Methods and Measures, 6, 190–213. doi:10.1080/19312458.2012.703834
49. Wickham H. (2009). ggplot2: Elegant graphics for data analysis. New York, NY: Springer-Verlag.
50. Williams L. E. Bargh J. A. (2008). Experiencing physical warmth promotes interpersonal warmth. Science, 322, 606–607. doi:10.1126/science.1162548
51. Xie Y. (2015). knitr: Elegant, flexible, and fast dynamic report generation with R. Retrieved from https://yihui.name/knitr/
52. Xie Y. (2016). bookdown: Authoring books and technical documents with R Markdown. Retrieved from https://github.com/rstudio/bookdown