Rethinking statistical analysis methods for CHI

Conference on Human Factors in Computing Systems - Proceedings

Volumn , Issue , 2012, Pages 1105-1113

  Kaptein, Maurits ^a   Robertson, Judy ^b  

^a EINDHOVEN UNIVERSITY OF TECHNOLOGY   (Netherlands)

^b HERIOT WATT UNIVERSITY   (United Kingdom)

Author keywords

Bayesian statistics; Research methods; Usability evaluation

Indexed keywords

BAYESIAN STATISTICS; EMPIRICAL STUDIES; P-VALUES; RESEARCH METHODS; SIGNIFICANCE TESTING; STATISTICAL SIGNIFICANCE; TESTING HYPOTHESIS; THREE PROBLEMS; USABILITY EVALUATION;

HUMAN ENGINEERING; RESEARCH;

STATISTICAL METHODS;

EID: 84862090713     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2207676.2208557     Document Type: Conference Paper

References (32)

1. American Psychological Association. (2009). Publication Manual of the American Psychological Association, Sixth Edition (6th ed., p. 272). APA.
2. Bargas-Avila, J., & Hornbæk, K. (2011). Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience. In Proceedings of the 2011 annual conference on Human factors in computing systems (CHI '11). ACM, New York, NY, USA, 2689-2698.
3. Bayes, T., & Price. (1763). An Essay towards Solving a Problem in the Doctrine of Chances. By the Late Rev. Mr. Bayes, F. R. S. Communicated by Mr. Price, in a Letter to John Canton, A. M. F. R. S. Philosophical Transactions of the Royal Society of London, 53, 370-418.
4. Cairns, P. (2007). HCI... not as it should be: inferential statistics in HCI research. Proceedings of the 21st British HCI Group Annual Conference on People and Computers: HCI... but not as we know it-Volume 1 (pp. 195-201). British Computer Society.
5. Champely, S. (2009). PWR package. Retrieved from http://cran.r-project. org/web/packages/pwr/index.html
6. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, New Jersey: Laurence Erlbaum Associates.
7. Cohen, J. (1992). A power primer. Psychological bulletin, 112(1), 155. American Psychological Association.
8. Cohen, J. (1994). The Earth is Round (p < .05). American Psychologist, 49(12), 997-1003.
9. Dienes, Z. (2008). Understanding Psychology as a Science: An Introduction to Scientific and Statistical Inference (1st ed., p. 150). Palgrave Macmillan.
10. Dienes, Z. (2011). Bayesian Versus Orthodox Statistics: Which Side Are You On? Perspectives on Psychological Science, 6(3), 274-290.
11. Dunlop, M. & Baillie, M. (2009). Paper Rejected (p<; 0.05): An Introduction to the Debate on Appropriateness of Null-Hypothesis Testing. International Journal of Mobile Human Computer Interaction, 1(3), 1-8.
12. Fowler, F. (1995). Improving Survey Questions. Thousand Oaks, California: Sage Publications.
13. Goldacre, B. (2008). Bad Science (p. 338). London: Harper Collins.
14. Gray, W., & Salzman, M. (1998). Damaged Merchandise? A Review of Experiments That Compare Usability Evaluation Methods. Human-Computer Interaction, 13(3), 203-261.
15. Grissom, R. J. (1994). Probability of the superior outcome of one treatment over another. Journal of Applied Psychology, 79(2), 314-316.
16. Hansen, W. B., & Collins, L. M. (1994). Seven ways to increase power without increasing N. In L. M. C. (Eds.) & L. A. Seitz (Eds.), Advances in data analysis for prevention intervention research (NIDA Research Monograph 142, NIH Publication No. 94-3599) (Vol. 191, pp. 184-195). Rockville, MD:: National Institutes of Health.
17. Hornbæk, K., & Frøkjær, E. (2008). A Study of the Evaluator Effect in Usability Testing. Human-Computer Interaction, 23(3), 251-277.
18. l Kampenes, V., Dyba, T., Hannay, J., & Sjoberg, D. (2007). A systematic review of effect size in software engineering experiments. Information and Software Technology, 49(11-12), 1073-1086.
19. Kaptein, M. C., Nass, C., & Markopoulos, P. (2010). Powerful and consistent analysis of likert-type rating scales. Proceedings of the 28th international conference on Human factors in computing systems - CHI '10 (p. 2391). New York, New York, USA: ACM Press.
20. Katki, H. a. (2008). Invited Commentary: Evidencebased Evaluation of p Values and Bayes Factors. American Journal of Epidemiology, 168(4), 384-388.
21. Kruschke, J. K. (2011). Introduction to Special Section on Bayesian Data Analysis. Perspectives on Psychological Science, 6(3), 272-273.
22. Kruschke, J. K. (2010). What to believe: Bayesian methods for data analysis. Trends in cognitive sciences, 14(7), 293-300. Elsevier Ltd.
23. Maxwell, S. E. (2004). The persistence of underpowered studies in psychological research: causes, consequences, and remedies. Psychological methods, 9(2), 147-63.
24. Morey, R., & Rouder, A. (2010). Package "BayesFactorPCL." Retrieved from https://r-forge.rproject.org/projects/bayesfactorpcl/
25. Oakes, M. (1986). Statistical inference: a commentary for the social and behavioural sciences. Wiley.
26. Rouder, Jeffrey N, Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic bulletin & review, 16(2), 225-37.
27. SIGCHI. (2011). More about SIGCHI. http://www.sigchi.org/. Retrieved July 8, 2011, from http://www.sigchi.org/about/more
28. Sharp, H., Preece, J., & Rogers, Y. (2007). Interaction Design (2nd ed.). Chichester: John Wiley and Sons, Ltd.
29. Sun, S., Pan, W., & Wang, L. L. (2010). A comprehensive review of effect size reporting and interpreting practices in academic journals in education and psychology. Journal of Educational Psychology, 102(4), 989-1004.
30. Wagenmakers, Eric-Jan, Wetzels, Ruud, Borsboom, D., & van der Maas, H. L. J. (2011). Why psychologists must change the way they analyze their data: the case of psi: comment on Bem (2011). Journal of personality and social psychology, 100(3), 426-32.
31. Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., & Wagenmakers, E.-J. (2011). Statistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 t-Tests. Perspectives on Psychological Science, 6(3), 291-298.
32. Ziliak, S., & McCloskey, D. (2008). The Cult of Statistical Significance: How the Standard Error Costs us Jobs, Justice and Lives. (p. 320). Ann Arbor, MI: University of Michigan Press.