Sample sizes for usability tests: Mostly math, not magic

Interactions

Volumn 13, Issue 6, 2006, Pages 29-33

  Lewis, James R ^a  

^a IBM   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

REPRESENTATIVE ENVIRONMENTS; REPRESENTATIVE PARTICIPANTS; USABILITY TESTING;

COSTS; PROBABILITY; TESTING; THEOREM PROVING;

USABILITY ENGINEERING;

EID: 33750853322     PISSN: 10725520     EISSN: None     Source Type: Trade Journal    
DOI: 10.1145/1167948.1167973     Document Type: Article

References (25)

1. Al-Awar, J., Chapanis, A., & Ford, R. (1981). Tutorials for first-time computer user. IEEE Transactions on professional Communication, 24, 30-37.
2. ANSI (2001), Common industry format for usability test reports (ANSI-NCITS 354-2001), Washington, DC: American National Standards Institute.
3. Bradley, J. V. (1968), Distribution-free statistical tests. Englewood Cliffs, NJ: Prentice-Hall.
4. Bradley, J. V. (1976), Probability; decision; statistics. Englewood Cllffs, NJ; Prentice-Hall,
5. Chapanis, A. (1981). Evaluating case of use. Unpublished manuscript prepared for IBM, available on request from J. R. Lewis.
6. Diamond, W. J. (1981), Practical experiment designs far engineers and scientists. Belmont, CA: Lifetime Learning Publications.
7. Hertzum, M., & Jacobsen, N. J. (2003). The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction, 15, 183-204.
8. ISO, (1998). Ergonomic requirements for office work with visual display terminals (VDTs). Part 11 : Guidance on usability (ISO 9241-11:1998(E)), Geneva, Switzerland: Author.
9. Kraemer, H. C., & Thiemann, S, (1987). How many subjects? Statistical power analysis in research. Newbury Park, CA: Sage.
10. Lewis, J. R. (1982). Testing small system customer set-up. In Proceedings of the Human Factors Society 26th Annual Meeting (pp. 718-720). Santa Monica, CA: Human Factors Society.
11. Lewis, J. R. (1993), Problem discovery in usability studies: A model based on the binomial probability formula. In Proceedings of the Fifth International Conference on Human-Computer Interaction (pp. 666-671), Orlando, FL: Elsevier.
12. Lewis, J. R. (1994), Sample sizes for usability studies: Additional considerations. Human Factors. 36, 368-378.
13. Lewis, J. R. (2001). Evaluation of procedures for adjusting problem-discovery rates estimated from small samples. International Journal of Human-Computer Interaction, 13, 445-479.
14. Lewis, J. R. (2006), Usability testing, In G. Salvendy (ed.), Handbook of Human Factors and Ergonomics (pp. 1275-1316). New York, NY: John Wiley.
15. Nielsen, J., & Landauer, T.K. (1993). A mathematical model of the finding of usability problems. In Proceedings of ACM INTERCHI'93 Conference (pp. 206-213). Amsterdam, Netherlands: ACM Press.
16. Nielsen, J., & Molich, R. (1990). Heuristic evaluation or user Interfaces, in Conference Proceedings on Human Factors in Computing Systems - CHI90 (pp. 249-266). New York, NY: ACM.
17. Perfetti, C., & Landesman, L. (2001). Eight is not enough. Retrieved July 4, 2006 from http://www.uie.com/artlcles/eight_is_not_enough/
18. Sauro, J. (2006). Ul problem discovery sample size. Downloaded from Measuring Usability website, July 20, 2006-http://www.measuringusability.com/ samplesize/problem.discovery.php.
19. Spool, J., & Schroeder, W. (2001), Testing web sites: Five users Is nowhere near enough. In CHI2001 Extended Abstracts (pp. 285-286). New York: ACM Press.
20. Turner, C. W., Lewis, J. R., & Nielsen, J. (2006). Determining usability test sample size. In W. Karwowski (ed.), International Encyclopedia of Ergonomics and Human Factors (pp. 3084-3088). Boca Raton, FL: CRC Press.
21. Virzi, R. A. (1990). Streamlining the design process: Running fewer subjects. In Proceedings of the Human Factors Society 34th Annual Meeting (pp. 291-294), Santa Monica, CA: Human Factors Society.
22. Virzi, R.A. (1992). Refining the test phase of usability evaluation: How many subjects is enough? Human Factors, 34, 457-468.
23. Walpole, R. E. (1976). Elementary statistical concepts. New York, NY: Macmillan.
24. Wixon, D, (2003). Evaluating usability methods: Why the current literature falls the practitioner, Interactions. 10(4), 28-34.
25. Wright, P. C., & Monk, A. F. (1991). A cost-effective evaluation method for use by designers, International Journal of Man-Machine Studies, 35, 891-912.