Sample size in usability studies

Communications of the ACM

Volumn 55, Issue 4, 2012, Pages 64-70

  Schmettow, Martin ^a  

^a UNIVERSITY OF TWENTE   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

INTERACTIVE DESIGN; MAGIC NUMBERS; SAMPLE SIZES; STATISTICAL MODELS; USABILITY PROBLEMS; USABILITY STUDIES; USABILITY TESTING; USER EXPERIENCE;

PRODUCT DESIGN; SAMPLING;

USABILITY ENGINEERING;

EID: 84861144290     PISSN: 00010782     EISSN: 15577317     Source Type: Journal    
DOI: 10.1145/2133806.2133824     Document Type: Conference Paper

References (22)

1. Briand, L.C., El Emam, K., Freimut, B.G., and Laitenberger, O. A comprehensive evaluation of capture-recapture models for estimating software defect content. IEEE Transactions on Software Engineering 26, 6 (June 2000), 518-540.
2. Burnham, K.P. and Anderson, D.R. Multimodel Inference. Understanding AIC and BIC in model selection. Sociological Methods & Research 33, 2 (Nov. 2004), 261-304. (Pubitemid 39519124)
3. Caulton, D.A. Relaxing the homogeneity assumption in usability testing. Behaviour & Information Technology 20, 1 (2001), 1-7. (Pubitemid 34915963)
4. Dorazio, R.M. and Royle, J.A. Mixture models for estimating the size of a closed population when capture rates vary among individuals. Biometrics 59, 2 (June 2003), 351-64. (Pubitemid 36750349)
5. Hornbæk, K. and Frøkjær, E. Comparison of techniques for matching of usability problem descriptions. Interacting with Computers 20, 6 (Dec. 2008), 505-514.
6. Hwang, W. and Salvendy, G. Number of people required for usability evaluation: The 10±2 rule. Commun. ACM 53, 5 (May 2010), 130-133.
7. John, B. and Marks, S. Tracking the effectiveness of usability evaluation methods. Behaviour & Information Technology 16, 4 (1997), 188-202. (Pubitemid 127623939)
8. Law, E.L.-C. and Hvannberg, E.T. Analysis of combinatorial user effect in international usability tests. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vienna, Austria, Apr. 24-29). ACM Press, New York, 2004, 9-16.
9. Lewis, J.R. Evaluation of procedures for adjusting problem-discovery rates estimated from small samples. International Journal of Human-Computer Interaction 13, 4 (2001), 445-479. (Pubitemid 33721372)
10. Lewis, J.R. Testing small system customer set-up. In Proceedings of the Annual Meeting of the Human Factors and Ergonomics Society. Human Factors Society, Santa Monica, CA, 1982, 718-720.
11. Molich, R. How many participants needed to test usability? Commun. ACM 53, 8 (Aug. 2010), 7.
12. Molich, R. and Dumas, J. Comparative usability evaluation (CUE-4). Behaviour & Information Technology 27, 3 (2008), 263-281.
13. Nielsen, J. Why You Only Need to Test with 5 Users. Jakob Nielsen's Alertbox (Mar. 19, 2000); http://www.useit.com/alertbox/20000319.html
14. Nielsen, J. and Landauer, T.K. A mathematical model of the finding of usability problems. In Proceedings of INTERCHI 1993 (Amsterdam, the Netherlands, Apr. 24-29). ACM Press, New York, 1993, 206-213.
15. Nørgaard, M. and Hornbæk, K. What do usability evaluators do in practice? An explorative study of think-aloud testing. In Proceedings of the Sixth Conference on Designing Interactive Systems (University Park, PA, June 26-28). ACM Press, New York, 2006, 209-218. (Pubitemid 44725733)
16. Schmettow, M. Controlling the usability evaluation process under varying defect visibility. In Proceedings of the 23rd British HCI Group Annual Conference on People and Computers: Celebrating People and Technology (Cambridge, U.K., Sept. 1-5). British Computer Society, Swinton, U.K., 2009, 188-197.
17. Schmettow, M. Heterogeneity in the usability evaluation process. In Proceedings of the 22nd British HCI Group Annual Conference on Human-Computer Interaction (Liverpool, U.K., Sept. 1-5). British Computer Society, Swinton, U.K., 2008, 89-98.
18. Spool, J. and Schroeder, W. Testing Web sites: Five users is nowhere near enough. CHI Extended Abstracts on Human Factors in Computing Systems (Seattle, Mar. 31-Apr. 5), ACM Press, New York, 2001, 285-286.
19. Turner, C.W., Lewis, J.R., and Nielsen, J. Determining usability test sample size. In International Encyclopedia of Ergonomics and Human Factors, W. Karwowski, Ed. CRC Press, Boca Raton, FL, 2006, 3084-3088.
20. Virzi, R.A. Refining the test phase of usability evaluation: How many subjects is enough? Human Factors: The Journal of the Human Factors and Ergonomics Society 34, 4 (1992), 457-468.
21. Walia, G.S. and Carver, J.C. Evaluation of capturerecapture models for estimating the abundance of naturally occurring defects. In Proceedings of the Second ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (Kaiserslautern, Germany, Oct. 9-10). ACM Press, New York, 2008, 158-167.
22. Woolrych, A. and Cockton, G. Why and when five test users aren't enough. In Proceedings of the IHM-HCI Conference, J. Vanderdonckt, A. Blandford, and A. Derycke, Eds. (Lille, France, Sept. 10-14). Cépadèus Éditions, Toulouse, France, 2001, 105-108.