Let your users do the testing: A comparison of three remote asynchronous usability testing methods

Conference on Human Factors in Computing Systems - Proceedings

Volumn , Issue , 2009, Pages 1619-1628

  Bruun, Anders ^a   Gull, Peter ^b   Hofmeister, Lene ^c   Stage, Jan ^d  

^a Mjølner Informatics A S   (Denmark)

^b Jyske Bank   (Denmark)

^c Nykredit A S   (Denmark)

^d AALBORG UNIVERSITY   (Denmark)

Author keywords

Asynchronous testing; Empirical study; Remote testing; Usability testing

Indexed keywords

ASYNCHRONOUS METHODS; CONVENTIONAL METHODS; EMPIRICAL STUDIES; PRACTICAL PROBLEMS; REMOTE TESTING; TEMPORAL SEPARATION; USABILITY TESTING; USABILITY TESTING METHODS;

DEMODULATION; HUMAN ENGINEERING;

TESTING;

EID: 84892471147     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1518701.1518948     Document Type: Conference Paper

References (38)

1. Andreasen, M. S., Nielsen, H. V., Schroder, S. O. and Stage, J. What happened to remote usability testing? An empirical study of three methods. Proceedings of CHI 2007, ACM Press (2007), 1405-1414.
2. Au, I., Boardman, R., Jeffries, R., Larvie, P., Pavese, A., Riegelsberger, J., Rodden, K., and Stevens, M. User experience at google: focus on the user and all else will follow. Proceedings of CHI 2008. ACM Press (2008), 3681-3686.
3. Bak, J. O., Nguyen, K., Risgaard, P., and Stage, J. Obstacles to usability evaluation in practice: A survey of software development organizations. Proceedings of NordiCHI 2008. ACM Press (2008), 23-32.
4. Brush, A. B., Ames, M., and Davis, J. A comparison of synchronous remote and local usability studies for an expert interface. Proceedings of CHI 2004, ACM Press (2004), 1179-1182.
5. Capra, M. G. An Exploration of End-User Critical Incident Classification. Master thesis, Virginia Polytechnic Institute and State University, 2001.
6. Castillo, J. C. The User-Reported Critical Incident Method for Remote Usability Evaluation. Master thesis, Virginia Polytechnic Institute and State University, 1997.
7. Castillo, J. C., Hartson, H. R. and Hix, D. Remote usability evaluation: Can users report their own critical incidents? Proceedings of CHI 1998, ACM Press (1998), 253-254.
8. Desurvire, Heather W. Faster, cheaper!! Are Usability Inspection Methods as Effective as Empirical Testing? John Wiley & Sons. 173-202, 1994.
9. Dray, S. and Siegel, D. Remote possibilities?: International usability testing at a distance, interactions 11. 2 (2004), 10-17.
10. Ericsson, K. A. and Simon, H. A. How to study thinking in everyday life: contrasting think-aloud protocols with descriptions and explanations of thinking. Mind. Culture, and Activity 3 (1998), 178-186.
11. Folstad, A., Brandtzaeg, P. B. and Heim, J. Usability Analysis and Evaluation of Mobile ICT Systems. http://www.hft.org/HFT01/paper01/mobility/25- 01.pdf.
12. Hammontree, M., Weiler, P. and Nayak, N. Remote usability testing. Interactions 1, 3 (1994), 21-25.
13. Hartson, H. R. and Castillo, J. C. Remote evaluation for post-deployment usability improvement. Proceedings of AVI 1998, ACM Press (1998), 22-29.
14. Hartson, H. R., Castillo, J. C., Kelso, J. and Neale, W. C. Remote evaluation: The network as an extension of the usability laboratory. Proceedings of CHI 1996, ACM Press (1996), 228-235.
15. Hertzum, M. and Jacobsen, N. E. The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction 15, 1 (2003), 183-204.
16. Hilbert, D. M. and Redmiles, D. F. Separating the wheat from the chaff in internet-mediated user feedback expectation-driven event monitoring. ACM SIGGROUP Bulletin 20. 1 (1999), 35-40.
17. Kjeldskov, J., Skov, M. B. and Stage, J. Instant Data Analysis: Evaluating Usability in a Day. Proceedings of NordiCHI 2004, ACM Press (2004), 233-240.
18. Larsen, J. M. Playful Interaction. Master thesis, Aalborg University, Department of Computer Science, 2008.
19. Marsh, S. L., Dykes, J. and Attilakou, F. Evaluating a geovisualization prototype with two approaches: remote instructional vs. face-to-face exploratory. Proceedings of Information Visualization 2006, IEEE (2006), 310- 315.
20. Millen, D. R. Remote usability evaluation: user participation in the design of a web-based email service. ACM SIGG ROUP Bulletin 20, 1 (1999), 40-45.
21. Nielsen, C. M., Overgaard, M., Pedersen, M. B., Stage, J. and Stenild, S. It's worth the hassle! the added value of evaluating the usability of mobile systems in the field. Proceedings of NordiCHI 2006, ACM Press (2006), 272-280.
22. Nielsen, J. Finding usability problems through heuristic evaluation. Proceedings of CHI 1992, ACM Press (1992), 373-380.
23. Nielsen, J. and Molich, R. Heuristic evaluation of user interfaces. Proceedings of CHI 1990, ACM Press (1990), 249-256.
24. Nielsen, J. Usability inspection methods. Proceedings of CHI 1994, ACM Press (1994), 377-378.
25. Olmsted, E. and Gill, M. In-person usability study compared with self-administered web (remote-different time-place) study: does mode of study produce similar results? Proceedings ofUPA 2005, UPA (2005).
26. Petrie, H., Hamilton, F., King, N. and Pavan, P. Remote usability evaluation with disabled people. Proceedings of CHI 2006, ACM Press (2006), 1133-1141.
27. Rubin, J. Handbook of Usability Testing. John Wiley and Sons, 1994.
28. Scholtz, J. A case study: developing a remote, rapid and automated usability testing methodology for on-line books. Proceedings ofHICSS 1999, IEEE (1999).
29. Scholtz, J. and Downey, L. Methods for identifying usability problems with web sites. Proceedings of lFlP Conference, ACM Press (1998), 191-206.
30. Steves, M. P. et al. A comparison of usage evaluation and inspection methods for assessing groupware usability. Proceedings ofCSCW2001, ACM Press, 125-134.
31. Thompson, J. A. Investigating the Effectiveness of Applying the Critical Incident Technique to Remote Usability Evaluation. Master thesis, Virginia Polytechnic Institute and State University, 1999.
32. Tullis, T., Fleischman, S., McNulty, M., Cianchette, C. and Bergel, M. An empirical comparison of lab and remote usability testing of web sites. http://home.comcast.net/tomtullis/publications/RemoteVsLab.pdf.
33. Vermeeren, A. P. O. S., van Kesteren, I. and Bekker, M. M. Managing the 'evaluator effect' in user testing. Proceedings of INTERACT 2003, IOS Press (2003).
34. Waterson, S., Landay, J. A. and Matthews, T. In the lab and out in the wild: remote web usability testing for mobile devices. Proceedings of CHI 2002, ACM Press, 796-797.
35. West, R. and Lehman, K. R. Automated Summative Usability Studies: An Empirical Evaluation. Proceedings of CHI 2006, ACM Press (2006), 631 -639.
36. Winckler, M. A. A., Freitas, C. M. D. S. and de Lima, J. V. Remote usability testing: A case study. Proceedings of OzCHl 1999, CHISIG (1999).
37. Winckler, M. A. A., Freitas, C. M. D. S. and de Lima, J. V. Usability remote evaluation for www. Proceedings of CHI 2000, ACM Press (2000), 131-132.
38. Aijo, R. and Mantere, J. Are Non-Expert Usability Evaluations Valuable? http://www.hft.org/HFT01/paperO1/acceptance/201.pdf.