Comparing inspections and user testing for the evaluation of virtual environments

International Journal of Human-Computer Interaction

Volumn 26, Issue 8, 2010, Pages 786-824

  Bach, Cedric ^a,b   Scapin, Dominique L ^c  

^a IRIT   (France)

^b Metapages   (France)

^c INRIA   (France)

Author keywords

[No Author keywords available]

Indexed keywords

DOCUMENT-BASED; END-USERS; QUALITATIVE ANALYSIS; QUANTITATIVE RESULT; USABILITY EVALUATION METHODS; USABILITY PROBLEMS; USER TESTING;

INSPECTION; VIRTUAL REALITY;

USABILITY ENGINEERING;

EID: 77954921902     PISSN: 10447318     EISSN: 15327590     Source Type: Journal    
DOI: 10.1080/10447318.2010.487195     Document Type: Article

References (71)

1. TM. In Proceedings of the 2008 International Conference (pp. 114-119). Washington, DC: IEEE Computer Society.
2. Agarawala, A., & Balakrishnan, R. (2006). Keepin' it real: Pushing the desktop metaphor with physics, pile and the pen. In Proceedings of the ACM CHI 2006 (pp. 1283-1292). New York: ACM Press.
3. Andre, T. S., Hartson, H. R., Belz, S. M, & McCreavy, F. A. (2001). The user action frame- work: a reliable foundation for usability engineering support tools. International Journal of Human-Computer Studies, 54, 107-136.
4. Bach, C. (2004). Élaboration et validation de Critères Ergonomiques pour les Interactions Homme-Environnements Virtuels [Development and validation of ergonomic criteria for human virtual-environments interactions]. Unpublished doctoral thesis, Université Paul Verlaine, Metz, France.
5. Bach, C., & Scapin, D. L. (2003). Adaptation of ergonomic criteria to human-virtual environments interactions. In Proceedings of Interact'03 (pp. 880-883). Amsterdam: IOS Press.
6. Bach, C., & Scapin, D. L. (2005). Critères ergonomiques pour les interactions homme-environnements virtuels: Définitions, justifications et exemples [Ergonomic criteria for human-virtual environments interactions: definitions, justifications, and examples] (Tech. Rep. No. 5531). Rocquencourt, France: French National Institute for Research in Computer Science and Control (INRIA).
7. Bastien, J. M. C., & Scapin, D. L. (1992). A validation of ergonomic criteria for the evaluation of human-computer interfaces. International Journal of Human-Computer Interaction, 4,183-196.
8. Bastien, J. M. C., & Scapin, D. L. (1993). Ergonomic criteria for the evaluation of human-computer interfaces (Tech. Rep. No. 156). Rocquencourt, France: French National Institute for Research in Computer Science and Control (INRIA).
9. Bastien, J. M. C., & Scapin, D. L. (1995). Evaluating a user interface with ergonomic criteria. International Journal of Human-Computer Interaction, 7, 105-121.
10. Bastien, J. M. C., Scapin, D. L., & Leulier, C. (1999). The ergonomic criteria and the ISO/DIS 9241 - 10 dialogue principles: A pilot comparison in an evaluation task. Interacting with Computers, 11, 299-322.
11. Bowman, D., Gabbard, J. L., & Hix, D. (2002). A survey of usability evaluation in virtual environments: classification of methods. Presence: Teleoperators and Virtual Environments, 11,435-455.
12. Bowman, D. A., & Hodges, L. F. (1997). An evaluation of techniques for grabbing and manipulating remote objects in immersive virtual environments. In Proceedings of the 1997 symposium on Interactive 3D graphics (pp. 35-38). New York: ACM Press.
13. Bowman, D., Kruijff, E., LaViola, J., & Poupyrev, I. (2005). 3D user interfaces: Theory and practice. Boston: Addison-Wesley.
14. Chapanis, A. (1982). Man/computer research at Johns Hopkins. In R. A. Kasschau, R. Lachman, & K. R. Laughery (Eds.), Information Technology and Psychology: Prospects for the future (pp. 238-249). New York: Praeger.
15. Chattratichart, J., & Lindgaard, G. (2008) A comparative evaluation of heuristic-based usability inspection methods. In Proceedings of the ACM CHI'08 extended abstracts on Human factors in computing systems (pp. 2213-2220). New York: ACM Press.
16. Cockton, G., & Lavery, D. (1999). A framework for usability problem extraction. In Proceedings of Interact'99 (pp. 344-352). Amsterdam: IOS Press.
17. Cockton, G., Woolrych, A., Hall, L., & Hidemarch, M. (2003). Changing analysts' tunes: The surprising impact of a new instrument for usability inspection method assessment. In Proceedings of People and Computers XVII: Designing for Society (pp. 145-162). New York: Springer Verlag.
18. Conkar, T., Noyes, J. M., & Kimble, C. (1999). CLIMATE: A framework for developing holistic requirement analysis in Virtual Environments. Interacting with Computers, 11, 387-402.
19. Connell, I. W., & Hammond, N. V. (1999). Comparing usability evaluation principles with heuristics: problem instances vs. problem types. In Proceedings of Interact'99 (pp. 621-629). Amsterdam: IOS Press.
20. Dow, S., MacIntyre, B., Lee, J., Oezbek, C., Bolter, J. D., & Gandy, M. (2005). Wizard of Oz support throughout an iterative design process. IEEE Pervasive Computing, 4(4), 18-26.
21. Dubois, E, Abou Moussa, W., Bach, C., & Bonnefoy, N. (2008). Modelling and simulation of mobile mixed systems. In J. Lumsden (Ed.), Handbook of research on user interface design and evaluation for mobile technology (pp. 346-363). Hershey: IGI Global.
22. Dubois, E., Truillet, P., & Bach, C. (2007). Evaluating advanced interaction techniques for navigating Google Earth. In Proceedings of the 21st BCS HCI Group Conference HCI 2007 (Vol. 2, pp. 4-7). London: BSC.
23. Dubois, E., Nedel, L. P., Dal Sasso Freitas, C. M., & Jacon, L. (2005). Beyond user experimentation: Notational-based systematic evaluation of interaction techniques in virtual reality environments. Virtual Reality, 8, 118-128.
24. Durlach, B. N. I., & Mavor, A. S. (1995). Virtual reality: Scientific and technological challenges. Washington, DC: National Academy Press.
25. Faulkner, L. (2002, July). Reducing variability-Research into structured approaches to usability testing and evaluation. In Proceedings of Annual Conference of the Usability Professionals Association.
26. Fischoff, B. (1982). Debiasing. In D. Kahneman, P. Slovic, & A. Tversky (Eds.), Judgment under uncertainty: Heuristics and biases (pp. 422-444). Cambridge, UK: Cambridge University Press.
27. Gabbard, J. L., & Hix, D. (1997). A taxonomy of usability characteristics in virtual environments. Unpublished master's thesis, Virginia Tech, Blacksburg, VA.
28. Gray, W. D., & Salzman, M. C. (1998). Damaged merchandise? A review of experiments that compare usability evaluation methods. Human-Computer Interaction, 13, 203-261.
29. Green, C. S., & Bavelier, D. (2006). Effect of action video games on the spatial distribution of visuospatial attention. Journal of Experimental Psychology: Human Perception and Performance, 32, 1465-1478.
30. Griffiths, G., Sharples, S., & Wilson, J. R. (2006). Performance of new participants in virtual environments: The Nottingham tool for assessment of interaction in virtual environments (NAIVE). International Journal of Human-Computer Studies, 64(3), 240-250.
31. Hand, C. (1997). A survey of 3D interactions techniques. Computer Graphics Forum, 16, 269-281.
32. Hartson, H. R., Andre, T. S., & Williges, R. C. (2001). Criteria for evaluating usability evaluation methods. International Journal of Human-Computer Interaction, 13, 373-410.
33. Hornbaek, K. (2009). Dogmas in the assessment of usability evaluation methods. Behaviour & Information Technology, pp. 1-15.
34. Hornbæk, K., & Frøkjær, E. (2005). Comparing usability problems and redesign proposals as input to practical systems development. In Proceedings of ACM Conference on Human Factors in Computing Systems (pp. 391-400). New York: ACM Press.
35. Hughes, C. E., Smith, E., Stapleton, C. B., & Hughes, D. E. (2004). Augmenting Museum Experiences with Mixed Reality. In Proceedings of Knowledge Sharing and Collaborative Engineering 2004. Calgary, Canada: Acta Press.
36. International Standards Organisation. (2000). ISO/TS 16982 Ergonomics of Human-System Interaction-Usability methods supporting Human Centred Design. Geneva: Author.
37. Kalawski, R. S. (1999). VRUSE: A computerised diagnostic tool for usability evaluation of virtual/synthetic environment systems. Applied Ergonomics, 30(1), 11-25.
38. Karampelas, P., Grammenos, D., Mourouzis, A., & Stephanidis, C. (2003). Towards I-Dove, an interactive support tool for building and using virtual environments with guidelines. In Proceedings of the 10th HCI International (Vol. 3, pp. 1411-1415). Mahwah, NJ: Erlbaum.
39. Kaur, K. (1998). Designing virtual environments for usability. Unpublished doctoral thesis, City University, London.
40. Kaur, K., Maiden, N., & Sutcliffe, A. (1996). Design practice and usability problems with virtual environments. In Proceedings of Virtual reality World'96 conference. Stuttgart, Germany: IDG conferences.
41. Kaur, K., Maiden, N., & Sutcliffe, A. (1999). Interacting with virtual environments: an evaluation of a model of interaction. Interacting with Computers, 11, 403-426.
42. Kennedy, R. S., Lane, N. E., Berbaum, K. S., & Lilienthal, M. G. (1993). Simulator sickness questionnaire an enhanced method for quantifying simulator sickness. International Journal of Aviation Psychology, 3, 203-220.
43. Law, E., Scapin, D. L., Cockton, G., Springett, M., Stary, C., & Winckler, M. (2009). Maturation of Usability Evaluation Methods: Retrospect and prospect. Proceedings of COST294-MAUSE Closing Conference. Toulouse, France: IRIT Press.
44. Leulier, C., Bastien, J. M. C., & Scapin, D. L. (1998). Compilation of ergonomic guidelines for the design and evaluation of Web sites (Contract report). Rocquencourt, France: French National Institute for Research in Computer Science and Control (INRIA).
45. Lindgaard, G. (2006). Notions of thoroughness, efficiency, and validity: Are they valid in HCI practice? International Journal of Industrial Ergonomics, 36, 1069-1074.
46. Mankoff, J., Fait, H., & Tran, T. (2005). Is your web page accessible?: a comparative study of methods for assessing web page accessibility for the blind. In Proceedings of the ACM CHI'05 (pp. 41-50). New York: ACM Press.
47. Molich, R., & Dumas, J. S. (2008). Comparative usability evaluation (CUE-4). Behaviour & Information Technology, 27, 263-281.
48. Microsoft Game Studios. (2000). StarLancer. Redmond, WA: Author. Retrieved June 12, 2008, from http://www.microsoft.com/games/da/starlancer/
49. Nielsen, J. (1993). Usability engineering. San Diego, CA: Academic.
50. Pollier, A. (1991). Evaluation d'une interface par des ergonomes: Diagnostics et stratégies. [User interface evaluation by ergonomists: diagnoses and strategies] (Research Rep. No. 1391). Rocquencourt, France: French National Institute for Research in Computer Science and Control (INRIA).
51. Poupyrev, I., & Ichikawa, T. (1999). Manipulating objects in virtual worlds: Categorization and empirical evaluation of interaction techniques. Journal of Visual Languages and Computing, 10(1), 19-35.
52. Scapin, D. L. (1990). Organizing human factors knowledge for the evaluation and design of interfaces. International Journal of Human-Computer Interaction, 2, 203-229.
53. Scapin, D. L., & Bastien, J. M. C. (1997). Ergonomic criteria for evaluating the ergonomic quality of interactive systems. Behaviour & Information Technology, 16, 220-231.
54. Scapin, D. L., & Law, E. (2007). Review, report and refine Usability Evaluation Methods (R3 UEMs). Proceedings of the COST294-MAUSE 3rd International Workshop. Athens, Greece.
55. Schmettow, M., & Vietze, W. (2008). Introducing item response thery for measuring usability inspection processes. In Proceedings of the ACM CHI'08 (pp. 893-902). New York: ACM Press.
56. Shneiderman, B. (1998). Designing the User Interface: Strategies for Human-Computer Interaction. MA: Addison-Wesley.
57. Stanney, K. M. (2002). Handbook of Virtual Environments: Design, implementation, and applications. Mahwah, NJ: Erlbaum.
58. Stanney, K. M., & Davies, R. C. (2005). Augmented reality in Internet applications. In R. W. Proctor & K-P L. Vu (Eds.), Handbook of human factors in Web design (pp. 647-657). Mahwah, NJ: Erlbaum.
59. Stanney, K. M., Mollaghasemi, M., Reeves, L., Breaux, R., & Graeber, D. A. (2003). Usability engineering of virtual environment (VEs): Identifying multiple criteria that drive effective VE system design. International Journal of Human-Computer Studies, 58, 447-481.
60. Stanney, K. M., Mourant, R. R., & Kennedy, R.S. (1998). Human Factors issues in Virtual Environments: A review of the literature. Presence: Teleoperators and Virtual Environments, 7,327-351.
61. Stapleton, C. B., & Hughes, C. E. (2005). Mixed reality and experiential movie trailers: Combining emotions and immersion to innovate entertainment marketing. In Proceedings of 2005 International Conference on Human-Computer Interface Advances in Modeling and Simulation (pp. 40-48). New Orleans.
62. Sutcliffe, A. (2003). Multimedia and virtual reality: designing multisensory user interfaces. Mahwah, NJ: Erlbaum.
63. Sutcliffe, A., & Gault, B. (2004). Heuristic evaluation of Virtual Reality applications. Interacting With Computers, 16, 831-849.
64. Sutcliffe, A., & Kaur, K. (2000). Evaluating the usability of virtual reality user interfaces. Behaviour & Information Technology, 19(6), 415-426.
65. Tromp, J. G. (2001). Systematic usability design and evaluation for collaborative virtual environments. Unpublished doctoral thesis, University of Nottingham, Nottingham, United Kingdom.
66. Tromp, J. G., Steed, A., & Wilson, J. R. (2003). Systematic Usability evaluation and design issues for collaborative Virtual Environments. Presence: Teleoperators and Virtual Environments, 12, 241-267.
67. Wickens, C. D., & Baker, P. (1995). Cognitives issues in virtual reality. In W. Barfield & T. A. Furness III (Eds.), Virtual environments and advanced interface design (pp. 514-541). New York: Oxford University Press.
68. Williams J. S., & Harrison M. D. (2001). A toolset supported approach for designing and testing virtual environment interaction techniques. International Journal of Human-Computer Studies, 55, 145-165.
69. Wilson, J. R., Eastgate, R. M., & D'Cruz, M. (2002). Structured development of Virtual Environments. In K. M. Stanney (Ed.), Handbook of Virtual Environments. Design, imple- mentation, and applications (pp. 353-378). Mahwah, NJ: Erlbaum.
70. Witmer, B. G., & Singer, M. J. (1998). Measuring presence in virtual environments: A presence questionnaire. Presence: Teleoperators and Virtual Environments, 7, 225-240.
71. Zachary, W. (1986). Cognitively based functional taxonomy of decision support techniques. Human-Computer Interaction, 2, 25-63.