Evaluation of multimedia applications using inspection methods: The Cognitive Walkthrough case

Interacting with Computers

Volumn 16, Issue 2, 2004, Pages 183-215

  Huart, Julien ^a,b   Kolski, Christophe ^b   Sagar, Mouldi ^b  

^a Logica SA   (France)

^b UNIVERSITÉ DE VALENCIENNES   (France)

Author keywords

Cognitive Walkthrough; Evaluation; Multimedia application

Indexed keywords

COGNITIVE SYSTEMS; ERGONOMICS; EVALUATION; HUMAN COMPUTER INTERACTION; INSPECTION; MAN MACHINE SYSTEMS; PROGRAM DOCUMENTATION;

COGNITIVE WALKTHROUGH; MULTIMEDIA APPLICATIONS;

MULTIMEDIA SYSTEMS;

EID: 1542376005     PISSN: 09535438     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.intcom.2003.12.005     Document Type: Article

References (94)

1. Abowd, G., 1995. Performing a Cognitive Walkthrough. Available at: http://www.cc.gatech.edu/computing/classes/cs3302/documents/cog.walk.html.
2. Ahuja J.S., Webster J. Perceived disorientation: an examination of a new measure to assess web design effectiveness. Interacting with Computers. 14:2001;15-29.
3. Balpe J.P. Hypertexte et interactivité Hypertextes et Hypermédias. 1:(1):1997;11-22.
4. Bastien, J.M.C., Scapin, D.L., 1993. Ergonomic criteria for the evaluation of human-computer interface. Research Report. INRIA, Paris.
5. Bastien J.M.C., Scapin D.L. Évaluation des systèmes d'information et Critères Ergonomiques. Kolski C. Environnements Évolués et Évaluation de l'IHM, Interaction Homme-Machine pour les SI 2. 2001;53-80 Hermes, Paris.
6. Bastien J.M.C., Leulier C., Scapin D.L. l'Ergonomie des sites web. Le Moal J.C., Hidoine B. Créer et Maintenir un Service Web. 1998;111-173 ADBS, Paris.
7. Bastien J.M.C., Scapin D.L., Leulier C. The ergonomic criteria and the ISO/DIS 9241-10 dialogue principles: a pilot comparison in an evaluation task. Interacting with Computers. 11:1999;299-322.
8. Bias R.G. The pluralistic usability walkthrough: coordinated empathies. Nielsen J., Mack R.L. Usability Inspection Methods. 1994;63-76 Elsevier, Amsterdam.
9. Blackmon M.H., Polson P.G., Kitajima M., Lewis C. Cognitive Walkthrough for the Web. 2002 ACM Conference on Human Factors in Computing Systems (CHI'2002). 2002;ACM Press, New York. pp. 463-470.
10. Bowers, N., 1996. Weblint: quality assurance for the World Wide Web. Proceedings of the Fifth International World Wide Web Conference, Paris.
11. Carretier, M., Nicot, S., Benoît, E., 1999. Utilisation de la méthode Cognitive Walkthrough pour l'évaluation de produits multimédias. Master Thesis in Industrial Ergonomics. LAMIH, University of Valenciennes (Unpublished), February.
12. Chignell H.M., Motoyama T., Melo V. Discount video analysis for usability engineering. Anzai Y., Ogawa K., Mori H. Symbiosis of Human and Artifact, Social aspects of Human-Computer Interaction. 1995;323-328 Elsevier, Amsterdam.
13. Collins, P., 2000. MetaWeb: A Plan for Cognitive Walkthrough. Available at: http://c2000.gatech.edu/c2000/cs6751_96_fall/projects/glass/walkthrough.html.
14. Conklin J. Hypertext: an introduction and survey. IEEE Computer. 20:(9):1987;17-41.
15. Cooper M. Evaluating accessibility and usability of web pages. Vanderdonckt J., Puerta A. Computer-Aided Design of User Interfaces II. 1999;33-42 Kluwer, Dordrecht.
16. Dholakia, R.R., Zhao, M., Dholakia, N., Fortin, D.R., 2000. Interactivity and revisits to websites: a theoretical framework. Working Paper. RITIM. Accessible at: http://ritim.cba.uri.edu/wp/.
17. Dix H., Finlay J., Abowd G., Beale R. Human-Computer Interaction. 1993;Prentice-Hall, Englewood Cliffs, NJ.
18. Duffy T., Lowyck J., Jonassen D. Designing environments for constructive learning. Scott Grabinger R. Hypermedia. 1993;144-149 Taylor Graham, Cambridge.
19. Durand, A., 1997. Modélisation moléculaire, vers un nouvel outil d'aide à la conception multimedia. PhD Thesis. University of Valenciennes.
20. Durand A., Huart J., Leleu-Merviel S. Vers un modèle de programme pour la conception de document. Hypertextes et Hypermédias. 1:(1):1997;79-101.
21. Durand A., Laubin J.M., Leleu-Merviel S. Vers une classification des procédés d'interactivité par niveaux corrélés aux données. Hypertextes et Hypermédias. 1:(2-4):1997;367-382. Hermès, Paris.
22. Eisenstein S.M. Dramaturgie de la Forme Filmique. 1929;Dunod, Paris.
23. Farenc, C., Barthet, M.F., Liberati, V., 1996. Automatic ergonomic evaluation: which are the limits. Proceedings CADUI'96, 2nd International Workshop on Computer-Aided Design of User Interfaces, Namur, Belgium, June 5-7.
24. Forse T. Qualimétrie des Systèmes Complexes, Mesure de la Qualité du Logiciel. 1989;Les Éditions d'Organisation, Paris.
25. Garzotto F., Matesa M. A systematic method for hypermedia usability inspection. The New Review of Hypermedia and Multimedia. 3:1997;39-65.
26. Garzotto F., Mainetti L., Paolini P. Hypermedia design analysis, and evaluation issues. Communications of the ACM. 38:(8):1995;74-86.
27. Gibbs W. Software's chronic crisis. Scientific American. 271:(3):1994;72-81.
28. Grislin M., Kolski C. Evaluation des interfaces homme-machine lors du développement de système interactif. Technique et Science Informatiques (TSI). 15:(3):1996;265-296.
29. Grudin J. Utility and usability: research issues and development contexts. Interacting with Computers. 4:(2):1992;209-217.
30. Head J.F., Archer N., Yuan Y. World wide web navigation aid. International Journal of Human-Computer Studies. 53:2000;301-330.
31. Hertzum M., Jacobsen N.E. The evaluator effect during first-time use of the cognitive-walkthrough technique. Bullinger H.J., Ziegler J. Proceedings of the 8th Human-Computer Interaction International Conference 1999 (HCI'99, Munich, Germany, August 22-26, 1999). vol. 2:1999;1063-1067 Lawrence Erlbaum Associates, London.
32. Hû O., Trigano P. A tool for evaluation using dynamic navigation in a set of questions. Vanderdonckt J., Farenc C. Tools for Working with Guidelines TFWWG'2000. 1999;273-281 Springer, Berlin.
33. Huart, J., 2000. Mieux concevoir pour mieux communiquer à l'ère des nouveaux médias, vers des méthodes de conduite de projets et d'évaluation qualité de documents multimédias. PhD Thesis. University of Valenciennes, France.
34. Huart, J., Kolski, C., Leleu-Merviel, S., 1998. Vers la correction et la prévention des erreurs méthodologiques dans le cycle de vie d'applications multimédias. Proceedings 6ème Colloque Ergonomie et Informatique Avancée ERGO'IA (November 4-6, Biarritz). ESTIA/ILS, Bayonne, pp. 59-68.
35. INA, 1994. Facteurs-clés de Succès des Produits Multimédias Interactifs, Étude Guide. 1994;Institut National de l'Audiovisuel Bry-sur-Marne, France.
36. ISO DIS 14915, 1999. Multimedia User Interface Design - Software Ergonomic Requirements. ISO.
37. John B.E., Packer H. Learning and using the Cognitive Walkthrough method: a case study approach. Proceedings of CHI'95 (May 7-11, Denver). 1995;ACM Press, New York. pp. 429-436.
38. Jordan P.W., Thomas B., Weardmuster B., McClelland I. Usability Evaluation in Industry. 1996;Taylor & Francis, London.
39. Kahn M.J., Prail A. Formal usability inspections. Nielsen J., Mack R.L. Usability Inspection Methods. 1994;141-171 Elsevier, Amsterdam.
40. Karat C.M. A comparison of user interface evaluation methods. Nielsen J., Mack R.L. Usability Inspection Methods. 1994;203-233 Elsevier, Amsterdam.
41. Karat C.M., Campbell R., Fiegel T. Comparison of empirical testing and walkthrough methods in user interface evaluation. Proceedings of CHI'92. 1992;ACM Press, New York. pp. 397-404.
42. Kelley T., Allender L. Why choose? A process approach to usability testing. Anzai Y., Ogawa K., Mori H. Symbiosis of Human and Artifact: Human and Social Aspects of Human-Computer Interaction. 1995;393-398 Elsevier, Amsterdam.
43. Kemp B., Buckner K. A taxonomy of design guidance for hypermedia design. Interacting with Computers. 12:1999;143-160.
44. Kirakowski, J., Cierlik, B., 1998. Measuring the usability of web sites. Proceedings of Human Factors and Ergonomics Society Annual Conference, Chicago.
45. Kolski C. Interfaces Homme-Machine, Application aux Systèmes Industriels Complexes. 1997;Hermes, Paris.
46. Kolski C., Millot P. A rule-based approach for the ergonomic evaluation of man-machine graphic interface. International Journal of Man-Machine Studies. 35:1991;657-674.
47. Kolski C., Riera B., Berger T. A questionnaire-based discount evaluation method using guidelines for process control interactive applications. Vanderdonckt J., Farenc C. Tools for Working With Guidelines TFWWG'2000. 2000;127-138 Springer, London.
48. Kouroupetroglou, G., Viglas, C., Metaxaki, C., 1994. A Generic methodology and instrument for evaluating interactive multimedia. Proceedings of Telematics for Education and Training Conference, November, Dusseldorf.
49. Layaïda, N., Keramane, C., 1995. Maintaining temporal consistency of multimedia documents. Proceedings of the ACM Workshop on Effective Abstractions in Multimedia, November.
50. Leleu-Merviel S. La Conception en Communication. 1997;Hermes, Paris.
51. Leulier, C., Bastien, C., Scapin, C.D., 1998. Compilation of ergonomic guidelines for the design and evaluation of web sites. Research Report. Esprit Project Commerce and Interactions (EP 22287).
52. Lewis C. Cognitive Walkthroughs. Helander M., Landauer T.K., Prablhu P. Handbook of Human-Computer Interaction. 1997;717-732 Elsevier, Amsterdam.
53. Lewis, C., Rieman, J., 1994. Task-centered User Interface Design - A Practical Introduction. Complete text on-line. Also available through anonymous ftp: cs.colorado.edu (Directory: /pub/cs/distribs/clewis/HCI-Design-Book/.
54. Löwgren J., Nordqvist T. Knowledge-based evaluation as design support for graphical user interfaces. Bauersfeld P., Bennett J., Lynch G. CHI'92 ACM Conference on Human factors in Computing Systems, Monterey, California, May 3-7. 1992;181-188 ACM Press, New York.
55. Mack R.L., Nielsen J. Executive summary. Nielsen J., Mack R.L. Usability Inspection Methods. 1994;4-23 Elsevier, Amsterdam.
56. McCall, J.A., Richards, P.K., Walters, G.F., 1997. Factors in Software Quality, vols. I, II and III. US Rome Air Development Center. Reports NTIS ADIA-049014,015,055. National Technical Information Services, US Department.
57. McDonald S., Stephenson R.J. Effects of text structure and prior knowledge of the learner on navigation in hypertext. Human Factors. 40:(1):1998;18-27.
58. Mendes M.E., Hall W., Harrison R. Applying metrics to the evaluation of educational hypermedia applications. Journal of Universal Computer Science. 4:(4):1998;382-403.
59. Mullin J., Anderson A.H., Smallwood L., Jackson M., Katsavras E. Eye-tracking explorations in multimedia communications. Blanford A., Vanderdonckt J., Gray P. People and Computer XV - Interaction without Frontiers - Joint Proceedings of HCI 2001 and IHM 2001. 2001;367-382 Springer, London.
60. Nielsen J. Usability Engineering. 1993;Academic Press, London.
61. Nielsen, J., 1994. Guerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier. Useit Papers and Essays. Available at: http://www.useit.com/papers/guerrilla_hci.html .
62. Nielsen J. Heuristic evaluation. Nielsen J., Mack R.L. Usability Inspection Methods. 1994;25-62 Elsevier, Amsterdam.
63. Nielsen, J., 1997. Loyalty on the Web. Useit Alertbox (August 1997). Available at: http://www.useit.com/alertbox/9708a.html.
64. Nielsen, J., 1998. The Web Usage Paradox: Why Do People Use Something This Bad? Useit Alertbox (August 1998). Available at: http://www.useit.com/alertbox/980809.html.
65. Nielsen J., Mack R.L. Usability Inspection Methods. 1994;Elsevier, Amsterdam.
66. Nielsen J., Molich R. Heuristic evaluation of user interfaces. Proceedings CHI'90 Conference, Seattle. 1990;ACM Press, New York. pp. 349-356.
67. Norman D.A. Cognitive engineering. Norman D.A., Draper S.W. User Centred System Design: New Perspectives on Human Computer Interaction. 1986;31-61 Erlbaum, Hillsdale, NJ.
68. Parlangeli O., Marchigiani E., Bagnara S. Multimedia systems in distance education: effects of usability on learning. Interacting with Computers. 12:1999;37-49.
69. Pollier, A., 1991. Evaluation d'une interface par des ergonomes: diagnostics et stratégies. Research Report No. 1391. INRIA, France.
70. Polson P.G., Lewis C.H., Rieman J., Wharton C. Cognitive Walkthroughs: a method for theory-based evaluation of use interfaces. International Journal of Man-Machine Studies. 36:1992;741-773.
71. Prümper J. Software Evaluation based upon ISO 9241 Part 10. Greching T., Tscheligi M. Human Computer Interaction Vienna Conference, VCHI'93 Proceedings. 1993;255-265 Springer, Berlin.
72. Ruokamo H., Pohjolainen S. Pedagogical principles for evaluation of hypermedia-based learning environments in mathematics. Journal of Universal Computer Science. 4:(3):1998;292-307.
73. Scapin D.L., Bastien J.M.C. Ergonomic criteria for evaluating the ergonomic quality of interactive systems. Behaviour and Information Technology. 16:1997;220-231.
74. Scholtz, J., Downey, L., 1998. Methods for identifying usability problems with web sites. Proceedings of the 7th International Conference on Engineering for Human-Computer Interaction ECHT'98 (September 14-18, Crète).
75. Scholtz, J., Laskowski, S., 1999. Developing Usability Tools and Techniques for Designing and Testing Web Site. National Institute of Standards and Technology (NIST). Available at: http://www-09.nist.gov/div894/vvrg/jpaper/hf_and_web.htm.
76. Sears A., Hess D.J. The effect of task description detail on evaluator performance with Cognitive Walkthroughs. Karat C.M., Lund A., Coutaz J., Karat J. Proceedings of the Conference on Human Factors in Computing Systems (CHI'98, Los Angeles, CA, April 18-23, 1998). 1998;259-260 ACM Press, New York.
77. Senach, B., 1990. Evaluation ergonomique des interfaces homme-machine: une revue de la littérature. Research Report No. 1180. INRIA, Sophia Antipolis, France, March.
78. Shackel B. Usability, context, framework, definition, design and evaluation. Shackel B., Richardson S. Human Factors for Informatics Usability. 1991;21-37 Cambridge University Press, Cambridge.
79. Simpson N. Managing the use of style guides in an organisational setting: practical lessons in ensuring UI consistency. Interacting with Computers. 11:1999;323-351.
80. Smith P.A. Towards a practical measure of hypertext usability. Interacting with Computers. 8:1996;365-381.
81. Suzuki S., Kaneko A., Ohmura K. QUIS: applying a new walkthrough method to a product design process. Bullinger H.J., Ziegler J. Proceedings of the 8th Human-Computer Interaction International Conference 1999 (HCI'99, Munich, Germany, August 22-26, 1999). vol. 2:1999;933-937 Lawrence Erlbaum Associates, London.
82. Sweeney M., Maguire M., Shackel B. Evaluating user-computer interaction: a framework. International Journal of Man-Machine Studies. 38:1993;689-711.
83. Trigano P. Evaluation de l'interface homme-machine des logiciels éducatifs. Le Journal du Multimédia. 18:1997;12-15.
84. Tullis T.S. A system for evaluating screen formats: research and applications. Hartson H.R., Dix D. Advances in Human-Computer Interactions. vol. 2:1988;214-286 Ablex Norwood, New Jersey.
85. Vanderdonckt J. Guide Ergonomique de la Présentation des Applications Hautement Interactives. 1994;Presses Universitaires de Namur, Namur, Belgium.
86. Vanderdonckt, J., 1998. Conception Ergonomique de Pages WEB. Vesale.
87. Vanderdonckt J. Development milestones towards a tool for working with guidelines. Interacting with Computers. 12:1999;81-118.
88. Virzi R.A. Usability inspection methods. Helander M., Landauer T.K., Prablhu P. Handbook of Human-Computer Interaction. 1997;705-715 Elsevier, Amsterdam.
89. Wharton C., Rieman L., Lewis C., Polson P. The Cognitive Walkthrough method: a practitioner's guide. Nielsen J., Mack R.L. Usability Inspection Methods. 1994;105-140 Elsevier, Amsterdam.
90. Wildman D. Getting the most from paired-user testing. Interactions. 2:(3):1995;21-27.
91. Wills, S., 1994. Beyond browsing: making interactive multimedia interactive. Proceedings of EdTech94 Conference, Rethinking the Role of Education in the Technological Age, May, Singapore, pp. 58-68.
92. Wixon D., Jones S., Tse L., Casaday G. Inspections and design reviews: framework, history, and reflection. Nielsen J., Mack R.L. Usability Inspection Methods. 1994;77-103 Elsevier, Amsterdam.
93. Wright P., Monk A. A cost-effective evaluation method for use by designers. International Journal of Man-Machine Studies. 35:(6):1991;891-912.
94. Ziegler J. Standards for multimedia user interfaces - opportunities and issues, in human-computer interaction: communication, cooperation, and application design. Bullinger H.J., Ziegler J. Proceedings of the 8th Human-Computer Interaction International Conference 1999 (HCI'99, Munich, Germany, August 22-26, 1999). vol. 2:1999;858-862 Lawrence Erlbaum Associates, London.