Structuring dimensions for collaborative systems evaluation

ACM Computing Surveys

Volumn 44, Issue 2, 2012, Pages

  Antunes, Pedro ^a   Herskovic, Valeria ^b   Ochoa, Sergio F ^b   Pino, Jose A ^b  

^a UNIVERSITY OF LISBON   (Portugal)

^b UNIVERSITY OF CHILE   (Chile)

Author keywords

Collaborative systems evaluation; Evaluation dimensions; Evaluation guidelines; Human computer interaction; Interaction assessment

Indexed keywords

COLLABORATIVE SYSTEMS; EVALUATION DIMENSIONS; EVALUATION GUIDELINES; INTERACTION ASSESSMENT; PERFORMANCE LEVEL; SYSTEM LIFECYCLE;

PRODUCT DEVELOPMENT;

RATING;

EID: 84858415932     PISSN: 03600300     EISSN: 15577341     Source Type: Journal    
DOI: 10.1145/2089125.2089128     Document Type: Review

References (71)

1. ANTUNES, P. AND COSTA, C. 2003. Perceived value: A low-cost approach to evaluate meetingware. In Proceedings of CRIWG'03. Lecture Notes in Computer Science, vol. 2806, 109-125.
2. ANTUNES, P., FERREIRA, A., AND PINO, J. 2006. Analyzing shared workspace design with human-performance models. In Proceedings of CRIWG'06. Lecture Notes in Computer Science, vol. 4154, 62-77. (Pubitemid 44577348)
3. ANTUNES, P., RAMIRES, J., AND RESPICIO, A. 2006. Addressing the conflicting dimension of groupware: A case study in software requirements validation. Comput. Informatics 25, 523-546.
4. ARAUJO, R., SANTORO, F., AND BORGES, M. 2002. The CSCW lab for groupware evaluation. In Proceedings of CRIWG'02. Lecture Notes in Computer Science, vol. 2440, 222-231.
5. BAECKER, R. M., GRUDIN, J., BUXTON, W., AND GREENBERG, S., EDS. 1995. Human-computer Interaction: Toward the Year 2000. Morgan Kaufmann, San Francisco, CA, 1995.
6. BAEZA-YATES, R. AND PINO, J. 1997. A first step to formally evaluate collaborative work. In Proceedings of the ACM International Conference on Supporting GroupWork (GROUP '97). 55-60.
7. BAEZA-YATES, R. AND PINO, J. 2006. Towards formal evaluation of collaborative work and its application to information retrieval. Info. Res. 11, 4.
8. BAKER, K., GREENBERG, S., AND GUTWIN, C. 2001. Heuristic evaluation of groupware based on the mechanics of collaboration. In Proceedings of the 8th IFIP International Conference on Engineering For Human-Computer interaction. Lecture Notes in Computer Science, vol. 2254, 123-140.
9. BAKER, K., GREENBERG, S., AND GUTWIN, C. 2002. Empirical development of a heuristic evaluation methodology for shared workspace groupware. In Proceedings of the ACM Conference on Computer Supported Cooperative Work. 96-105.
10. BIAS, R. 1991. Interface-walkthroughs: Efficient collaborative testing. IEEE Softw. 8, 5, 94-95.
11. BIAS, R. 1994. The pluralistic usability walkthrough: coordinated empathies. Usability Inspection Methods. J. Nielsen and R. Mack, Eds., John Wiley & Sons, New York, 63-76.
12. BRIGGS, R., ADIKNS, M.,MITTLEMAN, D., KRUSE, J.,MILLER, S., AND NUNAMAKER, J. 1998. A technology transition model derived from field investigation of GSS use aboard the U.S.S. CORONADO. J. Manage. Info. Syst. 15, 3, 151-195. (Pubitemid 32716615)
13. BRIGGS, R., QURESHI, S., AND REINIG, B. 2004. Satisfaction attainment theory as a model for value creation. In Proceedings of the 37th Annual Hawaii International Conference on Systems Sciences, IEEE Computer Society Press.
14. CARD, S.,MORAN, T., AND NEWELL, A. 1980. The keystroke-level model for user performance time with interactive systems. Comm. ACM 23, 7, 396-410. (Pubitemid 11433615)
15. CARROLL, J. 2000. Making use: Scenario-Based Design of Human-Computer Interactions. The MIT Press, Cambridge, MA.
16. COCKTON, G. AND WOOLRYCH, A. 2002. Sale must end: should discount methods be cleared off HCI's shelves? Interactions 9, 5, 13-18.
17. CONEVRTINO, G., NEALE, D., HOBBY, L., CARROLL, J., AND ROSSON, M. 2004. A laboratory method for studying activity awareness. In Proceedings of the 3rd Nordic Conference on Human-Computer Interaction. 313-322.
18. DAMIANOS, L.,HIRSCHMAN, L., KOZIEROK, R., KUTRZ, J., GREENBERG, A.,WALLS, K., LASKOWSKI, S., AND SCHOLTZ, J. 1999. Evaluation for collaborative systems. ACM Comput. Surv. 31, 2, 15.
19. DESANCTIS, G., SNYDER, J., AND POOLE, M. 1994. The meaning of the interface: a functional and holistic evaluation of a meeting software system. Decis. Supp. Syst. 11, 319-335.
20. EREBACK, A. AND HOÜOÜK, K. 1994. Using cognitive walkthrough for evaluating a CSCW application. In Proceedings of the Conference Companion on Human Factors in Computing Systems, 91-92.
21. FERREIRA, A., ANTUNES, P., AND PINO, J. 2009. Evaluating shared workspace performance using human information processing models. Info. Res. 14, 1, 388.
22. FJERMESTAD, J. AND HILTZ, S. 1999. An assessment of group support systems experimental research: methodology and results. J. Manag. Info. Syst. 15, 3, 7-149. (Pubitemid 32716614)
23. GREENBERG, S. AND BUXTON, B. 2008. Usability evaluation considered harmful (some of the time). In Proceedings of the 26th Annual SIGCHI Conference on Human Factors in Computing Systems, 111-120.
24. GUTWIN, C. AND GREENBERG, S. 1999. The effects of workspace awareness support on the usability of real-time distributed groupware. ACM Trans. Comput.-Human Interact. 6, 3, 243-281.
25. GUTWIN, C. AND GREENBERG, S. 2000. The mechanics of collaboration: Developing low cost usability evaluation methods for shared workspaces. In Proceedings of the IEEE International Workshops on Enabling Technologies Infrastructures for Collaborative Enterprises, 98-103.
26. GUTWIN, C., ROSEMAN, M., AND GREENBERG, S. 1996. A usability study of awareness widgets in a shared workspace groupware system. In Proceedings of the ACMConference on Computer Supported Cooperative Work, 258-267.
27. GUY, E. 2005. ". . .real, concrete facts about what works. . .": integrating evaluation and design through patterns. In Proceedings of the International ACM SIGGROUP Conference on Supporting Group Work, 99-108.
28. HAYNES, S., PURAO, S., AND SKATTEBO, A. 2004. Situating evaluation in scenarios of use. In Proceedings of the ACM Conference on Computer Supported Cooperative Work, 92-101. (Pubitemid 40928506)
29. HERSKOVIC, V., PINO, J. A., OCHOA, S. F., AND ANTUNES, P. 2007. Evaluation methods for groupware systems. In Proceedings of CRIWG, Lecture Notes in Computer Science, vol. 4715, 328-336.
30. HIX, D. AND HARTSON, H. R. 1993. Developing User Interfaces: Ensuring Usability Through Product and Process. John Wiley & Sons, Inc., New York, NY.
31. HUANG, J. P. H. 2005. A conceptual framework for understanding collaborative systems evaluation. In Proceedings of the 14th IEEE International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprise. 215-220. (Pubitemid 46090876)
32. HUGHES, J., KING, V., RODDEN, T., AND ANDERSEN, H. 1994a. Moving out from the control room: ethnography in system design. In Proceedings of the ACM Conference on Computer Supported Cooperative Work. 429-439.
33. HUGHES, J., SHARROCK, W., RODDEN, T., O'BRIEN, J., ROUNCEFIELD, M., AND CALVEY, D. 1994b. Field Studies and CSCW. Lancaster University, Lancaster, U.K.
34. HUMPHRIES, W.,NEALE, D.,MCCRICKARD, D., AND CARROLL, J. 2004. Laboratory simulation methods for studying complex collaborative tasks. In Proceedings of the 48th Annual Meeting Human Factors and Ergonomics Society, 2451-2455.
35. INKPEN, K.,MANDRYK, R., DIMICCO, J., AND SCOTT, S. 2004. Methodology for evaluating collaboration behaviour in co-located environments. In Proceedings of the ACM Conference on Computer Supported Cooperative Work.
36. KIERAS, D. AND SANTORO, T. 2004. Computational GOMS modeling of a complex team task: lessons learned. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 97-104.
37. MCGRATH, J. 1984. Groups: Interaction and Performance. Prentice-Hall, Englewood Cliffs, NJ.
38. NEALE, D. AND CARROLL, J. 1999. Multi-faceted evaluation for complex, distributed activities. In Proceedings of the Conference on Computer Support For Collaborative Learning. 425-433.
39. NEALE, D., CARROLL, J., AND ROSSON, M. 2004. Evaluating computer-supported cooperative work: models and frameworks. In Proceedings of the ACM Conference on Computer Supported Cooperative Work, 112-121. (Pubitemid 40928508)
40. NEWELL, A. 1990. Unified Theories of Cognition. Harvard University Press, Cambridge, MA.
41. NIELSEN, J. 1989. Usability engineering at a discount. In Designing and Using Human-Computer Interfaces and Knowledge Based Systems, G. Salvendy and M. Smith, Eds., Elsevier Science Publishers, Amsterdam, 394-401.
42. NIELSEN, J. 1994. Usability inspection methods. In Proceedings of the Conference on Human Factors in Computing Systems, 413-414.
43. NIELSEN, J. AND MOLICH, R. 1990. Heuristic evaluation of user interfaces. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, 249-256.
44. OCHOA, S., PINO, J., BRAVO, G., DUJOVNE, N., AND NEYEM, A. 2008. Mobile shared workspaces to support construction inspection activities. In Collaborative Decision Making: Perspectives and Challenges, P. Zarate, J. Belaud, G. Camileri, and F. Ravat, Eds., IOS Press, Amsterdam, 211-220.
45. PIDD, M. 1996. Tools for Thinking. J. Wiley & Sons, Chichester.
46. PINELLE, D. AND GUTWIN, C. 2000. A review of groupware evaluations. In Proceedings of the 9th IEEE WETICE Infrastructure for Collaborative Enterprises, 86-91.
47. PINELLE, D. AND GUTWIN, C. 2002. Groupware walkthrough: adding context to groupware usability evaluation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 455-462.
48. PINELLE, D. AND GUTWIN, C. 2008. Evaluating teamwork support in tabletop groupware applications using collaboration usability analysis. Pers. Ubiq. Comput. 12, 3, 237-254. (Pubitemid 351176346)
49. PINELLE, D., GUTWIN, C., AND GREENBERG, S. 2003. Task analysis for groupware usability evaluation: modeling shared-workspace tasks with the mechanics of collaboration. ACM Trans. Comput.-Human Interact. 10, 4, 281-311.
50. PINSONNEAULT, A. AND KRAEMER, K. 1989. The impact of technological support on groups: an assessment of the empirical research. Decis. Supp. Syst. 5, 3, 197-216.
51. PLOWMAN, L., ROGERS, Y., AND RAMAGE, M. 1995. What are workplace studies for? In Proceedings of the 4th European Conference on Computer-Supported Cooperative Work, 309-324.
52. POLSON, P. G., LEWIS, C., RIEMAN, J., AND WHARTON, C. 1992. Cognitive walkthroughs: a method for theorybased evaluation of user interfaces. Int. J. Man-Mach. Stud. 36, 5, 741-773.
53. RASMUSSEN, J. AND JENSEN, A. 1974. Mental procedures in real-life tasks : a case-study of electronic trouble shooting. Ergonomics 17, 293-307.
54. REASON, J. 2008. The Human Contribution: Unsafe Acts, Accidents and Heroic Recoveries. Ashgate, Surrey, UK.
55. ROSS, S., RAMAGE, M., AND ROGERS, Y. 1995. PETRA: participatory evaluation through redesign and analysis. Interact. Comput. 7, 4, 335-360.
56. ROWLEY, D. AND RHOADES, D. 1992. The cognitive jogthrough: a fast-paced user interface evaluation procedure. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 389-395. (Pubitemid 23609815)
57. RUHLEDER, K. AND JORDAN, B. 1998. Video-based interaction analysis (VBIA) in distributed settings: a tool for analyzing multiple-site, technology-supported interactions. In Proceedings of the Participatory Design Conference, 195-196.
58. SCRIVEN, M. 1967. The methodology of evaluation. In Perspectives of Curriculum Evaluation, R. Tyler, R. Gagne, and M. Scriven, Eds., Rand McNally, Chicago, 39-83.
59. SONNENWALD, D.,MAGLAUGHLIN, K., AND WHITTON, M. 2001. Using innovation diffusion theory to guide collaboration technology evaluation: work in progress. In Proceedings of the IEEE International Workshop on Enabling Technologies, 114-119.
60. STEVES, M., MORSE, E., GUTWIN, C., AND GREENBERG, S. 2001. A comparison of usage evaluation and inspection methods for assessing groupware usability. In Proceedings of the International ACM SIGGROUP Conference on Supporting Group Work, 125-134. (Pubitemid 32950977)
61. STIEMERLING, O. AND CREMERS, A. 1998. The use of cooperation scenarios in the design and evaluation of a CSCW system. IEEE Trans. Softw. Eng. 24, 12, 1171-1181. (Pubitemid 128741301)
62. SUCHMAN, L. 1987. Plans and Situated Actions: The Problem of Human-Machine Communication. Cambridge University Press, Cambridge, U.K.
63. TANG, J. 1991. Findings from observational studies of collaborative work. Intern. J. Man-Machine Stud. 34, 2, 143-160.
64. TWIDALE, M., RANDALL, D., AND BENTLEY, R. 1994. Situated evaluation for cooperative systems. In Proceedings of the ACM Conference on Computer Supported Cooperative Work, 441-452.
65. URQUIJO, S., SCRIVENER, S., AND PALMEN, H. 1993. The use of breakdown analysis in synchronous CSCW system design. In Proceedings of the 3rd European Conference on Computer Supported Cooperative Work-ECSCW 93, 289-302.
66. VAN DER VEER, G. 2000. Task based groupware design: putting theory into practice. In Proceedings of the Symposium on Designing Interactive Systems. 326-337.
67. VAN DER VEER, G., LENTING, B., AND BERGEVOET, B. 1996. Gta: Groupware task analysis-modeling complexity. Acta Psychologica 91, 3, 297-322. (Pubitemid 126375256)
68. VELD, M. A. A. H. I. T., ANDRIESSEN, J. H. E., AND VERBURG, R. M. 2003. E-MAGINE: The development of an evaluation method to assess groupware applications. In Proceedings of the 12th International Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprises, 153-158.
69. VIZCÁINO, A., MARTINEZ, M., ARANDA, G., AND PIATTINI, M. 2005. Evaluating collaborative applications from a knowledgemanagement approach. In Proceedings of the 14th IEEE InternationalWorkshops on Enabling Technologies: Infrastructure for Collaborative Enterprise (WETICE'05), 221-225. (Pubitemid 46090877)
70. WHARTON, C., RIEMAN, J., LEWIS, C., AND POLSON, P. 1994. The cognitive walkthrough method: a practictioner's guide. In Usability Inspection Methods, J. Nielsen and R. Mack, Eds., John Wiley & Sons, New York, 105-140.
71. YOURDON, E. 1978. Structured Walkthroughs. Yourdon Inc, New York.