Scenario-based methods for evaluating collaborative systems

Computer Supported Cooperative Work

Volumn 18, Issue 4, 2009, Pages 331-356

  Haynes, Steven R ^a   Purao, Sandeep ^a   Skattebo, Amie L ^a  

^a PENNSYLVANIA STATE UNIVERSITY   (United States)

Author keywords

Claims analysis; Collaborative systems; Evaluation methods; Field studies; Scenario based approaches

Indexed keywords

CLAIMS ANALYSIS; COLLABORATIVE SYSTEMS; EVALUATION METHODS; FIELD STUDIES; SCENARIO-BASED APPROACHES;

INSPECTION;

EID: 68349144690     PISSN: 09259724     EISSN: 15737551     Source Type: Journal    
DOI: 10.1007/s10606-009-9095-x     Document Type: Article

References (58)

1. M Ackerman 2000 The intellectual challenge of CSCW: the gap between social requirements and technical flexibility Human-Computer Interaction 15 2&3 179 203
2. SE Asche 1955 Opinions and social pressures Scientific American 193 5 31 35
3. JE Bardram 2000 Scenario-based design of cooperative systems: redesigning an hospital information system in Denmark Group Decision and Negotiation 9 237 250
4. Bass, L., Clements, P., & Kazman, R. (1998). Software architecture in practice. Reading, MA: Addison-Wesley.
5. Bernstein, A., & Vorburger, P. (2006). A scenario-based approach for direct interruptability prediction on wearable devices. Journal of Pervasive Computing and Communication, 2(2).
6. Beyer, H., & Holtzblatt, K. (1998). Contextual design: defining customer-centered systems. San Francisco, Calif.: Morgan Kaufmann Publishers.
7. A Blandford 2002 A case study on integrating contextual information with analytical usability evaluation International Journal of Human-Computer Studies 57 75 99
8. A Blanford G Rugg 2002 A case study on integrating contextual information with analytical usability evaluation International Journal of Human-Computer Studies 57 75 99
9. Blythin, S., Hughes, J. A., Kristoffersen, S., Tom, R., & Rouncefield, M. (1997). Recognising "success" and "failure": evaluating groupware in a commercial context. Proceedings of the international ACM SIGGROUP conference on Supporting group work: the integration challenge: the integration challenge, Phoenix, Arizona, United States, pp. 39-46.
10. G Button 2000 The ethnographic tradition and design Design Studies 21 4 319 332
11. M Campion E Pursell B Brown 1988 Structured interviewing: raising the psychometric properties of the employment interview Personnel Psychology 41 25 42
12. Carroll, J. M. (1995). Scenario-based design: Envisioning work and technology in system development. New York: Wiley.
13. Carroll, J. M. (2000). Making use: Scenario-based design of human-computer interactions. Cambridge, MA: MIT Press.
14. Carroll, J. M., Koenemann-Belliveau, J., Rosson, M. B., & Singley, M. K. (1993). People and computers VIII. In J. L. Alty, D. Diaper & S. Guest (Eds.), Critical incidents and critical themes in empirical usability evaluation (pp. 279-292). Cambridge, UK: Cambridge University Press.
15. JM Carroll MB Rosson 1992 Getting around the task-artifact cycle: how to make claims and design by scenario ACM Transactions on Information Systems 10 2 181 212
16. CM Chewar DS McCrickard JM Carroll 2005 Analyzing the social capital value chain in community network interfaces Internet Research 15 3 262 280
17. Clements, P., Kazman, R., & Klein, M. (2001). Evaluating software architectures: Methods and case studies. Boston: Addison-Wesley.
18. J Cohen 1960 A coefficient for agreement for nominal scales Education and Psychological Measurement 20 37 46
19. Damianos, L., Drury, J., Fanderclai, T., Hirschman, L., Kurtz, J., & Oshika, B. (2000). Scenario-Based Evaluation of Loosely-Integrated Collaborative Systems (pp. 127-128). The Hague: Conference on Computer-Human Interaction (CHI).
20. Damianos, L., Hirschman, L., Kozierok, R., Kurtz, J., Greenberg, A., Walls, K., Laskowski, S., & Scholtz, J. (1999). Evaluation for collaborative systems. ACM Computing Surveys, 31(3es).
21. De Bruin, H., & Van Vliet, H. (2001). Scenario-based generation and evaluation of software architectures. Proceedings of the third international conference on generative and component-based software engineering; Lecture Notes in Computer Science, pp. 128-13
22. P Dewan 2001 An integrated approach to designing and evaluating collaborative applications and infrastructures Computer Supported Cooperative Work 10 1 75 111
23. J Drury 1999 Methodologies for evaluation of collaborative systems workshop SIGGROUP Bull 20 2 50 51
24. Ge, G., & Whitehead, J. J. (2006). Scenario-based evaluation for configuration management systems (Technical Report): School of Engineering University of California, Santa Cruz.
25. Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. Chicago: Aldine de Gruyter.
26. WD Gray MC Salzman 1998 Damaged merchandise? A review of experiments that compare usability evaluation methods Human-Computer Interaction 13 203 261
27. Grudin, J. (1988). Why CSCW applications fail: Problems in the design and evaluation of organizational interfaces. Proceedings of the 1988 ACM Conference on computer-supported cooperative work, Portland, Oregon, pp. 85-93.
28. J Grudin 1994 Groupware and social dynamics: eight challenges for developers Communications of the ACM 37 1 92 105
29. Grudin, J. (2004). Return on investment and organizational adoption. Proceedings of the 2004 ACM conference on Computer supported cooperative work, Chicago, Illinois, pp. 324-327.
30. Hughes, J., King, V., Rodden, T., & Andersen, H. (1994). Moving out from the control room: ethnography in system design, Proceedings of the 1994 ACM conference on computer supported cooperative work. Chapel Hill, North Carolina, United States: ACM Press.
31. Ionita, M. T., Hammer, D. K., & Obbink, H. (2002). Scenario-based software architecture evaluation methods: An overview. Orlando, FL: Workshop on Methods and Techniques for Software Architecture Review and Assessment at the International Conference on Software Engineering.
32. John, B. E. (1996). Evaluating usability evaluations. ACM Computing Surveys, 28(4).
33. Kahn, H. (1962). Thinking about the unthinkable. New York: Avon.
34. Kahn, H. (1967). The Year 2000 a framework for speculation on the next thirty-three years. In H. Kahn & A. J. Wiener (Eds.), The use of scenarios (pp. 262-264). New York: Macmillan.
35. MacLean, A., & McKerlie, D. (1995). Scenario-based design: Envisioning work and technology in system development. In J. M. Carroll (Ed.), Design space analysis and use-representations (pp. 183-207). New York: John Wiley.
36. McGrenere, J. (2000). Bloat: The objective and subject dimensions (pp. 337-338). The Hague: CHI '00 extended abstracts on Human factors in computing systems.
37. Mukkamalla, R., Britton, M., & Sundaram, P. (2002). Scenario-based specifications and evaluation of architectures for health monitoring of aerospace structures. Proceedings of the 21st Digital Avionics Systems Conference, pp. 1-12.
38. Nielsen, J. (1994). Usability engineering: Morgan Kaufmann.
39. Nielsen, J. (1995). Scenario-based design: Envisioning work and technology in system development. In J. M. Carroll (Ed.), Scenarios in discount usability engineering (pp. 59-83). New York: John Wiley.
40. Nielsen, J., & Mack, R. L. (1994). Usability inspection methods. New York, NY: John Wiley & Sons.
41. Oppenheim, A. N. (1992). Questionnaire design, interviewing and attitude measurement (2nd ed.). London: Pinter Publishers.
42. W Orlikowski D Gash 1994 Technological frames: making sense of information technology in organizations ACM Transactions on Information Systems 12 2 174 207
43. Pang, N. L. S., Sanxing, C., Schauder, D., & Klein, R. R. (2005). A hybrid approach in the evaluation of usability for multimedia objects: case study of the media assets management platform for an advertainment production project toward Beijing Olympics 2008. Proceedings of the 3rd International Conference on Information Technology and Applications (ICITA), pp. 82-87.
44. Pinelle, D., & Gutwin, C. (2002). Groupware walkthrough: adding context to groupware usability evaluation. Proceedings of the SIGCHI conference on human factors in computing systems: Changing our world, changing ourselves, CHI '02, Minneapolis, Minnesota, USA, pp. 455-462.
45. PM Podsakoff SB MacKenzie J Lee NP Podsakoff 2003 Common method biases in behavioral research: a critical review of the literature and recommended remedies Journal of Applied Psychology 88 5 879 903
46. PG Polson C Lewis J Rieman C Wharton 1992 Cognitive walkthroughs: a method for theory- based evaluation of user interfaces International Journal of Man-Machine Studies 36 741 773
47. Ramage, M. (1999). The learning way: Evaluating cooperative systems. Lancaster, UK: Lancaster University.
48. Y Rogers V Bellotti 1997 Grounding blue-sky research: how can ethnography help? Interactions 4 3 58 63
49. S Ross M Ramage Y Rogers 1995 PETRA: Participatory Evaluation Through Redesign and Analysis Interacting with Computers 7 4 335 360
50. M Scriven 1996 Types of evaluation and types of evaluator Evaluation Practice 17 2 151 162
51. PB Seddon V Gaeser LP Willcocks 2002 Measuring organizational IS effectiveness: An overview and update of senior management perspectives The DATA BASE for Advances in Information Systems 33 2 11 28
52. J Sokolov 1999 Methodologies for evaluation of collaborative systems SIGGROUP Bulletin 20 2 44 46
53. Steves, M. P., Morse, E., Gutwin, C., & Greenberg, S. (2001). A comparison of usage evaluation and inspection methods for assessing groupware usability. In Proceedings of the 2001 International ACM SIGGROUP Conference on Supporting Group Work, pp. 125-134. USA: ACM Press.
54. O Stiemerling AB Cremers 1998 The use of cooperation scenarios in the design and evaluation of a CSCW system IEEE Transactions on Software Engineering 24 12 1171 1181
55. Sutcliffe, A. (2002). The domain theory: Patterns for knowledge and software reuse. Mahwah, NJ: Lawrence Erlbaum.
56. Twidale, M., Randall, D., & Bentley, R. (1994). Situated evaluation for cooperative systems. Proceedings of the 1994 ACM conference on computer supported cooperative work, Chapel Hill, North Carolina, pp. 441-452.
57. Winckler, M. A., Palanque, P., & Freitas, C. M. D. S. (2004). Tasks and scenario-based evaluation of information visualization techniques. Proceedings of the 3rd Annual ACM international conference on task models and diagrams, Prague, Czech Republic, pp. 165-172.
58. Yin, R. K. (2003). Case study research: Design and methods (3rd ed.). Thousand Oaks: Sage.