Information visualization evaluation in large companies: Challenges, experiences and recommendations

Information Visualization

Volumn 10, Issue 3, 2011, Pages 248-266

  Sedlmair, Michael ^a   Isenberg, Petra ^b   Baur, Dominikus ^c   Butz, Andreas ^c  

^a UNIVERSITY OF BRITISH COLUMBIA   (Canada)

^b INRIA   (France)

^c UNIVERSITY OF MUNICH   (Germany)

Author keywords

Company; Field evaluation; Information visualization

Indexed keywords

AUTOMOTIVE COMPANIES; COMPANY; FIELD EVALUATION; INFORMATION VISUALIZATION; PRACTICAL GUIDANCE; VISUAL ENCODINGS;

DATA REDUCTION; INFORMATION ANALYSIS; INFORMATION SYSTEMS; RESEARCH; VISUALIZATION;

DATA VISUALIZATION;

EID: 80054800123     PISSN: 14738716     EISSN: 14738724     Source Type: Journal    
DOI: 10.1177/1473871611413099     Document Type: Article

References (52)

1. Plaisant C. The challenge of information visualization evaluation. Proceedings of the Advanced Visual Interfaces (AVI), ACM: New York, USA, 2004; 109-116.
2. Thomas JJ and Cook KA. Illuminating the Path. IEEE: Los Alamitos, USA, 2005.
3. Ellis G and Dix A. An explorative analysis of user evaluation studies in information visualisation. Proceedings of the Workshop Beyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV), ACM: New York, USA, 2006.
4. Preece J, Rogers Y and Sharp H. Beyond Interaction Design: Beyond Human-Computer Interaction. John Wiley & Sons, Inc.: New York, 2001.
5. Saraiya P, North C and Duca K. An insight-based methodology for evaluating bioinformatics visualizations. IEEE Transactions on Visualization and Computer Graphics 2005; 11(4): 443-456. (Pubitemid 41049906)
6. Tory M and Möller T. Human factors in visualization research. IEEE Transactions on Visualization and Computer Graphics 2004; 10(1): 72-84.
7. Paley WB. Interface and mind - a "paper lecture" about a domain-specific design methodology based on contemporary mind science. it-Information Technology 2009; 51(3): 131-141.
8. McGrath JE. Methodology matters: Doing research in the behavioral and social sciences. Human-Computer Interaction: Toward the Year 2000. Morgan Kaufmann Publishers: San Francisco, CA, USA, 1995, pp.152-169.
9. Daft RL. Organization Theory and Design. West St. Paul, MN, USA: South-Western Pub, 1995.
10. Munzner T. A nested process model for visualization design and validation. IEEE Transactions on Visualization and Computer Graphics 2009; 15(6): 921-928.
11. Isenberg P, Zuk T, Collins C and Carpendale S. Grounded evaluation of information visualizations. Proceedings of the Workshop Beyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV), ACM: New York, USA, 2008.
12. Carpendale S. Evaluating information visualizations. In: Kerren A, Stasko JT, Fekete JD and North C (eds) Information Visualization: Human-Centered Issues and Perspectives. vol. 4950. Berlin: Springer LNCS, 2008, pp.19-45.
13. Sedlmair M, Isenberg P, Baur D and Butz A. Evaluating information visualization in large companies: Challenges, experiences and recommendations. Proceedings of the Workshop on Beyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV). ACM: New York, USA, 2010, pp.79-86.
14. Perer A and Shneiderman B. Integrating statistics and visualization: Case studies of gaining clarity during exploratory data analysis. Proceedings of the Conference on Human Factors in Computing Systems (CHI). ACM: New York, USA, 2008, pp.265-274.
15. Tory M and Staub-French S. Qualitative analysis of visualization: A building design field study. Proceedings of the Workshop Beyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV), ACM: New York, USA, 2008.
16. Shneiderman B and Plaisant C. Strategies for evaluating information visualization tools: Multidimensional in-depth long-term case studies. Proceedings of the Workshop Beyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV), ACM: New York, USA, 2006.
17. González V and Kobsa A. A Workplace study of the adoption of information visualization systems. 3rd International Conference on Knowledge Management (I-KNOW), Graz University of Technology, Austria, 2003; 92-102.
18. McLachlan P, Munzner T, Koutsofios E and North S. LiveRAC: Interactive visual exploration of system management time-series data. Proceedings of the Conference on Human Factors in Computing Systems (CHI). ACM: New York, USA, 2008, pp.1483-1492.
19. Saraiya P, North C, Lam V and Duca KA. An insight-based longitudinal study of visual analytics. IEEE Transactions on Visualization and Computer Graphics 2006; 12: 1511-1522. (Pubitemid 44523094)
20. González V and Kobsa A. Benefits of information visualization systems for administrative data analysts. Proceedings of the International Conference on Information Visualization (IV), IEEE: Los Alamitos, USA, 2003; 331-336.
21. Bak JO, Nguyen K, Risgaard P and Stage J. Obstacles to usability evaluation in practice: A survey of software development organizations. Proceedings of the Nordic Conference on Human-Computer Interaction (NordiCHI). ACM: New York, USA, 2008, pp.23-32.
22. Mao JY, Vredenburg K, Smith PW and Carey T. The state of user-centered design practice. Communications of the ACM 2005; 48(3): 105-109. (Pubitemid 40303819)
23. Grudin J. Obstacles to participatory design in large product development organizations. In: Schuler D and Namioka A (eds) Participatory Design: Principles and Practices. Lawrence Erlbaum Assiociates: Hillsdale, 1993, pp.99-119.
24. Poltrock SE and Grudin J. Organizational obstacles to interface design and development: Two participant-observer studies. ACM Transactions on Computer-Human Interaction (TOCHI) 1994; 1(1): 52-80.
25. Jeffries R, Miller JR, Wharton C and Uyeda K. User interface evaluation in the real world: A comparison of four techniques. Proceedings of the Conference on Human Factors in Computing Systems (CHI). ACM: New York, USA, 1991, pp.119-124.
26. Boivie I, A ̊ borg C, Persson J and Loefberg M. Why usability gets lost or usability in in-house software development. Interac Comput 2003; 15(4): 623-639.
27. Holtzblatt K, Barr J and Holtzblatt L. Driving user centered design into IT organizations: Is it possible? Extended Abstracts of the Conference on Human Factors in Computing Systems (CHI). ACM: New York, USA, 2009, pp.2727-2730.
28. Pugh DS, Hickson DJ and Hinings CR. An empirical taxonomy of structures of work organizations. Administrative Science Quarterly 1969; 14(1): 115-126.
29. Drucker PF. The coming of the new organization. Harvard Business Rev 1988; 66(1): 45-53.
30. Dix A, Finlay J, Abowd GD and Beale R. Human-Computer Interaction. Prentice Hall: Upper Saddle River, NJ, 2004.
31. Beyer H and Holtzblatt K. Contextual Design: Defining Customer- Centered Systems. Morgan Kaufmann Publishers Inc.: San Francisco, 1998.
32. Bernard HR. Social Research Methods: Qualitative and Quantitative Approaches. Sage Publications Inc.: London, 2000.
33. McNeeSMand Arnette B. Productivity as a metric for visual analytics: Reflections on e-discovery. Proceedings of theWorkshop Beyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV). ACM: New York, USA, 2008.
34. Broy M. Challenges in automotive software engineering. Proceedings of the Software Engineering (ICSE), ACM: New York, USA, 2006, pp.33-42.
35. Grimm K. Software technology in an automotive company: Major challenges. Proceedings of the Conference on Software Engineering (ICSE), IEEE: Los Alamitos, USA, 2003, pp.498-503.
36. Heinecke H. Automotive system design-challenges and potential. Proceedings of the Conference Design, Automation and Test (DATE), IEEE: Los Alamitos, USA, 2005, pp.656-657.
37. Pretschner A, Broy M, Kruger IH and Stauner T. Software engineering for automotive systems: A roadmap. Proceedings of the Conference on Software Engineering, IEEE: Los Alamitos, USA, 2007, pp.55-71. (Pubitemid 47484917)
38. Sedlmair M, Hintermaier W, Stocker K, Büring T and Butz A. A dual-view visualization of in-car communication processes. Proceedings of the Conference on Information Visualization (IV), IEEE: Los Alamitos, USA, 2008, pp.157-162.
39. Sedlmair M, Kunze B, Hintermaier W and Butz A. User-centered development of a visual exploration system for in-car communication. Proceedings of the Symposium on Smart Graphics (SG). Springer: Berlin, Germany, 2009, pp.105-116.
40. Sedlmair M, Ruhland K, Hennecke F, Butz A, Bioletti S and O'Sullivan C. Towards the big picture: Enriching 3D models with information visualisation and vice versa. Proceedings of the Symposium on Smart Graphics (SG) Springer: 2009; 27-39.
41. Sedlmair M, Bernhold C, Herrscher D, Boring S and Butz A. MostVis: An interactive visualization supporting automotive engineers in MOST catalog exploration. Procedings of the Conference on Information Visualization (IV), IEEE: Los Alamitos, USA, 2009; 173-182.
42. Sedlmair M, Isenberg P, Baur D, Mauerer M, Pigorsch C and Butz A. Proceedings of the Conference on Human Factors in Computing Systems (CHI), ACM: New York, USA, 2011; 1727-1736.
43. Moggridge B. Designing Interactions. Massachusetts Institute of Technology: Cambridge, MA, USA, 2007.
44. Holtzblatt K and Jones S. Contextual inquiry: A participatory technique for system design. Participatory Design: Principles and Practices. Lawrence Erlbaum Associates: Hillsdale, NJ, 1993, pp.177-210.
45. Tory M and Möller T. Evaluating visualizations: Do expert reviews work? IEEE Computer Graphics and Applications 2005; 25(5): 8-11. (Pubitemid 41555420)
46. Morgan DL. Focus Groups as Qualitative Research. Sage Publications, Inc: London, 1997.
47. Lam H and Munzner T. Increasing the utility of quantitative empirical studies for meta-analysis. Proceedings of the Workshop on Beyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV), ACM: New York, USA, 2008.
48. Kensing F, Simonsen J and Bodker K. MUST: A method for participatory design. Human-Computer Interaction 1998; 13(2): 167-198. (Pubitemid 128447687)
49. Nielsen J and Mack RL. Usability Inspection Methods. John Wiley & Sons, Inc.: New York, NY, 1994.
50. Chen C and Czerwinski MP. Empirical evaluation of information visualizations: An introduction. International Journal of Human Computer Studies 2000; 53(5): 631-635. (Pubitemid 32032814)
51. Munzner T, Barsky A, Williams M. Reflections on QuestVis: A visualization system for an environmental sustainability model. UBC Computer Science Technical Report TR-2009- 24; 2009.
52. Johnson C, Moorhead R, Munzner T, Pfister H, Rheingans P and Yoo TS. NIH-NSF visualization research challenges report. IEEE: Los Alamitos, USA, 2006.