Design study methodology: Reflections from the trenches and the stacks

IEEE Transactions on Visualization and Computer Graphics

Volumn 18, Issue 12, 2012, Pages 2431-2440

  Sedlmair, Michael ^a   Meyer, Miriah ^b   Munzner, Tamara ^a  

^a UNIVERSITY OF BRITISH COLUMBIA   (Canada)

^b UNIVERSITY OF UTAH   (United States)

Author keywords

Design study; framework; methodology; visualization

Indexed keywords

FLOW VISUALIZATION; PROBLEM SOLVING; VISUALIZATION;

DESIGN STUDIES; FRAMEWORK; METHODOLOGICAL APPROACH; METHODOLOGICAL FRAMEWORKS; METHODOLOGY; VISUALIZATION DESIGNS; VISUALIZATION RESEARCH; VISUALIZATION SYSTEM;

DESIGN;

EID: 84867631961     PISSN: 10772626     EISSN: None     Source Type: Journal    
DOI: 10.1109/TVCG.2012.213     Document Type: Article

References (95)

1. M. Agrawala,W. Li, and F. Berthouzoz. Design principles for visual communication. Communications of the ACM, pages 60-69, Apr. 2011.
2. R. Amar, J. Eagan, and J. Stasko. Low-level components of analytic activity in information visualization. IEEE Symp. Information Visualization (InfoVis), pages 111-117, 2005. (Pubitemid 44500045)
3. R. Amar and J. Stasko. Knowledge precepts for design and evaluation of information visualizations. IEEE Trans. Visualization and Computer Graphics (TVCG), 11(4):432-442, 2005. (Pubitemid 41049905)
4. A. Barsky, T. Munzner, J. Gardy, and R. Kincaid. Cerebral: visualizing multiple experimental conditions on a graph with biological context. IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis 2008), 14(6):1253-60, 2008.
5. D. Baur, F. Seiffert, M. Sedlmair, and S. Boring. The streams of our lives: Visualizing listening histories in context. IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis 2010), 16(6):1119-1128, 2010.
6. H. R. Bernard. Research Methods in Anthropology: Qualitative and Quantitative Approaches. AltaMira Press, 5th edition, 2011.
7. H. Beyer and K. Holtzblatt. Contextual design: defining customer-centered systems. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1998.
8. J. Blomberg, J. Giacomi, A. Mosher, and P. Swenton-Wall. Ethnographic field methods and their relation to design. Participatory design: Principles and practices, pages 123-155, 1993.
9. K. Bødker, F. Kensing, and J. Simonsen. Participatory IT design: Designing for business and workplace realities. The MIT Press, 2004.
10. M. Booshehrian, T. Moller, R. M. Peterman, and T. Munzner. Vismon: Facilitating analysis of trade-offs, uncertainty, and sensitivity in fisheries management decision making. Computer Graphics Forum (Proc. EuroVis 2012), 2012. in press.
11. M. A. Borkin, K. Z. Gajos, A. Peters, D. Mitsouras, S. Melchionna, F. J. Rybicki, C. L. Feldman, and H. Pfister. Evaluation of artery visualizations for heart disease diagnosis. IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis 2011), pages 2479-2488, 2011.
12. B. Brown, S. Reeves, and S. Sherwood. Into the wild: Challenges and opportunities for field trial methods. In Proc. ACM Conf. Human Factors in Computing Systems (CHI), pages 1657-1666, 2011.
13. S. Carpendale. Evaluating information visualizations. In A. Kerren, J. T. Stasko, J.- D. Fekete, and C. North, editors, Information Visualization: Human-Centered Issues and Perspectives, volume 4950, pages 19-45. Springer LNCS, 2008.
14. K. Charmaz. Constructing grounded theory. A practical guide through qualitative analysis. Sage Publications, Inc, 2006.
15. A. Cockburn. Agile Software Development: The Cooperative Game. Addison- Wesley Professional, 2006.
16. A. Crabtree, T. Rodden, P. Tolmie, and G. Button. Ethnography considered harmful. In Proc. ACM Conf. Human Factors in Computing Systems (CHI), pages 879-888, 2009.
17. P. Craig and J. Kennedy. Coordinated graph and scatter-plot views for the visual exploration of microarray time-series data. In Proc. IEEE Symp. Information Visualization (InfoVis), pages 173-180, 2003.
18. M. Crotty. The foundations of social research: Meaning and perspective in the research process - Chapter 1. Sage Publications Ltd, 1998.
19. A. Dix. Theoretical analysis and theory creation. In Research Methods in Human- Computer Interaction, pages 175-195. Cambridge Univ. Press, 2008.
20. P. Dourish. Implications for design. In Proc. ACM Conf. Human Factors in Computing Systems (CHI), pages 541-550, 2006. (Pubitemid 44032142)
21. P. Dourish. Responsibilities and implications: further thoughts on ethnography and design. In Proc. ACM Conf. Designing for User eXperiences (DUX), pages 2-16, 2007.
22. S. P. Dow, A. Glassco, J. Kass, M. Schwarz, D. L. Schwartz, and S. R. Klemmer. Parallel prototyping leads to better design results, more divergence, and increased self-efficacy. ACM Trans. Computer-Human Interaction (ToCHI), 17(4):1-24, 2010.
23. K. A. Ericsson and H. A. Simon. Verbal reports as data. Psychological Review, 87(3):215-251, 1980.
24. D. M. Fetterman. Ethnography: Step by Step. Sage Publications, 2nd edition, 1998.
25. D. Furniss, A. Blandford, and P. Curzon. Confessions from a Grounded Theory PhD: Experiences and Lessons Learnt. In ACM Trans. Computer-Human Interaction (ToCHI), pages 113-122, 2011.
26. S. Greenberg and B. Buxton. Usability evaluation considered harmful (some of the time). In Proc. ACM Conf. Human Factors in Computing Systems (CHI), pages 111-120, 2008.
27. G. R. Hayes. The relationship of action research to human-computer interaction. ACM Trans. Computer-Human Interaction (ToCHI), 18(3):1-20, 2011.
28. K. Holtzblatt and S. Jones. Contextual inquiry: A participatory technique for system design. In D. Schuler, A. Namioka (eds.): Participatory Design: Principles and Practices, pages 177-210. Lawrence Erlbaum Associates, 1993.
29. J. Hughes, V. King, and T. Rodden. Moving out from the control room: ethnography in system design. Proc. ACM Conf. Computer Supported CooperativeWork (CSCW), pages 429-439, 1994.
30. J. Hughes, J. O'Brien, and T. Rodden. Designing with ethnography: a presentation framework for design. In Proc. Conf. Designing Interactive Systems (DIS), pages 147-158, 1997.
31. H. Hutchinson et al. Technology probes: inspiring design for and with families. In Proc. ACM Conf. Human Factors in Computing Systems (CHI), pages 17-24, 2003.
32. M. J. Intons-Peterson. Imagery paradigms: How vulnerable are they to experimenters' expectations? Journ. Experimental Psychology - Human Perception and Performance, 9:394-412, 1983.
33. P. Isenberg, T. Zuk, C. Collins, and S. Carpendale. Grounded evaluation of information visualizations. In Proc. ACM CHI Wkshp. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV). 2008.
34. D. F. Keefe, M. Ewert,W. Ribarsky, and R. Chang. Interactive coordinated multipleview visualization of biomechanical motion data. IEEE Trans. Visualization and Computer Graphics (Proc. Vis 2009), 15(6):1383-1390, 2009.
35. R. Kincaid. SignalLens: Focus+Context Applied to Electronic Time Series. IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis 2010), 16(6):900-907, 2010.
36. B. L. Kovitz. Practical Sofware Requirements: A Manual of Content & Style. Manning, 1999.
37. J. Kruger and D. Dunning. Unskilled and unaware of it: How difficulties in recog- nizing one's own incompetence lead to inflated self-assessments. Journ. Personality and Social Psychology, 77(6):1121-1134, 1999.
38. M. Kuniavsky. Observing the User Experience: A Practitioner's Guide To User Research. Morgan Kaufmann, 2003.
39. H. Lam, E. Bertini, P. Isenberg, C. Plaisant, and S. Carpendale. Empirical studies in information visualization: Seven scenarios. IEEE Trans. Visualization and Computer Graphics (TVCG), 2012. Appeared online: 30 Nov. 2011.
40. H. Lam, D. Russell, D. Tang, and T. Munzner. Session Viewer: Visual Exploratory Analysis of Web Session Logs. In Proc. IEEE Symp. Visual Analytics Science and Technology (VAST), pages 147-154, 2007.
41. C. Larman and V. R. Basili. Iterative and incremental development: A brief history. IEEE Computer, 36(6):47-56, 2003.
42. D. Lloyd and J. Dykes. Human-centered approaches in geovisualization design: investigating multiple methods through a long-term case study. IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis 2011), 17(12):498-507, 2011.
43. P. McLachlan, T. Munzner, E. Koutsofios, and S. North. LiveRAC: interactive visual exploration of system management time-series data. In Proc. ACM Conf. on Human Factors in Computing Systems (CHI), pages 1483-1492, 2008.
44. M. Meyer, T. Munzner, A. dePace, and H. Pfister. MulteeSum: A tool for exploring space-time expression data. IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis 2010), 16(6):908-917, 2010.
45. M. Meyer, T. Munzner, and H. Pfister. MizBee: A multiscale synteny browser. IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis 2009), 15(6):897-904, 2009.
46. M. Meyer, B. Wong, M. Styczynski, T. Munzner, and H. Pfister. Pathline: A tool for comparative functional genomics. Computer Graphics Forum (Proc. EuroVis 2010), 29(3):1043-1052, 2010.
47. D. Millen. Rapid ethnography: Time deepening strategies for HCI field research. In Proc. ACM Conf. Designing Interactive Systems (DIS), pages 280-286, 2000.
48. T. Munzner. Interactive Visualization of Large Graphs and Networks. PhD thesis, Stanford University Department of Computer Science, 2000.
49. T. Munzner. Process and pitfalls in writing information visualization research papers. In Information Visualization: Human-Centered Issues and Perspectives, pages 134- 153. Springer LNCS 4950, 2008.
50. T. Munzner. A Nested Model for Visualization Design and Validation. IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis 2009), 15(6):921-928, 2009.
51. T. Munzner, A. Barsky, and M. Williams. Reflections on QuestVis: A visualization system for an environmental sustainability model. In Scientific Visualization: Interactions, Features, Metaphors, volume 2, pages 240-259. Dagstuhl Follow-Ups, 2011.
52. T. Munzner, F. Guimbretiere, and G. Robertson. Constellation: a visualization tool for linguistic queries from MindNet. In Proc. IEEE Symp. Information Visualization (InfoVis), pages 132-135, 1999. (Pubitemid 32213893)
53. T. Munzner, E. Hoffman, K. Claffy, and B. Fenner. Visualizing the Global Topology of the MBone. In Proc. IEEE Symp. Information Visualization (InfoVis), pages 85- 92, 1996.
54. T. Munzner, Q. Kong, R. T. Ng, J. Lee, J. Klawe, D. Radulovic, and C. K. Leung. Visual mining of power sets with large alphabets. Technical report, Univ. British Columbia Computer Science TR-2005-25, Dec. 2005.
55. T. Munzner and S. North. InfoVis 2003 Call for Participation: Papers. http://www.infovis.org/infovis/2003/CFP/#papers.
56. C. B. Nielsen, S. D. Jackman, I. Birol, and S. J. Jones. ABySS-Explorer: Visualizing genome sequence assemblies. IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis 2009), 15(6):881-8, 2009.
57. J. Nielsen. Guerrilla HCI: Using discount usability engineering to penetrate the intimidation barrier. In Cost-Justifying Usability, pages 245-272. Academic Press, 1994.
58. A. Perer, I. Guy, E. Uziel, I. Ronen, and M. Jacovi. Visual Social Network Analytics for Relationship Discovery in the Enterprise. In Proc. IEEE Conf. Visual Analytics Science and Technology (VAST), pages 71-79, 2011.
59. C. Plaisant. The challenge of information visualization evaluation. In Proc. Advanced Visual Interfaces (AVI), pages 109-116. ACM Press, 2004.
60. J. Preece, Y. Rogers, and H. Sharp. Interaction Design: Beyond Human-Computer Interaction. John Wiley & Sons, 2002.
61. A. J. Pretorius and J. J. van Wijk. What does the user want to see? what do the data want to be? Information Visualization, 8(3):153-166, 2009.
62. L. Richardson and E. A. S. Pierre. Writing: A method for inquiry. In Handbook of Qualitative Research, pages 959-978. Sage, 3rd edition, 2005.
63. J. Rode. Reflexivity in Digital Anthropology. In ACM Trans. Computer-Human Interaction (ToCHI), pages 123-132, 2011.
64. Y. Rogers. New theoretical approaches for HCI. Annual review of information science and technology, 38(1):87-143, 2004.
65. Y. Rogers, K. Connelly, L. Tedesco, W. Hazlewood, A. Kurtz, R. Hall, J. Hursey, and T. Toscos. Why it's worth the hassle: the value of in-situ studies when designing Ubicomp. In Proc. Intl. Conf. on Ubiquitous Computing (UbiComp), pages 336-353. Springer-Verlag, 2007.
66. B. E. Rogowitz, L. A. Treinish, and S. Bryson. How not to lie with visualization. Comput. Phys., 10(3):268-273, June 1996.
67. K. Ruhland, M. Sedlmair, S. Bioletti, and C. O'Sullivan. LibViz: Data Visualisation of the Old Library. Intl. Journ. Architectural Computing, 7(1):177-192, 2009.
68. P. Saraiya, C. North, and K. Duca. An insight-based methodology for evaluating bioinformatics visualizations. IEEE Trans. Visualization and Computer Graphics, 11(4):443-456, 2005. (Pubitemid 41049906)
69. P. Saraiya, C. North, V. Lam, and K. Duca. An insight-based longitudinal study of visual analytics. IEEE Trans. Visualization and Computer Graphics, 12(6):1511- 1522, 2006. (Pubitemid 44523094)
70. M. Sedlmair. Car-x-ray: Closely Integrating Both Worlds. In Visual Analysis of In-Car Communication Networks (PhD Thesis). LMU, 2010.
71. M. Sedlmair. ProgSpy2010: Visualizing Flash Activities. In Visual Analysis of In- Car Communication Networks (PhD Thesis). LMU, 2010.
72. M. Sedlmair. WiKeVis: Visualizing Functional Networks. In Visual Analysis of In-Car Communication Networks (PhD Thesis). LMU, 2010.
73. M. Sedlmair, C. Bernhold, D. Herrscher, S. Boring, and A. Butz. MostVis: An Interactive Visualization Supporting Automotive Engineers in MOST Catalog Exploration. In Proc. Intl. Conf. Information Visualization (IV), pages 173-182, 2009.
74. M. Sedlmair, A. Frank, T. Munzner, and A. Butz. RelEx: A design study of visual network analysis in the automotive domain. IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis 2012), 2012. In press.
75. M. Sedlmair, W. Hintermaier, K. Stocker, T. Boring, and A. Butz. A dual-view visualization of in-car communication processes. In Proc. Intl. Conf. Information Visualization (IV), pages 157-162, 2008.
76. M. Sedlmair, P. Isenberg, D. Baur, and A. Butz. Information Visualization Evaluation in Large Companies: Challenges, Experiences and Recommendations. Information Visualization, 10(3):248-266, 2011.
77. M. Sedlmair, P. Isenberg, D. Baur, M. Mauerer, C. Pigorsch, and A. Butz. Cardiogram: Visual Analytics for Automotive Engineers. In Proc. ACM Conf. Human Factors in Computing Systems (CHI), pages 1727-1736, 2011.
78. M. Sedlmair, B. Kunze, W. Hintermaier, and A. Butz. User-centered development of a visual exploration system for in-car communication. In Proc. Intl. Symp. Smart Graphics (SG), pages 105-116, 2009.
79. B. Shneiderman. The eyes have it: A task by data type taxonomy for information visualizations. In Proc. IEEE Symp. Visual Languages (VL), pages 336-343. IEEE, 1996.
80. B. Shneiderman and C. Plaisant. Strategies for evaluating information visualization tools: multi-dimensional in-depth long-term case studies. In Proc. AVI Wkshp. on BEyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV). ACM, 2006.
81. D. Silverman. A Very Short, Fairly Interesting and Reasonably Cheap Book About Qualitative Research. Sage Publications Ltd., 2007.
82. A. Slingsby, J. Dykes, and J. Wood. Configuring hierarchical layouts to address research questions. IEEE Trans. Vis. Comput. Graph., 15(6):977-984, 2009.
83. H. Thimbleby. Write now! In Research Methods in Human-Computer Interaction, pages 196-211. Cambridge Univ. Press, 2008.
84. J. Thomas and K. Cook. Illuminating the path: The research and development agenda for visual analytics. National Visualization and Analytics Center, 2005.
85. M. Tory and T. Moller. Human factors in visualization research. IEEE Trans. Visualization and Computer Graphics (TVCG), 10(1):72-84, 2004.
86. M. Tory and T. Moller. Evaluating visualizations: do expert reviews work? IEEE Computer Graphics and Applications, 25(5):8-11, 2005. (Pubitemid 41555420)
87. J. van Niekerk and J. Roode. Glaserian and Straussian grounded theory: Similar or completely different? In Proc. South African Institute of Computer Scientists and Information Technologists (SAICSIT), pages 96-103, 2009.
88. J. J. van Wijk. Views on visualization. IEEE Trans. Visualization and Computer Graphics (TVCG), 12(4):421-432, 2006. (Pubitemid 43824556)
89. K. L. Wagstaff. Machine Learning that Matters. In Proceedings of the 29th International Conference on Machine Learning. Copyright 2012 California Institute of Technology, June 2012.
90. M. Q. Wang Baldonado, A. Woodruff, and A. Kuchinsky. Guidelines for using multiple views in information visualization. In Proc. ACM Advanced Visual Interfaces (AVI), pages 110-119, 2000.
91. C. Ware. Information Visualization: Perception for Design. Morgan Kaufmann, 2004.
92. W. Willinger, D. Alderson, and J. C. Doyle. Mathematics and the Internet: A source of enormous confusion and great potential. Notices of the American Mathemematical Society, 56(5):586-599, 2009.
93. H. Wolcott. Writing up qualitative research, volume 20 of Qualitative Research Methods. Sage, 3rd edition, 2009.
94. J.Wood, D. Badawood, J. Dykes, and A. Slingsby. BallotMaps: Detecting name bias in alphabetically ordered ballot papers. IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis 2011), 17(12):2384-2391, 2011.
95. T. Zuk, L. Schlesier, P. Neumann, M. S. Hancock, and S. Carpendale. Heuristics for information visualization evaluation. In Proc. ACM CHI Wkshp. on BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV). 2008.