Empirical studies in information visualization: Seven scenarios

IEEE Transactions on Visualization and Computer Graphics

Volumn 18, Issue 9, 2012, Pages 1520-1536

  Lam, Heidi ^a   Bertini, Enrico ^b   Isenberg, Petra ^c   Plaisant, Catherine ^d   Carpendale, Sheelagh ^e  

^a GOOGLE INC   (United States)

^b UNIVERSITY OF KONSTANZ   (Germany)

^c INRIA   (France)

^d UNIVERSITY OF MARYLAND   (United States)

^e UNIVERSITY OF CALGARY   (Canada)

Author keywords

evaluation; Information visualization

Indexed keywords

EMPIRICAL STUDIES; EVALUATION; INFORMATION VISUALIZATION; LITERATURE REVIEWS; RESEARCH QUESTIONS; USER EXPERIENCE; USER PERFORMANCE; VISUAL DATA ANALYSIS; VISUALIZATION ALGORITHMS; WORK PRACTICES;

DISTILLATION; INFORMATION SYSTEMS; RESEARCH; VISUALIZATION;

INFORMATION ANALYSIS;

EID: 84864130051     PISSN: 10772626     EISSN: None     Source Type: Journal    
DOI: 10.1109/TVCG.2011.279     Document Type: Review

References (95)

1. D.L. Alonso, A. Rose, C. Plaisant, and K.L. Norman, "Viewing Personal History Records: A Comparison of Tabular Format and Graphical Presentation Using Lifelines," Behaviour & Information Technology, vol. 17, no. 5, pp. 249-262, 1998. (Pubitemid 128608832)
2. K. Andrews, "Evaluation Comes in Many Guises," Proc. CHI Workshop BEyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV), pp. 7-8, 2008.
3. A.O. Artero, M.C.F. de Oliveira, and H. Levkowitz, "Uncovering Clusters in Crowded Parallel Coordinates Visualizations," Proc. IEEE Symp. Information Visualization (InfoVis), pp. 81-88, 2004. (Pubitemid 40449131)
4. K. Baker, S. Greenberg, and C. Gutwin, "Heuristic Evaluation of Groupware Based on the Mechanics of Collaboration," Proc. Int'l Conf. Eng. for Human-Computer Interaction, pp. 123-139, 2001.
5. L. Barkhuus and J.A. Rode, "From Mice to Men: 24 Years of Evaluation in CHI," Proc. alt.chi: Extended Abstracts of the Conf. Human Factors in Computing Systems (CHI), 2007.
6. L. Bartram and C. Ware, "Filtering and Brushing with Motion," Information Visualization, vol. 1, no. 1, pp. 66-79, 2002.
7. B.B. Bederson, B. Shneiderman, and M. Wattenberg, "Ordered and Quantum Treemaps: Making Effective Use of 2D Space to Display Hierarchies," ACM Trans. Graphics, vol. 21, no. 4, pp. 833-854, 2002.
8. E. Bertini, A. Perer, C. Plaisant, and G. Santucci, "Beyond Time and Errors: Novel Evaluation Methods for Information Visualization (beliv)," Proc. Extended Abstracts of the Conf. Human Factors in Computing Systems (CHI), pp. 3913-3916, 2008.
9. I. Brewer, A.M. MacEachren, H. Abdo, J. Gundrum, and G. Otto, "Collaborative Geographic Visualization: Enabling Shared Understanding of Environmental Processes," Proc. IEEE Symp. Information Visualization (InfoVis), pp. 137-141, 2000. (Pubitemid 32072851)
10. S. Carpendale, "Evaluating Information Visualizations," Information Visualization: Human-Centered Issues and Perspectives, A. Kerren, J.T. Stasko, J.-D. Fekete, and C. North, eds., pp. 19-45, Springer, 2007.
11. C. Chen and Y. Yu, "Empirical Studies of Information Visualization: A Meta-Analysis," Int'l J. Human-Computer Studies, vol. 53, no. 5, pp. 851-866, 2000. (Pubitemid 32032824)
12. J. Chen, A.M. MacEachren, and D.J. Peuquet, "Constructing Overview + Detail Dendrogram-Matrix Views," IEEE Trans. Visualization and Computer Graphics, vol. 15, no. 6, pp. 889-896, Nov. 2009.
13. Y. Chen, L. Wang, M. Dong, and J. Hua, "Exemplar-Based Visualization of Large Document Corpus," IEEE Trans. Visualization and Computer Graphics, vol. 15, no. 6, pp. 1161-1168, Nov./Dec. 2009.
14. R.K. Christopher, C.G. Healey, V. Interrante, D.H. Laidlaw, and C. Ware, "Thoughts on User Studies: Why, How, and When," IEEE Computer Graphics and Applications, vol. 23, no. 4, pp. 20-25, July 2003.
15. J. Corbin and A. Strauss, Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. Sage Publications, 2008.
16. L. Costello, G. Grinstein, C. Plaisant, and J. Scholtz, "Advancing User-Centered Evaluation of Visual Analytic Environments through Contests," Information Visualization, vol. 8, pp. 230-238, 2009.
17. J.W. Creswell, Research Design, Qualitative, Quantitative, and Mixed Methods Approaches, second ed. Sage Publications, Inc., 2002.
18. M.P. Czerwinski, M.V. Dantzich, G. Robertson, and H. Hoffman, "The Contribution of Thumbnail Image, Mouse-over Text and Spatial Location Memory to Web Page Retrieval in 3D," Proc. Int'l Conf. INTERACT, pp. 163-170, 1999.
19. J. Drury and M.G. Williams, "A Framework for Role-Based Specification and Evaluation of Awareness Support in Synchronous Collaborative Applications," Proc. Workshops Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE), pp. 12-17, 2002.
20. T. Dwyer and D.R. Gallagher, "Visualising Changes in Fund Manager Holdings in Two and a Half-Dimensions," Information Visualization, vol. 3, no. 4, pp. 227-244, 2004.
21. R. Eccles, T. Kapler, R. Harper, and W. Wright, "Stories in Geotime," Information Visualization, vol. 7, no. 1, pp. 3-17, 2008. (Pubitemid 351464149)
22. G. Ellis and A. Dix, "An Exploratory Analysis of User Evaluation Studies in Information Visualization," Proc. AVI Workshop BEyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV), 2006.
23. L. Grammel, M. Tory, and M.-A. Storey, "How Information Visualization Novices Construct Visualizations," IEEE Trans. Visualization and Computer Graphics, vol. 16, no. 6, pp. 943-952, Nov./Dec. 2010.
24. S. Greenberg and B. Buxton, "Usability Evaluation Considered Harmful (Some of the Time)," Proc. Conf. Human Factors in Computing Systems (CHI), pp. 217-224, 2008.
25. S. Greenberg, G. Fitzpatrick, C. Gutwin, and S. Kaplan, "Adapting the Locales Framework for Heuristic Evaluation of Groupware," Australian J. Information Systems, vol. 7, no. 2, pp. 102-108, 2000.
26. S. Greenberg, "Observing Collaboration: Group-Centered Design," HCI Remixed: Reflections on Works That Have Influenced the HCI Community, T. Erickson, and D.W. McDonald, eds., chapter 18, pp. 111-118, MIT Press, 2008.
27. J. Grudin, "Why Cscw Applications Fail: Problems in the Design and Evaluation of Organizational Interfaces," Proc. Conf. Computer-Supported Cooperative Work (CSCW), pp. 85-93, 1988.
28. C. Gutwin and S. Greenberg, "The Mechanics of Collaboration: Developing Low Cost Usability Evaluation Methods for Shared Workspaces," Proc. Workshops Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE), pp. 98-103, 2000.
29. M. Hao, D.A. Keim, U. Dayal, D. Oelke, and C. Tremblay, "Density Displays for Data Stream Monitoring," Computer Graphics Forum, vol. 27, no. 3, pp. 895-902, 2008.
30. J. Heer and M. Bostock, "Crowdsourcing Graphical Perception: Using Mechanical Turk to Assess Visualization Design," Proc. Conf. Human Factors in Computing Systems (CHI), pp. 203-212, 2010.
31. J. Heer and D. Boyd, "Vizster: Visualizing Online Social Networks," Proc. Symp. Information Visualization (InfoVis), pp. 33-40, 2005. (Pubitemid 44500035)
32. J. Heer and G. Robertson, "Animated Transitions in Statistical Data Graphics," IEEE Trans. Visualization and Computer Graphics, vol. 13, no. 6, pp. 1240-1247, Nov./Dec. 2007. (Pubitemid 350076595)
33. J. Heer, F.B. Viégas, and M. Wattenberg, "Voyagers and Voyeurs: Supporting Asynchronous Collaborative Information Visualization," Proc. Conf. Human Factors in Computing Systems (CHI), pp. 1029-1038, 2007. (Pubitemid 47581694)
34. D.M. Hilbert and D.F. Redmiles, "Extracting Usability Information from User Interface Events," ACM Computing Survey, vol. 32, no. 4, pp. 384-421, 2000.
35. U. Hinrichs, H. Schmidt, and S. Carpendale, "EMDialog: Bringing Information Visualization into the Museum," IEEE Trans. Visualization and Computer Graphics, vol. 14, no. 6, pp. 1181-1188, Nov./Dec. 2008.
36. K. Holtzblatt and S. Jones, Contextual Inquiry: A Participatory Technique for Systems Design. Lawrence Earlbaum, 1993.
37. K. Hornbæk and E. Frokjær, "Reading of Electronic Documents: The Usability of Linear, Fisheye and Overview+Detail Interfaces," Proc. Conf. Human Factors in Computing Systems (CHI), pp. 293-300, 2001. (Pubitemid 32393369)
38. P. Isenberg, A. Bezerianos, N. Henry, S. Carpendale, and J.-D. Fekete, "CoCoNutTrix: Collaborative Retrofitting for Information Visualization," Computer Graphics and Applications: Special Issue on Collaborative Visualization, vol. 29, no. 5, pp. 44-57, 2009.
39. P. Isenberg, D. Fisher, M. Ringel Morris, K. Inkpen, and M. Czerwinski, "An Exploratory Study of Co-Located Collaborative Visual Analytics around a Tabletop Display," Proc. Conf. Visual Analytics Science and Technology (VAST), pp. 179-186, 2010.
40. P. Isenberg, A. Tang, and S. Carpendale, "An Exploratory Study of Visual Information Analysis," Proc. Conf. Human Factors in Computing Systems (CHI), pp. 1217-1226, 2008.
41. P. Isenberg, T. Zuk, C. Collins, and S. Carpendale, "Grounded Evaluation of Information Visualizations," Proc. CHI Workshop BEyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV), pp. 56-63, 2008.
42. M.Y. Ivory and M.A. Hearst, "The State of the Art in Automating Usability Evaluation of User Interfaces," ACM Computing Surveys, vol. 33, no. 4, pp. 470-516, 2001. (Pubitemid 33720523)
43. F. Janoos, S. Singh, O. Irfanoglu, R. Machiraju, and R. Parent, "Activity Analysis Using Spatio-Temporal Trajectory Volumes in Surveillance Applications," Proc. Symp. Visual Analytics Science and Technology (VAST), pp. 3-10, 2007.
44. Y.-A. Kang, C. Görg, and J. Stasko, "Evaluating Visual Analytics Systems for Investigative Analysis: Deriving Design Principles from a Case Study," Proc. Symp. Visual Analytics Science and Technology (VAST), pp. 139-146, 2009.
45. A. Kittur, E.H. Chi, and B. Suh, "Crowdsourcing User Studies with Mechanical Turk," Proc. Conf. Human Factors in Computing Systems (CHI), pp. 453-456, 2008.
46. O. Kulyk, R. Kosara, J. Urquiza, and I. Wassin, "Human-Centered Aspects," Human-Centered Visualization Environments, G. Goos, J. Hartmanis, and J. van Leeuwen, eds., pp. 13-76, Springer, 2007.
47. H. Lam and T. Munzner, "Increasing the Utility of Quantitative Empirical Studies for Meta-Analysis," Proc. CHI Workshop BEyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV), pp. 21-27, 2008.
48. H. Lam, R.A. Rensink, and T. Munzner, "Effects of 2D Geometric Transformations on Visual Memory," Proc. Symp. Applied Perception in Graphics and Visualization (APGV), pp. 119-126, 2006. (Pubitemid 46957064)
49. J.D. Mackinlay, P. Hanrahan, and C. Stolte, "Show Me: Automatic Presentation for Visual Analysis," IEEE Trans. Visualization and Computer Graphics, vol. 13, no. 6, pp. 1137-1144, Nov./Dec. 2007. (Pubitemid 350076582)
50. N. Mahyar, A. Sarvghad, and M. Tory, "A Closer Look at Note Taking in the Co-Located Collaborative Visual Analytics Process," Proc. Conf. Visual Analytics Science and Technology (VAST), pp. 171-178, 2010.
51. R. Mandryk, "Modeling User Emotion in Interactive Play Environments: A Fuzzy Physiological Approach," PhD thesis, Simon Fraser Univ., 2005.
52. G. Mark and A. Kobsa, "The Effects of Collaboration and System Transparency on CIVE Usage: An Empirical Study and Model," Presence, vol. 14, no. 1, pp. 60-80, 2005. (Pubitemid 40609711)
53. J. McGrath, "Methodology Matters: Doing Research in the Behavioral and Social Sciences," Readings in Human-Computer Interaction: Toward the Year 2000, Morgan Kaufmann, 1994.
54. T. Munzner, "A Nested Process Model for Visualization Design and Validation," IEEE Trans. Visualization and Computer Graphics, vol. 15, no. 6, pp. 921-928, Nov./Dec. 2009.
55. D.C. Neale, J.M. Carroll, and M.B. Rosson, "Evaluating Computer-Supported Cooperative Work: Models and Frameworks," Proc. Conf. Computer-Supported Cooperative Work (CSCW), pp. 112-121, 2004. (Pubitemid 40928508)
56. W. Peng, M.O. Ward, and E.A. Rundensteiner, "Clutter Reduction in Multi-Dimensional Data Visualization Using Dimension Reordering," Proc. Symp. Information Visualization (InfoVis), pp. 89-96, 2004. (Pubitemid 40449132)
57. A. Perer and B. Shneiderman, "Integrating Statistics and Visualization for Exploratory Power: From Long-Term Case Studies to Design Guidelines," IEEE Computer Graphics and Applications, vol. 29, no. 3, pp. 39-51, May/June 2009.
58. D. Pinelle and C. Gutwin, "A Review of Groupware Evaluations," Proc. Workshops Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE), pp. 86-91, 2000.
59. D. Pinelle and C. Gutwin, "Groupware Walkthrough: Adding Context to Groupware Usability Evaluation," Proc. Conf. Human Factors in Computing Systems (CHI), pp. 455-462, 2002.
60. D. Pinelle, C. Gutwin, and S. Greenberg, "Task Analysis for Groupware Usability Evaluation: Modeling Shared-Workspace Tasks with the Mechanics of Collaboration," ACM Trans. Human Computer Interaction, vol. 10, no. 4, pp. 281-311, 2003.
61. P. Pirolli and S. Card, "The Sensemaking Process and Leverage Points for Analyst Technology as Identified through Cognitive Task Analysis," Proc. Int'l Conf. Intelligence Analysis, 2005.
62. C. Plaisant, J. Grosjean, and B. Bederson, "Spacetree: Supporting Exploration in Large Node Link Tree, Design Evolution and Empirical Evaluation," Proc. Symp. Information Visualization (InfoVis), pp. 57-64, 2007.
63. C. Plaisant, "The Challenge of Information Visualization Evaluation," Proc. Working Conf. Advanced Visual Interfaces (AVI), pp. 109-116, 2004.
64. Z. Pousman, J. Stasko, and M. Mateas, "Casual Information Visualization: Depictions of Data in Everyday Life," IEEE Trans. Visualization and Computer Graphics, vol. 13, no. 6, pp. 1145-1152, Nov./Dec. 2007. (Pubitemid 350076583)
65. A.J. Pretorius and J.J. van Wijk, "Visual Inspection of Multivariate Graphs," Computer Graphics Forum, vol. 27, no. 3, pp. 967-974, 2008.
66. G. Robertson, M. Czerwinski, K. Larson, D.C. Robbins, D. Thiel, and M. van Dantzich, "Data Mountain: Using Spatial Memory for Document Management," Proc. Symp. User Interface Software and Technology, pp. 153-162, 1998.
67. A. Robinson, "Collaborative Synthesis of Visual Analytic Results," Proc. Symp. Visual Analytics Science and Technology (VAST), pp. 67-74, 2008.
68. J. Sajaniemi and M. Kuittinen, "Visualizing Roles of Variables in Program Animation," Information Visualization, vol. 3, no. 3, pp. 137-153, 2004.
69. P. Saraiya, C. North, and K. Duca, "An Evaluation of Microarray Visualization Tools for Biological Insight," Proc. Symp. Information Visualization (InfoVis), pp. 1-8, 2004. (Pubitemid 40449121)
70. P. Saraiya, C. North, V. Lam, and K.A. Duca, "An Insight-Based Longitudinal Study of Visual Analytics," IEEE Trans. Visualization and Computer Graphics, vol. 12, no. 6, pp. 1511-1522, Nov./Dec. 2006. (Pubitemid 44523094)
71. K. Sedig, S. Rowhani, J. Morey, and H.-N. Liang, "Application of Information Visualization Techniques to the Design of a Mathematical Mindtool: A Usability Study," Information Visualization, vol. 2, no. 3, pp. 142-159, 2003.
72. M. Sedlmair, D. Baur, S. Boring, P. Isenberg, M. Jurmu, and A. Butz, "Requirements for a Mde System to Support Collaborative In-Car Communication Diagnostics," Proc. CSCW Workshop Beyond the Laboratory: Supporting Authentic Collaboration with Multiple Displays, 2008.
73. M. Sedlmair, P. Isenberg, D. Baur, and A. Butz, "Information Visualization Evaluation in Large Companies: Challenges, Experiences and Recommendations," Information Visualization J., vol. 10, no. 3, pp. 248-266, 2011.
74. J. Seo and B. Shneiderman, "Knowledge Discovery in High-Dimensional Data: Case Studies and a User Survey for the Rankby-Feature Framework," IEEE Trans. Visualization and Computer Graphics, vol. 12, no. 3, pp. 311-322, May/June 2006.
75. W. Shadish, T. Cook, and D. Campbell, Experimental and Quasi-Experimental Designs. Houghton Mifflin Company, 2002.
76. B. Shneiderman and C. Plaisant, "Strategies for Evaluating Information Visualization Tools: Multi-Dimensional In-Depth Long-Term Case Studies," Proc. AVI Workshop BEyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV), 2006.
77. B. Shneiderman and C. Plaisant, Designing the User Interface: Strategies for Effective Human-Computer Interaction, fifth ed. Addison-Wesley, 2009.
78. T. Skog, S. Ljungblad, and L.E. Holmquist, "Between Aesthetics and Utility: Designing Ambient Information Visualizations," Proc. Symp. Information Visualization (InfoVis), pp. 233-240, 2003.
79. H. Song, E. Curran, and R. Sterritt, "Multiple Foci Visualisation of Large Hierarchies with Flextree," Information Visualization, vol. 3, no. 1, pp. 19-35, 2004.
80. G. Stahl, Group Cognition. MIT Press, 2006.
81. G. Symon, Qualitative Research Diaries, pp. 94-117. Sage Publications, 1998.
82. Illuminating the Path: The Research and Development Agenda for Visual Analytics, J. Thomas, and K.A. Cook, eds. IEEE CS Press, 2005.
83. M. Tory and T. Möller, "Evaluating Visualizations: Do Expert Reviews Work?" Computer Graphics and Applications, vol. 25, no. 5, pp. 8-11, 2005. (Pubitemid 41555420)
84. M. Tory, D.W. Sprague, F. Wu, W.Y. So, and T. Munzner, "Spatialization Design: Comparing Points and Landscapes," IEEE Trans. Visualization and Computer Graphics, vol. 13, no. 6, pp. 1262-1285, Nov./Dec. 2007.
85. M. Tory and S. Staub-French, "Qualitative Analysis of Visualization: A Building Design Field Study," Proc. CHI Workshop BEyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV), 2008.
86. J.G. Trafton, S.S. Kirschenbaum, T.L. Tsui, R.T. Miyamoto, J.A. Ballas, and P.D. Raymond, "Turning Pictures into Numbers: Extracting and Generating Information from Complex Visualizations," Int'l J. Human-Computer Studies, vol. 53, no. 5, pp. 827-850, 2000. (Pubitemid 32032823)
87. M. Twidale, D. Randall, and R. Bentley, "Situated Evaluation for Cooperative Systems," Proc. Conf. Computer-Supported Cooperative Work (CSCW), pp. 441-452, 1994.
88. Usability.net. Usability.net Methods, http://www.usabilitynet. org/tools/methods.htm, 2009.
89. F. Viégas, E. Perry, E. Howe, and J. Donath, "Artifacts of the Presence Era: Using Information Visualization to Create an Evocative Souvenir," Proc. Symp. Information Visualization (InfoVis), pp. 105-111, 2004. (Pubitemid 40449134)
90. F. Viégas, M. Wattenberg, F. van Ham, J. Kriss, and M. McKeon, "Many Eyes: A Site for Visualization at Internet Scale," IEEE Trans. Visualization and Computer Graphics, vol. 12, no. 5, pp. 1121-1128, Nov./Dec. 2007. (Pubitemid 350076580)
91. C. Weaver, D. Fyfe, A. Robinson, D. Holdsworth, D. Peuquet, and A.M. MacEachren, "Visual Exploration and Analysis of Historic Hotel Visits," Information Visualization, vol. 6, no. 1, pp. 89-103, 2007. (Pubitemid 46598310)
92. W. Willett, J. Heer, and M. Agrawala, "Scented Widgets: Improving Navigation Cues with Embedded Visualizations," IEEE Trans. Visualization and Computer Graphics, vol. 12, no. 5, pp. 1129-1136, Nov./Dec. 2007. (Pubitemid 350076581)
93. K. Wongsuphasawat and B. Shneiderman, "Finding Comparable Temporal Categorical Records: A Similarity Measure with an Interactive Visualization," Proc. Symp. Visual Analytics Science and Technology (VAST), pp. 27-34, 2009.
94. B. Yost and C. North, "The Perceptual Scalability of Visualization," IEEE Trans. Visualization and Computer Graphics, vol. 13, no. 6, pp. 837-844, Sept./Oct. 2007. (Pubitemid 44955611)
95. T. Zuk, L. Schlesier, P. Neumann, M.S. Hancock, and S. Carpendale, "Heuristics for Information Visualization Evaluation," Proc. AVI Workshop BEyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV), pp. 55-60, 2006.