A systematic review on the practice of evaluating visualization

IEEE Transactions on Visualization and Computer Graphics

Volumn 19, Issue 12, 2013, Pages 2818-2827

  Isenberg, Tobias ^a   Isenberg, Petra ^a   Chen, Jian ^b   Sedlmair, Michael ^c   Moller, Torsten ^c  

^a INRIA   (France)

^b UNIVERSITY OF MARYLAND BALTIMORE COUNTY   (United States)

^c UNIVERSITY OF VIENNA   (Austria)

Author keywords

Evaluation; information visualization; scientific visualization; systematic review; validation; visualization

Indexed keywords

ALGORITHM PERFORMANCE; EVALUATION; HISTORIC DEVELOPMENT; INFORMATION VISUALIZATION; REASONING CAPABILITIES; REQUIREMENTS ANALYSIS; SYSTEMATIC REVIEW; VALIDATION;

DATA VISUALIZATION; FLOW VISUALIZATION; INFORMATION ANALYSIS; INFORMATION SYSTEMS; PAPER; TOOLS;

VISUALIZATION;

ALGORITHM; AUTOMATED PATTERN RECOGNITION; COMPUTER ASSISTED DIAGNOSIS; COMPUTER GRAPHICS; COMPUTER INTERFACE; HUMAN; IMAGE ENHANCEMENT; INFORMATION RETRIEVAL; META ANALYSIS; METHODOLOGY; REPRODUCIBILITY; REVIEW; SENSITIVITY AND SPECIFICITY; THREE DIMENSIONAL IMAGING; PROCEDURES;

ALGORITHMS; COMPUTER GRAPHICS; HUMANS; IMAGE ENHANCEMENT; IMAGE INTERPRETATION, COMPUTER-ASSISTED; IMAGING, THREE-DIMENSIONAL; INFORMATION STORAGE AND RETRIEVAL; PATTERN RECOGNITION, AUTOMATED; REPRODUCIBILITY OF RESULTS; SENSITIVITY AND SPECIFICITY; USER-COMPUTER INTERFACE;

EID: 84886683579     PISSN: 10772626     EISSN: None     Source Type: Journal    
DOI: 10.1109/TVCG.2013.126     Document Type: Article

References (74)

1. D. Acevedo, C. D. Jackson, F. Drury, and D. H. Laidlaw. Using visual design experts in critique-based evaluation of 2D vector visualization methods. IEEE Transactions on Visualization and Computer Graphics, 14(4):877-884, July/Aug. 2008. doi10.1109/TVCG.2008.29 (Pubitemid 351780213)
2. N. Ahmed, Z. Zheng, and K. Mueller. Human computation in visualization: Using purpose driven games for robust evaluation of visualization algorithms. IEEE Transactions on Visualization and Computer Graphics (Proc. SciVis), 18(12):2104-2113, Dec. 2012. doi10.1109/TVCG.2012.234
3. K. Andrews. Evaluating information visualisations. In Proc. BELIV, pp. 1:1-1:5. ACM, New York, 2006. doi10.1145/1168149.1168151 (Pubitemid 47117204)
4. I. Babuska and J. T. Oden. Verification and validation in computational engineering and science:Basic concepts. ComputerMethods inAppliedMechanics and Engineering, 193(36-38):4057-4066, 2004. doi10.1016/j.cma.2004.03.002 (Pubitemid 39142612)
5. R. Bausell Barker and Y.-F. Li. Power Analysis for Experimental Research. Cambridge University Press, 2002
6. K. Bødker, F. Kensing, and J. Simonsen. Participatory IT Design: Designing for Business and Workplace Realities. The MIT Press, Cambridge, MA, USA, 2004
7. B. Brown, S. Reeves, and S. Sherwood. Into the wild: Challenges and opportunities for field trial methods. In Proc. CHI, pp. 1657-1666. ACM, New York, 2011. doi10.1145/1978942.1979185
8. S. Bruckner, V. Solt́eszov́a, M. E. Gr̈oller, J. Hladuvka, K. Buhler, J. Y. Yu, and B. J. Dickson. BrainGazer-Visual queries for neurobiology research. IEEE Transactions on Visualization and Computer Graphics (Proc. Vis), 15(6):1497-1504, Nov./Dec. 2009. doi10.1109/TVCG.2009.121
9. S. Carpendale. Evaluating information visualizations. In Information Visualization: Human-Centered Issues and Perspectives, volume 4950 of LNCS, pp. 19-45. Springer, Berlin, 2008. doi10.1007/978-3-540-70956-5 2
10. C. Chen and M. P. Czerwinski. Empirical evaluation of information visualizations: An introduction. International Journal of Human-Computer Studies, 53(5):631-635, Nov. 2000. doi10.1006/ijhc.2000.0421 (Pubitemid 32032814)
11. C. Chen and Y. Yu. Empirical studies of information visualization: A metaanalysis. International Journal of Human-Computer Studies, 53(5):851-866, Nov. 2000. doi10.1006/ijhc.2000.0422 (Pubitemid 32032824)
12. J. Cohen. The earth is round (p10.1037/0003-066X.49.12.997
13. G. Cumming. Understanding The New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis. Routledge, New York, 2012
14. N. Dell, V. Vaidyanathan, I. Medhi, E. Cutrell, and W. Thies. "Yours is better!": Participant response bias in HCI. In Proc. CHI, pp. 1321-1330. ACM, New York, 2012. doi10.1145/2208516.2208589
15. D. L. Donoho. An invitation to reproducible computational research. Biostatistics, 11(3):385-388, July 2010. doi10.1093/biostatistics/kxq028
16. T. Etiene, L. G. Nonato, C. Scheidegger, J. Tierny, T. J. Peters, V. Pascucci, R. M. Kirby, and C. T. Silva. Topology verification for isosurface extraction. IEEE Transactions on Visualization and Computer Graphics, 18(6):952-965, June 2012. doi10.1109/TVCG.2011.109
17. T. Etiene, C. Scheidegger, L. G. Nonato, R. M. Kirby, and C. T. Silva. Verifiable visualization for isosurface extraction. IEEE Transactions on Visualization and Computer Graphics (Proc. Vis), 15(6):1227-1234, Nov./Dec. 2009. doi10.1109/TVCG.2009.194
18. A. Field and G. Hole. How to Design and Report Experiments. Sage Publications, Ltd., London, 2003
19. C. Forsell. A guide to scientific evaluation in information visualization. In Proc. Information Visualization, pp. 162-169. IEEE Computer Society, Los Alamitos, 2010. doi10.1109/IV.2010.33
20. M. Gleicher. Why ask why Considering motivation in visualization evaluation. In Proc. BELIV, pp. 10:1-10:3. ACM, New York, 2012. doi 10.1145/2442576.2442586
21. A. Globus and S. Uselton. Evaluation of visualization software. Technical Report NAS-95-005, NASA Advanced Supercomputing Div., Feb. 1995
22. B. N. Green, C. D. Johnson, and A. Adams. Writing narrative literature reviews for peer-reviewed journals: Secrets of the trade. Journal of Chiropractic Medicine, 5(3):101-117, Fall 2006. doi10.1016/S0899-3467(07)60142-6
23. S. Greenberg and B. Buxton. Usability evaluation considered harmful (some of the time). In Proc. CHI, pp. 111-120. ACM, New York, 2008. doi10.1145/1357054.1357074
24. M. J. Intons-Peterson. Imagery paradigms: How vulnerable are they to experimenters' expectations Journal of Experimental Psychology: Human Perception and Performance, 9(3):394-412, June 1983. doi10. 1037/0096-1523.9.3.394
25. P. Isenberg, N. Elmqvist, J. Scholtz, D. Cernea, K.-L. Ma, and H. Hagen. Collaborative visualization: Definition, challenges, and research agenda. Information Visualization, 10(4):310-326, Oct. 2011. doi: 10 .1177/1473871611412817
26. B. Jackson, D. Coffey, L. Thorson, D. Schroeder, A. M. Ellingson, D. J. Nuckley, and D. F. Keefe. Toward mixed method evaluations of scientific visualizations and design process as an evaluation tool. In Proc. BELIV, pp. 4:1-4:6. ACM, New York, 2012. doi:10.1145/2442576.2442580
27. C. D. Jackson, D. Acevedo, D. H. Laidlaw, F. Drury, E. Vote, and D. Keefe. Designer-critiqued comparison of 2D vector visualization methods: A pilot study. In ACM SIGGRAPH Sketches & Applications. ACM, New York, 2003. doi:10.1145/965400.965505
28. C. Johnson. Top scientific visualization research problems. IEEE Computer Graphics and Applications, 24(4):13-17, July/Aug. 2004. doi:10. 1109/MCG.2004.20
29. D. F. Keefe, D. Acevedo, J. Miles, F. Drury, S. M. Swartz, and D. H. Laidlaw. Scientific sketching for collaborative VR visualization design. IEEE Transactions on Visualization and Computer Graphics, 14(4):835-847, July 2008. doi:10.1109/TVCG.2008.31 (Pubitemid 351780210)
30. D. F. Keefe, D. B. Karelitz, E. L. Vote, and D. H. Laidlaw. Artistic collaboration in designing VR visualizations. IEEE Computer Graphics and Applications, 25(2):18-23, Mar./Apr. 2005. doi:10.1109/MCG.2005.34 (Pubitemid 40556054)
31. R. M. Kirby and C. T. Silva. The need for verifiable visualization. IEEE Computer Graphics and Applications, 28(5):78-83, Sept. 2008. doi:10. 1109/MCG.2008.103
32. R. B. Kline. What's wrong with statistical tests-and where we go from here. In Beyond Significance Testing: Reforming Data Analysis Methods in Behavioral Research, chap. 3, pp. 61-91. APA, Washington, DC, 2004. doi:10.1037/10693-003
33. P. Kok, M. Baiker, E. A. Hendriks, F. H. Post, J. Dijkstra, C. W. G. M. L̈owik, B. P. F. Lelieveldt, and C. P. Botha. Articulated planar reformation for change visualization in small animal imaging. IEEE Transactions on Visualization and Computer Graphics (Proc. Vis), 16(6):1396-1404, Nov./Dec. 2010. doi:10.1109/TVCG.2010.134
34. R. Kosara. Visualization criticism-the missing link between information visualization and art. In Proc. Information Visualization, pp. 631-636. IEEE Computer Society, Los Alamitos, 2007. doi:10.1109/IV.2007.130 (Pubitemid 47591802)
35. R. Kosara, F. Drury, L. E. Holmquist, and D. H. Laidlaw. Visualization criticism. IEEE Computer Graphics and Applications, 28(3):13-15, May/June 2008. doi:10.1109/MCG.2008.63 (Pubitemid 351712169)
36. R. Kosara, C. G. Healey, V. Interrante, D. H. Laidlaw, and C. Ware. User studies: Why, how, and when IEEE Computer Graphics and Applications, 23(4):20-25, July/Aug. 2003. doi:10.1109/MCG.2003.1210860
37. K. Krippendorff. Content Analysis: An Introduction to Its Methodology. SAGE Publications, Inc., Thousand Oaks, CA, USA, 3rd ed., 2013
38. H. Lam, E. Bertini, P. Isenberg, C. Plaisant, and S. Carpendale. Empirical studies in information visualization: Seven scenarios. IEEE Transactions on Visualization and Computer Graphics, 18(9):1520-1536, Sept. 2012. doi:10.1109/TVCG.2011.279
39. P. Lindstrom and M. Isenburg. Fast and efficient compression of floatingpoint data. IEEE Transactions on Visualization and Computer Graphics, 12(5):1245-1250, Sept./Oct. 2006. doi:10.1109/TVCG.2006.143 (Pubitemid 44955661)
40. M. A. Livingston, J. W. Decker, and Z. Ai. Evaluation of multivariate visualization on a multivariate task. IEEE Transactions on Visualization and Computer Graphics (Proc. SciVis), 18(12):2114-2121, Dec. 2012. doi:10.1109/TVCG.2012.223
41. B. Lorensen. On the death of visualization. In Position Papers of the NIH/NSF Fall 2004 Workshop on Visualization Research Challenges, 2004
42. C. Lundstr̈om, T. Rydell, C. Forsell, A. Persson, and A. Ynnerman. Multitouch table system for medical visualization: Application to orthopedic surgery planning. IEEE Transactions on Visualization and Computer Graphics (Proc. VIS), 17(12):1775-1784, Dec. 2011. doi:10.1109/TVCG.2011.224
43. J. E. McGrath. Methodology matters: Doing research in the behavioral and social sciences. In R. M. Baecker, J. Grudin,W. Buxton, and S. Greenberg, eds., Readings in Human-Computer Interaction: Towards the Year 2000, pp. 152-169. Morgan Kaufmann, 2nd ed., 1995
44. S. B. Merriam. Qualitative Research: A Guide to Design and Implementation. Jossey-Bass, San Francisco, 2nd ed., 2009
45. M. Meyer, T. Munzner, and H. Pfister. MizBee: A multiscale synteny browser. IEEE Transactions on Visualization and Computer Graphics (Proc. InfoVis), 15(6):897-904, Nov./Dec. 2009. doi:10.1109/TVCG.2009.167
46. E. Morse, M. Lewis, and K. Olsen. Evaluating visualizations: Using a taxonomic guide. International Journal of Human-Computer Studies, 53(5):637-662, Nov. 2000. doi:10.1006/ijhc.2000.0412 (Pubitemid 32032815)
47. T. Munzner. A nested model for visualization design and validation. IEEE Transactions on Visualization and Computer Graphics (Proc. InfoVis), 15(6):921-928, Nov./Dec. 2009. doi:10.1109/TVCG.2009.111
48. B. Nelson, R. M. Kirby, and R. Haimes. GPU-based interactive cut-surface extraction from high-order finite element fields. IEEE Transactions on Visualization and Computer Graphics (Proc. Vis), 17(12):1803-1811, Dec. 2011. doi:10.1109/TVCG.2011.206
49. J. Nielsen and R. Molich. Heuristic evaluation of user interfaces. In Proc. CHI, pp. 249-256. ACM, New York, 1990. doi:10.1145/97243.97281
50. C. North. Toward measuring visualization insight. IEEE Computer Graphics and Applications, 26(3):6-9, May/June 2006. doi:10.1109/MCG.2006.70 (Pubitemid 46404978)
51. M. T. Orne. Demand characteristics and the concept of quasi-controls. In R. Rosenthal and R. L. Rosnow, eds., Artifacts in Behavioral Research: Robert Rosenthal and Ralph L. Rosnow's Classic Books, chap. 5, pp. 110-137. Oxford University Press, New York, 2009
52. A. Perer and B. Shneiderman. Integrating statistics and visualization for exploratory power: From long-term case studies to design guidelines. IEEE Computer Graphics and Applications, 29(3):39-51, May/June 2009. doi:10.1109/MCG.2009.44
53. C. Plaisant. The challenge of information visualization evaluation. In Proc. AVI, pp. 109-116. ACM, New York, 2004. doi:10.1145/989863.989880
54. K. Popper. The Logic of Scientific Discovery. Routledge Classics, Taylor & Francis, London, 2010
55. M. Schmettow. Sample size in usability studies. Communications of the ACM, 55(4):64-70, Apr. 2012. doi:10.1145/2133806.2133824
56. T. Schultz and G. L. Kindlmann. Superquadric glyphs for symmetric secondorder tensors. IEEE Transactions on Visualization and Computer Graphics (Proc. Vis), 16(6):1595-1604, Nov./Dec. 2010. doi:10.1109/TVCG.2010.199
57. T. Schultz and H.-P. Seidel. Estimating crossing fibers: A tensor decomposition approach. IEEE Transactions on Visualization and Computer Graphics (Proc. Vis), 14(6):1635-1642, Nov./Dec. 2008. doi:10.1109/TVCG.2008.128
58. M. Sedlmair, P. Isenberg, D. Baur, and A. Butz. Information visualization evaluation in large companies: Challenges, experiences and recommendations. Information Visualization, 10(3):248-266, July 2011. doi: 10 .1177/1473871611413099
59. M. Sedlmair, M. Meyer, and T. Munzner. Design study methodology: Reflections from the trenches and the stacks. IEEE Transactions on Visualization and Computer Graphics (Proc. InfoVis), 18(12):2431-2440, Dec. 2012. doi:10.1109/TVCG.2012.213
60. J. Seo and B. Shneiderman. Knowledge discovery in high-dimensional data: Case studies and a user survey for the rank-by-feature framework. IEEE Transactions on Visualization and Computer Graphics, 12(3):311-322, May/June 2006. doi:10.1109/TVCG.2006.50
61. H. Sharp, Y. Rogers, and J. Preece. Interaction Design: Beyond Human-Computer Interaction. John Wiley & Sons, Chichester, UK, 2nd ed., 2007
62. B. Shneiderman and C. Plaisant. Strategies for evaluating information visualization tools: Multi-dimensional in-depth long-term case studies. In Proc. BELIV, pp. 8:1-8:7. ACM, New York, 2006. doi:10.1145/1168149.1168158 (Pubitemid 47117215)
63. B. Shneiderman, C. Plaisant, M. Cohen, and S. Jacobs. Designing the User Interface: Strategies for Effective Human-Computer Interaction. Pearson/Prentice Hall, 5th ed., 2010
64. J. P. Simmons, L. D. Nelson, and U. Simonsohn. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11):1359-1366, Nov. 2011. doi:10.1177/0956797611417632
65. M. Tory and T. M̈oller. Human factors in visualization research. IEEE Transactions on Visualization and Computer Graphics, 10(1):72-84, Jan./Feb. 2004. doi:10.1109/TVCG.2004.1260759
66. M. Tory and T. M̈oller. Evaluating visualizations: Do expert reviews work IEEE Computer Graphics and Applications, 25(5):8-11, Sept./Oct. 2005. doi:10.1109/MCG.2005.102 (Pubitemid 41555420)
67. S. P. Uselton, G. Dorn, C. Farhat, M. W. Vannier, K. Esbensen, and A. Globus. Validation, verification and evaluation. In Proc. Visualization, pp. 414-418. IEEE Computer Society, Los Alamitos, 1994. doi:10.1109/VISUAL.1994. 346285
68. J. van Wijk, T. Isenberg, J. B. Roerdink, A. C. Telea, and M. Westenberg. Evaluation. In D. Keim, J. Kohlhammer, G. Ellis, and F. Mansmann, eds., Mastering the Information Age: Solving Problems with Visual Analytics, chap. 8, pp. 131-144. Eurographics Assoc., Goslar, Germany, 2010
69. J. J. van Wijk. The value of visualization. In Proc. Visualization, pp. 79-86. IEEE Computer Society, Los Alamitos, 2005. doi:10.1109/VISUAL. 2005.1532781
70. J. J. vanWijk. Views on visualization. IEEE Transactions on Visualization and Computer Graphics, 12(4):421-432, July/Aug. 2006. doi:10.1109/TVCG.2006.80
71. G. R. VandenBos, ed. Publication Manual of the American Psychological Association. APA, Washington, DC, 6th ed., 2009
72. J.Waser, R. Fuchs, H. Ribicíc, B. Schindler, G. Bl̈oschl, and M. E. Gr̈oller. World lines. IEEE Transactions on Visualization and Computer Graphics (Proc. Vis), 16(6):1458-1467, Nov./Dec. 2010. doi:10.1109/TVCG.2010. 223
73. A.Wiebel, F. Vos, D. Foerster, and H.-C. Hege.WYSIWYP: What you see is what you pick.IEEE Transactions on Visualization and Computer Graphics (Proc. SciVis), 18(12):2236-2244, Dec. 2012. doi:10.1109/TVCG.2012.292
74. J. Wood, D. Badawood, J. Dykes, and A. Slingsby. BallotMaps: Detecting name bias in alphabetically ordered ballot papers. IEEE Transactions on Visualization and Computer Graphics (Proc. InfoVis), 17(12):2384-2391, Dec. 2011. doi:10.1109/TVCG.2011.174