Promoting insight-based evaluation of visualizations: From contest to benchmark repository

IEEE Transactions on Visualization and Computer Graphics

Volumn 14, Issue 1, 2008, Pages 120-134

  Plaisant, Catherine ^a   Fekete, Jean Daniel ^b   Grinstein, Georges ^c  

^a UNIVERSITY OF MARYLAND   (United States)

^b INRIA   (France)

^c UNIVERSITY OF MASSACHUSETTS LOWELL   (United States)

Author keywords

Benchmark; Competition; Contest; Information; Measure; Metrics; Repository; Visualization

Indexed keywords

INFORMATION VISUALIZATION; REPOSITORY;

BENCHMARKING; COMPETITION; DATA STRUCTURES; INFORMATION MANAGEMENT;

DATA VISUALIZATION;

ALGORITHM; ARTICLE; COMPUTER ASSISTED DIAGNOSIS; COMPUTER GRAPHICS; COMPUTER INTERFACE; COMPUTER PROGRAM; EVALUATION; FACTUAL DATABASE; METHODOLOGY; QUALITY CONTROL;

ALGORITHMS; BENCHMARKING; COMPUTER GRAPHICS; DATABASES, FACTUAL; EVALUATION STUDIES AS TOPIC; IMAGE INTERPRETATION, COMPUTER-ASSISTED; SOFTWARE; SOFTWARE VALIDATION; USER-COMPUTER INTERFACE;

EID: 36349035237     PISSN: 10772626     EISSN: None     Source Type: Journal    
DOI: 10.1109/TVCG.2007.70412     Document Type: Article

References (70)

1. A. Ahmed, T. Dwyer, C. Murray, L. Song, and Y.X. Wu, "WilmaScope, " Poster Compendium of IEEE Information Visualization, 2004.
2. D. Auber, M. Delest, J.-P. Domenger, P. Ferraro, and R. Strandh, "EVAT-Environment for Visualization and Analysis of Trees," Poster Compendium of IEEE Information Visualization, 2003.
3. Proc. Workshop Beyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV '06), http://www.dis.uniroma1.it/~beliv06/, 2007.
4. Information Visualization Benchmark Repository, www.cs.umd.edu/hcil/ InfovisRepository, 2007.
5. InfoVis Cyberlnfrastructure, http://iv.slis.indiana.edu, 2007.
6. K. Börner, L. Dall'Asta, W. Ke, and A. Vespignani, "Studying the Emerging Global Brain: Analyzing and Visualizing the Impact of Co-Authorship Teams," Complexity, special issue on understanding complex systems, vol. 10, no. 4, pp. 58-67, 2005.
7. Proc. Sixth Int'l Conf. Critical Assessment of Microarray Data Analysis (CAMDA '06), http://www.camda.duke.edu/camda06, 2006.
8. Council of European Social Science Data Archives (CESSDA), http://www.nsd.uib.no/cessda, 2007.
9. "Introduction to the Special Issue on Empirical Evaluation of Information Visualizations," Int'l J. Human-Computer Studies, C. Chen and M. Czerwinski, eds., vol. 53, no. 5, pp. 631-635, 2000.
10. C. Chen, "Citation and Co-Citation Perspective," Poster Compendium of IEEE Information Visualization, 2004.
11. J. Chen, D. Guo, and A.M. MacEachren, "Space-Time-Attribute Analysis and Visualization of US Company Data," Poster Compendium of IEEE Information Visualization, 2005.
12. J. Chen, A.M. MacEachren, and D. Guo, "Visual Inquiry Toolkit-An Integrated Approach for Exploring and Interpreting Space-Time, Multivariate Patterns," Proc. AutoCarto '06, CD-ROM, June 2006.
13. J. Chen, A.M. MacEachren, and D. Guo, "Supporting the Process of Exploring and Interpreting Space-Time, Multivariate Patterns: The Visual Inquiry Toolkit," Cartography and Geographic Information Science, to be published.
14. N. Chinchor and L. Hirschman, "Evaluating Message Under-standing Systems: An Analysis of the Third Message Under-standing Conference (MUC-3)," Computational Linguistics, vol. 19, no. 3, pp. 409-449, 1993.
15. P. Cowley, L. Nowell, and J. Scholtz, "Glassbox: An Instrumented Infrastructure for Supporting Human-Interaction with Information," Proc. 38th Ann. Hawaii Int'l Conf. System Sciences (HICSS '05), p. 296.3, 2005.
16. M. Delest, T. Munzner, D. Auber, and J.-P. Domenger, "Tulip," Poster Compendium of IEEE Information Visualization, 2004.
17. S. Faisal, P. Cairns, and A. Blandford, "Subjective Information Visualisations," Proc. Workshop Combining Visualisation and Interaction to Facilitate Scientific Exploration and Discovery, 2006.
18. J.-D. Fekete, "The InfoVis Toolkit," Proc. IEEE Symp. Information Visualization (INFOVIS '04), pp. 167-174, 2004.
19. J. Genre, P. Ginsparg, and J. Kleinburg, "Overview of the 2003 KDD Cup," Proc. SIGKDD Explorations, vol. 5, no. 2, pp. 149-151, 2004.
20. V. Gonzales and A. Kobsa, "Benefits of Information Visualization for Administrative Data Analysts," Proc. Seventh Int'l Conf. Information Visualization (IV '03), pp. 331-337, 2003.
21. G. Grinstein, P. Hoffman, R. Pickett, and S. Laskowski, "Benchmark Development for the Evaluation of Visualization for Data Mining," Information Visualization in Data Mining and Knowledge Discovery, U. Fayyad, G. Grinstein, and A. Wierse, eds., pp. 129-176, Morgan Kaufmann, 2002.
22. G. Grinstein, T. O'Connell, S. Laskowski, C. Plaisant, J. Scholtz, and M. Whiting, "VAST 2006 Contest: A Tale of Alderwood," Proc. First IEEE Int'l Symp. Visual Analytics Science and Technology (VAST '06), pp. 215-216, 2006.
23. D. Guo, J. Chen, A.M. MacEachren, and K. Liao, "A Visual Inquiry System for Space-Time and Multivariate Patterns (VIS-STAMP)," IEEE Trans. Visualization and Computer Graphics, vol. 12, no. 6, pp. 1461-1474, Nov./Dec. 2006.
24. F. van Ham, "Technische Universiteit Eindhoven Contest Submission," Poster Compendium of IEEE Information Visualization, 2004.
25. H. Hofmann, H. Wickham, D. Cook, J. Sun, and C. Röttger, "Boom and Bust of Technology Companies at the Turn of the 21st Century," Poster Compendium of IEEE Information Visualization, 2005.
26. J.Y. Hong, J. D'Andries, M. Richman, and M. Westfall, "Zoomology: Comparing Two Large Hierarchical Trees," Poster Compendium of IEEE Information Visualization, 2003.
27. B. Hoser, M. Blume, J. Schröder, and M. Franke, "CPS- Company Positioning System: Visualizing the Economic Environment," Poster Compendium of IEEE Information Visualization, 2005.
28. H. Tzu-Wei, L.I. Farabaugh, D. McColgin, and K. Stamper, "MonkEllipse," Poster Compendium of IEEE Information Visualization, 2004.
29. J.-D. Fekete, G. Grinstein, and C. Plaisant, 10th IEEE Symp. Information Visualization (InfoVis '04) Contest, www.cs.umd.edu/hcil/ iv04contest, 2004.
30. G. Grinstein, U. Cvek, M. Derthick, and M. Trutschl, Proc. 11th IEEE Symp. Information Visualization (InfoVis '05) Contest, Technology Data in the US, http://ivpr.cs.uml.edu/infovis05, 2005.
31. http://sun.cs.lsus.edu/iv06/, 2006.
32. D. Keim, C. Panse, M. Sips, J. Schneidewind, and H. Barro Univ. of Konstanz Contest Submission, Poster Compendium of IEEE Information Visualization, 2004.
33. D. Keim, R.D. Bergeron, and R. Pickett, "Test Datasets for Evaluating Data Visualization Techniques," Perceptual Issues in Visualization, G. Grinstein and H. Levkowitz, eds., pp. 9-22, Springer, 1995.
34. A. Komlodi, A. Sears, and E. Stanziola, "InformationVisualization Evaluation Rev.," ISRC Technical Report UMBC-ISRC-2004-1, Dept. of Information Systems, Univ. of Maryland, Baltimore County, http://www.research. umbc.edu/~komlodi/IV_eval, 2004.
35. L. Bongshin, M. Czerwinski, G. Robertson, and B.B. Bederson, "PaperLens," Poster Compendium of IEEE Information Visualization, 2004.
36. L. Xia, J. Buzydlowski, and H.D. White, "Associative Information Visualizer," Poster Compendium of IEEE Information Visualization, 2004.
37. S. Laskowski and C. Plaisant, "Evaluation Methodologies for Visual Analytics," Illuminating the Path - the Research and Development Agenda for Visual Analytics, J. Thomas and K. Cook, eds., pp. 150-157, IEEE Press, 2005.
38. J. Mackinlay, "Automating the Design of Graphical Presentations of Relational Information," ACM Trans. Graphics, vol. 5, no. 2, pp. 110-141, 1986.
39. R. Minghim, F. Paulovich, and A. Lopes, "Content-Based Text Mapping Using Multi-Dimensional Projections for Exploration of Document Collections," Proc. Conf. Visualization and Data Analysis (VDA '06), 2006.
40. Univ. of California, Irvine, Machine Learning Repository, http://www.ics.uci.edu/~mlearn/MLRepository.html, 2007.
41. D.R. Morse, N. Ytow, D.M. Roberts, and A. Sato, "Comparison of Multiple Taxonomie Hierarchies Using TaxoNote," Poster Compendium of IEEE Information Visualization, 2003.
42. K. Mullet, C. Fry, and D. Schiano, "On Your Marks, Get Set, Browsel (The Great CHI '97 Browse Off)," Panel Description in ACM Conf. Human Factors in Computing Systems (CHI '97) Extended Abstracts, pp. 113-114, 1997.
43. J. Slack, T. Munzner, and F. Guimbretiere, "Treejuxtaposer: InfoVis03 Contest Entry," Poster Compendium of IEEE Information Visualization, 2003.
44. T. Munzner, F. Guimbretière, S. Tasiran, L. Zhang, and Y. Zhou, "Treejuxtaposer: Scalable Tree Comparison Using Focus+Context with Guaranteed Visibility," ACM Trans. Graphics - Proc. ACM SIGGRAPH '03, pp. 453-462, 2003.
45. North American Industry Classification System, www.census.gov/epcd/www/ naics.html, 2007.
46. New York Times-Election 2004, http://www.nytimes.com/packages/html/ politics/2004_ELECTIONGUIDE_GRAPHIC/?oref=login, (retrieved June 2005).
47. D. Pallett, J. Garofolo, and J. Fiscus, "Measurement in Support of Research Accomplishments," Comm. ACM, vol. 43, no. 2, pp. 75-79, 2000.
48. C. Plaisant, J. Grosjean, and B.B. Bederson, "SpaceTree: Supporting Exploration in Large Node-Link Tree: Design Evolution and Empirical Evaluation," Proc. IEEE Symp. Information Visualization (InfoVis '02), pp. 57-64, 2002.
49. C. Plaisant, "The Challenge of Information Visualization Evaluation," Proc. Working Conf. Advanced Visual Interfaces (AVI '04), pp. 109-116, 2004.
50. P. Saraiya, C. North, and K. Duca, "An Evaluation of Microarray Visualization Tools for Biological Insight," Proc. IEEE Symp. Information Visualization (InfoVis '04), pp. 1-8, 2004.
51. J. Scholtz, L. Arnstein, M. Kim, T. Kindberg, and S. Consolvo, User-Centered Evaluations of Ubicomp Applications, Technical Report IRS-TR-02-006, Intel Corp., May 2002.
52. J. Scholtz and M.P. Steves, "A Framework for Evaluating Collaborative Systems in the Real World," Proc. 38th Hawaii Int'l Conf System Sciences (HICSS '05), 2005.
53. N. Sheth, K. Börner, J. Baumgartner, K. Mane, and E. Wernert, "Treemap, Radial Tree, and 3D Tree Visualizations," Poster Compendium of IEEE Information Visualization, 2003.
54. B. Shneiderman and C. Plaisant, "Strategies for Evaluating Information Visualization Tools: Multidimensional In-Depth Long-Term Case Studies," Proc. Workshop BEyond Time and Errors: Novel EvaLuation Methods for Information Visualization (BELIV '06), pp. 38-43, 2006.
55. Smart Money Map of the Market, www.smartmoney.com, (retrieved June 2005).
56. M. Spenke and C. Beilken, "Analysing Formula One Racing Results with an Interactive Data Mining and Visualization Tool," Data Mining II, N. Ebecken, ed., pp. 455-464, 2000.
57. S.T. Teoh and K.-L. Ma One-for-All - Univ. of California, Davis, Contest Submission, Poster Compendium of IEEE Information Visualization, 2004.
58. J. Trafton, T. Tsui, R. Miyamoto, J. Ballas, and P. Raymond, "Turning Pictures into Numbers: Extracting and Generating Information from Complex Visualizations," Int'l J. Human Computer Studies, vol. 53, no. 5, pp. 827-850, 2000.
59. J. Thomas and K. Cook, Illuminating the Path: The Research and Development Agenda for Visual Analytics, CS Press, http://nvac.pnl.gov/ agenda.stm, 2005.
60. Proc. Text REtrieval Conf. (TREC), http://trec.nist.gov, 2007.
61. J. Tyman, G.P. Gruetzmacher, and J. Stasko, "InfoVisExplorer," Poster Compendium of IEEE Information Visualization, 2004.
62. A.R. Unwin, M. Theus, and H. Hofmann, Graphics of Large Datasets. Springer, 2006.
63. www.cs.umd.edu/hcil/VASTcontest06, 2006.
64. E. Voorhees and D. Harman, "Overview of the Sixth Text Retrieval Conf. (TREC-6)," Information Processing and Management, vol. 36, pp. 3-35, 2000.
65. W. Ke, K. Borner, and L. Viswanath, "Indiana Univ. Contest Submission," Poster Compendium of IEEE Information Visualization, 2004.
66. M.A. Whiting, W. Cowley, J. Haack, D. Love, S. Tratz, C. Varley, and K. Wiessner, "Threat Stream Data Generator: Creating the Known Unknowns for Test and Evaluation of Visual Analytics Tools," Proc. Workshop BEyond Time and Errors: Novel EvaLuation Methods for Information Visualization (BELIV '06), pp. 1-3, 2006.
67. W.P. Chung, B. Hetzler, C. Posse, M. Whiting, S. Havre, N. Cramer, A. Shah, M. Singhal, A. Turner, and J. Thomas, "INSPIRE," Poster Compendium of IEEE Information Visualization, 2004.
68. Z. Annerose, S. Potapov, M. Theus, and A. Unwin, "Analyzing Company Data with Interactive Statistical Graphics," Poster Compendium of IEEE Information Visualization, 2005.
69. C. North, "Visualization Viewpoints: Toward Measuring Visualization Insight," IEEE Computer Graphics and Applications, vol. 26, no. 3, pp. 6-9, May/June 2006.
70. J.-D. Fekete and C. Plaisant InfoVis 2003 Contest, www.cs.umd.edu/hcil/ iv03contest, 2003.