Evaluation of visual analytics environments: The road to the Visual Analytics Science and Technology challenge evaluation methodology

Information Visualization

Volumn 13, Issue 4, 2014, Pages 326-335

  Scholtz, Jean ^a   Plaisant, Catherine ^b   Whiting, Mark ^a   Grinstein, Georges ^c  

^a PACIFIC NORTHWEST NATIONAL LABORATORY   (United States)

^b UNIVERSITY OF MARYLAND   (United States)

^c UNIVERSITY OF MASSACHUSETTS LOWELL   (United States)

Author keywords

Community evaluations; Utility evaluations; Visual analytics environments

Indexed keywords

GRAPHICAL USER INTERFACES;

ALGORITHM CORRECTNESS; COMMUNITY EVALUATIONS; END USERS; EVALUATION METHODOLOGIES; SCIENCE AND TECHNOLOGY; UTILITY EVALUATIONS; VISUAL ANALYTICS;

VISUALIZATION;

EID: 84928007456     PISSN: 14738716     EISSN: 14738724     Source Type: Journal    
DOI: 10.1177/1473871613490290     Document Type: Article

References (29)

1. Thomas J and Cook K (eds) Illuminating the path: The research and development agenda for visual analytics. Los Alamitos, CA: IEEE CS Press, 2005.
2. Lam H, Bertini E, Isenberg P, et al. Empirical studies in information visualization: Seven scenarios. IEEE T Vis Comput Gr 2012; 18(9): 1520-1536.
3. Carpendale S. Evaluating information visualizations. In: Kerren A, Stasko JT, Fekete J-D, et al. (eds) Information visualization: Human-centered issues and perspectives. Berlin/Heidelberg: Springer LNCS, 2007, pp. 19-45.
4. Plaisant C. The challenge of information visualization evaluation. In: Proceedings of the working conference on advanced visual interfaces, Gallipoli, Italy, 25-28 May 2004, pp. 109-116. New York: ACM.
5. Chinchor N and Hirschman L. Evaluating message understanding systems: An analysis of the third message understanding conference (MUC-3). Comput Linguist 1993; 19(3): 409-449.
6. Voorhees E. TREC: continuing information retrieval's tradition of experimentation. Commun ACM 2007; 50(11): 51-54.
7. Face recognition grand challenge, http://www.nist.gov/itl/iad/ig/frgc.cfm (accessed 8 November 2012).
8. Beyer H and Holtzblatt K. Contextual design: defining customer-centered systems. Burlington, MA: Morgan Kaufmann, 1997.
9. Shneiderman B and Plaisant C. Designing the user interface: Strategies for effective human-computer interaction. 5th ed. Boston, MA: Addison-Wesley Publishing Co., 2009.
10. Plaisant C, Fekete JD and Grinstein G. Promoting insight based evaluation of visualizations: From contest to benchmark repository. IEEE T Vis Comput Gr 2008; 14(1): 120-134.
11. NSF International Science & Engineering Visualization Challenge, http://www.nsf.gov/news/special-reports/scivis/challenge.jsp (accessed November 2012).
12. Data Viz Challenge, http://datavizchallenge.org/aboutchallenge (2011, accessed 8 November 2012).
13. Saraiya P, North C, Lam V, et al. An insight-based longitudinal study of visual analytics. IEEE T Vis Comput Gr 2006; 12(6): 1511-1522.
14. Cowley PJ, Nowell LT and Scholtz J. Glass box: An instrumented infrastructure for supporting human interaction with information. In: Proceedings of the 38th IEEE annual Hawaii international conference, Waikoloa, Hawaii, 3-6 January 2005, pp. 296-304. IEEE.
15. Redish J. Expanding usability testing to evaluate complex systems. J Usability Stud 2007; 2(3): 102-111.
16. Costello L, Grinstein G, Plaisant C, et al. Advancing user-centered evaluation of visual analytic environments through contests. Inform Visual 2009; 8: 230-238.
17. Netflix Prize, http://www.netflixprize.com (accessed 8 November 2012).
18. ODNI Directive 203, http://www.fas.org/irp/dni/icd/icd-203.pdf (accessed 20 March 2010).
19. Heuer R and Pherson R. Structured analytic techniques for intelligence analysis. Washington, DC: CQ Press, 2010.
20. Jones MD. The thinker's toolkit: 14 powerful techniques for problem solving. New York: Three Rivers Press, 1998.
21. Apple guidelines, http://developer.apple.com/library/ios/#DOCUMENTATION/UserExperience/Conceptual/MobileHIG/Introduction/Introduction.html(accessed November 2012).
22. National Cancer Institute. Research-based web design and usability guidelines. Department of Health and Human Services, National Institutes of Health, http://guidelines.usability.gov/(2006, accessed 8 November 2012).
23. Scholtz J. Developing guidelines for assessing visual analytics environments. Inform Visual 2011; 10(3): 212-231.
24. Carr D. Guidelines for designing information visualization applications. In: Proceedings of the 1999 Ericsson conference on usability engineering, ECUE '99, Stockholm, Sweden, December 1999.
25. Scholtz J. Developing qualitative metrics for visual analytic environments. In: The Proceedings of BELIV '10: BEyond time and errors: novel evaLuation methods for Information Visualization, Atlanta, GA, USA, 10-11 April 2007, pp. 1-7. New New York: ACM.
26. Scholtz J, Whiting M, Plaisant C, et al. A reflection on seven years of the VAST challenge. In: Proceedings of BELIV '12: BEyond time and errors: novel evaLuation methods for Information Visualization, a workshop of the IEEE VisWeek 2012 conference, Seattle, WA, USA, 14-15 October 2012, pp. 1-8. New York: ACM.
27. Munzner T. A nested model for visualization design and validation. IEEE T Vis Comput Gr 2009; 15(6): 921-928.
28. Kang Y, Görg C and Stasko J. How can visual analytics assist investigative analysis? Design implications from an evaluation. IEEE T Vis Comput Gr 2011; 17(5): 570-583.
29. Whiting M, North C, Endert A, et al. VAST contest dataset use in education. In: Visual Analytics Science and Technology, 2009. VAST 2009, IEEE Symposium on. Atlantic City, NJ, USA, 11-16 Oct, pp. 115-122. IEEE.