Crowdsourcing graphical perception: Using mechanical Turk to assess visualization design

Conference on Human Factors in Computing Systems - Proceedings

Volumn 1, Issue , 2010, Pages 203-212

  Heer, Jeffrey ^a   Bostock, Michael ^a  

^a Stanford University   (United States)

Author keywords

crowdsourcing; evaluation; experimentation; graphical perception; information visualization; mechanical turk; user study

Indexed keywords

INFORMATION VISUALIZATION; LARGE DESIGNS; LOW COSTS; LUMINANCE CONTRAST; PERCEPTION EXPERIMENT; PERFORMANCE DATA; RECTANGULAR AREA; SPATIAL ENCODING; TREE-MAPS; USER STUDY;

DESIGN; HUMAN ENGINEERING; INFORMATION ANALYSIS; INFORMATION SYSTEMS; VISUALIZATION;

EXPERIMENTS;

EID: 77953992027     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1753326.1753357     Document Type: Conference Paper

References (37)

1. V. Beattie and M. J. Jones. The impact of graph slope on rate of change judgements in corporate reports. ABACUS, 38:177-199, 2002.
2. J. Bertin. Sémiologie Graphique. Gauthier-Villars, 1967.
3. D. M. Bruls, C. Huizing, and J. J. van Wijk. Squarified treemaps. In Data Visualization 2000, Eurographics/IEEE TVCG Visualization Symp., pp. 33-42, 2000.
4. C. Cheshire. Selective incentives and generalized information exchange. Social Psychology Quarterly, 70, 2007.
5. W. S. Cleveland. Visualizing Data. Hobart Press, 1993.
6. W. S. Cleveland, P. Diaconis, and R. McGill. Variables on scatterplots look more highly correlated when the scales are increased. Science, 216:1138-1141, Jun 1982.
7. W. S. Cleveland and R. McGill. Graphical perception: Theory, experimentation, and application to the development of graphical methods. J. Am. Statistical Assoc., 79:531-554, 1984.
8. F. Cole, K. Sanik, D. DeCarlo, A. Finkelstein, T. Funkhouser, S. Rusinkiewicz, and M. Singh. How well do line drawings depict shape? In ACM SIGGRAPH, pp. 1-9, 2009.
9. B. D. Dent. Cartography: Thematic Map Design. William C. Brown Publishing, 1998.
10. W. R. Garner. The processing of information and structure. Erlbaum, 1974.
11. P. P. Gilmartin. Influences of map context on circle perception. Annals Assoc. of Am. Geographers, 71:253-258, 1981.
12. J. Heer, N. Kong, and M. Agrawala. Sizing the horizon: the effects of chart size and layering on the graphical perception of time series visualizations. In ACM CHI, pp. 1303-1312, 2009.
13. P. Ipeirotis. Mechanical Turk: The Demographics, 2008. behind-the-enemy-lines.blogspot.com/2008/03/mechanical-turk- demographics.html.
14. A. Kittur, E. H. Chi, and B. Suh. Crowdsourcing user studies with mechanical turk. In ACM CHI, pp. 453-456, 2008.
15. R. Kohavi, R. M. Henne, and D. Sommerfield. Practical guide to controlled experiments on the web: listen to your customers not to the hippo. In ACM KDD, pp. 959-967, 2007.
16. R. Kraut, J. Olson, M. Banaji, A. Bruckman, J. Cohen, and M. Couper. Psychological research online: Report of board of scientific affairs' advisory group on the conduct of research on the internet. American Psychologist, 59:105-117, 2004.
17. S. Lewandowsky and I. Spence. Discriminating strata in scatterplots. J. Am. Statistical Assoc., 84:682-688, Sep 1989.
18. K. Ling, G. Beenen, P. Ludford, X. Wang, K. Chang, X. Li, D. Cosley, D. Frankowski, L. Terveen, A. M. Rashid, P. Resnick, and R. Kraut. Using social psychology to motivate contributions to online communities. J. CMC, 10, 2005.
19. A. M. MacEachren. How Maps Work: Representation, Visualization, and Design. Guilford, 1995.
20. W. E. Mackay, C. Appert, M. Beaudouin-Lafon, O. Chapuis, Y. Du, J.-D. Fekete, and Y. Guiard. Touchstone: exploratory design of experiments. In ACM CHI, pp. 1425-1434, 2007.
21. J. D. Mackinlay. Automating the design of graphical presentations of relational information. ACM TOG, 5:110-141, 1986.
22. J. D. Mackinlay, P. Hanrahan, and C. Stolte. Show Me: Automatic presentation for visual analysis. IEEE TVCG, 13:1137-1144, Nov/Dec 2007.
23. W. A. Mason and D. J. Watts. Financial incentives and the "performance of crowds". In KDD-HCOMP, 2009.
24. D. F. Reilly and K. M. Inkpen. White rooms and morphing don't mix: setting and the evaluation of visualization techniques. In ACM CHI, pp. 111-120, 2007.
25. M. J. Salganik and D. J. Watts. Web-based experiments for the study of collective social dynamics in cultural markets. Topics in Cognitive Science, 1:439-468, 2009.
26. B. Shneiderman. Tree visualization with tree-maps: 2-d space-filling approach. ACM TOG, 11:92-99, 1992.
27. B. G. Shortridge. Stimulus processing models from psychology: Can we use them in cartography? The American Cartographer, 9:155-167, 1982.
28. D. Simkin and R. Hastie. An information-processing analysis of graph perception. J. Am. Stat. Assoc., 82:454-465, Jun 1987.
29. I. Spence and S. Lewandowsky. Displaying proportions and percentages. Applied Cog. Psych., 5:61-77, 1991.
30. M. Stone and L. Bartram. Alpha, contrast and the perception of visual metadata. In Color Imaging Conf., 2009.
31. L. Tremmel. The visual separability of plotting symbols in scatterplots. J. Comp. and Graph. Stat., 4:101-112, Jun 1995.
32. Turkit. http://groups.csail.mit.edu/uid/turkit/.
33. F. B. Viégas, M. Wattenberg, F. van Ham, J. Kriss, and M. McKeon. ManyEyes: A site for visualization at internet scale. IEEE TVCG, 13(6):1121-1128, 2007.
34. C. Ware. Information Visualization: Perception for Design. Morgan-Kaufmann, 2nd edition, 2004.
35. M. Wattenberg. Visualizing the stock market. In ACM CHI Extended Abstracts, pp. 188-189, 1999.
36. D. Wigdor, C. Shen, C. Forlines, and R. Balakrishnan. Perception of elementary graphical elements in tabletop and multisurface environments. In ACM CHI, pp. 473-482, Apr 2007.
37. C. Ziemkiewicz and R. Kosara. The shaping of information by visual metaphors. IEEE TVCG, 14:1269-1276, Nov/Dec 2008.