Human computation in visualization: Using purpose driven games for robust evaluation of visualization algorithms

IEEE Transactions on Visualization and Computer Graphics

Volumn 18, Issue 12, 2012, Pages 2104-2113

  Ahmed, Nafees ^a   Zheng, Ziyi ^a   Mueller, Klaus ^a  

^a STONY BROOK UNIVERSITY   (United States)

Author keywords

color blending; evaluation; Human computation; perception

Indexed keywords

COLOR-BLENDING; COMMON SENSE; COMPUTATION RESOURCES; COMPUTATIONAL PROBLEM; COMPUTATIONAL SYSTEM; COST EFFECTIVE; DEVELOPMENT PROCESS; EVALUATION; EVALUATION PLATFORMS; HUMAN BRAIN; HUMAN COGNITION; HUMAN COMPUTATION; HUMAN KNOWLEDGE; HYBRID COMPUTATIONAL; INHERENT CHARACTERISTICS; SENSORY SYSTEM; VISUALIZATION ALGORITHMS; VISUALIZATION METHOD; VISUALIZATION PROCESS; VISUALIZATION RESEARCH;

ALGORITHMS; BLENDING; BRAIN; SENSORY PERCEPTION;

VISUALIZATION;

ALGORITHM; ARTIFICIAL INTELLIGENCE; BIOLOGICAL MODEL; COLOR; COMPUTER SIMULATION; HUMAN; VIDEO GAME; VISION;

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COLOR; COMPUTER SIMULATION; HUMANS; MODELS, NEUROLOGICAL; VIDEO GAMES; VISUAL PERCEPTION;

EID: 84867489001     PISSN: 10772626     EISSN: None     Source Type: Journal    
DOI: 10.1109/TVCG.2012.234     Document Type: Article

References (46)

1. (March 2012). ASP.NET AJAX. Available: http://www.asD.net/aiax
2. (March 2012). Facebook social plugin. Available: http://developers. facebook.com/docs/plugins/
3. (March 2012). GLSL. Available: http://www.opengl.org/documentation/glsl/
4. (March 2012). Visualization Pipeline. Available: http://www.infovis-wiki. net/index.php/VisualizationPipeline
5. (March 2012). WebGL. Available: http://www.khronos.ore/webel/
6. L. von Ahn, B. Maurer, C. McMillen, D. Abraham, and M. Blum. "recaptcha: Human-based character recognition via web security measures", Science, vol. 321, pp. 1465-1468, 2008.
7. Luis von Ahn and Laura Dabbish, "Labeling images with a computer game", presented at the Proceedings of the SIGCHI, Vienna, Austria, pp. 319-326,2004.
8. Luis von Ahn, Shiry Ginosar, Mihir Kedia, Ruoran Liu, and Manuel Blum, "Improving accessibility of the web with a computer game", presented at the Proceedings of the SIGCHI, Montreal, Quebec, Canada! pp. 79-82, 2006. (Pubitemid 44032089)
9. Luis von Ahn, Mihir Kedia, and Manuel Blum, "Verbosity: a game for collecting common-sense facts", presented at the Proceedings of the SIGCHI, Montreal, Quebec, Canada, pp. 75-78, 2006. (Pubitemid 44032088)
10. Luis von Ahn, Ruoran Liu, and Manuel Blum, "Peekaboom: a game for locating objects in images", presented at the Proceedings of the SIGCHI. Montreal, Quebec, Canada, pp. 55-64, 2006. (Pubitemid 44032086)
11. Michael S. Bernstein, Greg Little, Robert C Miller, Bjorn Hartmann, Mark S. Ackerman, David R Karger, David Crowell, and Katrina Panovich, "Soylent: a word processor with a crowd inside", presented at the Proceedings of UIST, New York, New York, USA, pp. 313-322, 2010.
12. Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, and Tom Yeh, "VizWiz: nearly real-time answers to visual questions", presented at the Proceedings of UIST, New York. New York, USA, pp. 333-342, 2010.
13. Chaomei Chen, "Top 10 Unsolved Information Visualization Problems". IEEE Comput. Graph. Appi., vol. 25, pp. 12-16, 2005. (Pubitemid 41046857)
14. Johnson Chuang, Daniel Weiskopf, and Torsten Moller, "Hue-Preserving Color Blending", IEEE Transactions on Visualization and Computer Graphics, vol. 15, pp. 1275-1282,2009.
15. Forrester Cole, Kevin Sanik, Doug DeCarlo, Adam Finkelstein, Thomas Funkhouser, Szymon Rusinkiewicz, and Manish Singh, "How Well Do Line Drawings Depict Shape", ACM Transactions on Graphics (Proc. SIGGRAPH),vol.28,2009.
16. Seth Cooper, Firas Khatib, Adrien Tremile, Janos Barbero, Jeehyung Lee, Michael Beenen, Andrew Leaver-Fay, David Baker, Zoran Popovic, and Foldit players, "Predicting protein structures with a multiplayer online game", Nature, vol. 466, pp. 756-760, 2010.
17. C. Correa and Ma Kwan-Liu, "The Occlusion Spectrum for Volume Classification and Visualization", Visualization and Computer Graphics, IEEE Transactions on, vol. 15, pp. 1465-1472, 2009.
18. J. Davis, J. Arderiu, H. Lin, Z. Nevins, S. Schuon, O. Gallo, and Yang Ming-Hsuan, "The HPU", in Computer Vision and Pattern Recognition Workshops (CVPRW), 2010, pp. 9-16.
19. M. Franklin, D. Kossmann, T. Kraska, S. Ramesh, and R. Xin. "CrowdDB: Answering queries with crowdsourcing", presented at the Proceedings of the 2011 ACM SIGMOD, pp. 61-72, 2011.
20. David Alan Grier, When Computers were Human: Princeton University Press, 2005.
21. R. B. Haber and D. A. McNabb, "Visualization idioms: A conceptual model for scientific visualization systems", Visualization in Scientific Computing, pp. 74-93, 1990.
22. Jeffrey Heer and Michael Bostock, "Crowdsourcing graphical perception: using mechanical turk to assess visualization design", presented at the Proceedings of SIGCHI, Atlanta, Georgia, USA, pp! 203-212,2010.
23. Chang Hu, Benjamin B. Bederson, Philip Resnik, and Yakov Kronrod. "MonoTrans2: a new human computation system to support monolingual translation", presented at the Proceedings of SIGCHI, Vancouver, BC, Canada, pp. 1133-1136, 2011.
24. Panagiotis G Ipeirotis, Foster Provost, and Jing Wang, "Quality management on Amazon Mechanical Turk", presented at the Proceedings of the ACM SIGKDD Workshop on Human Computation, Washington DC, pp. 64-67, 2010.
25. Aniket Kittur, Ed H. Chi, and Bongwon Suh, "Crowdsourcing user studies with Mechanical Turk", presented at the Proceedings of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, Florence, Italy, pp. 453-456, 2008.
26. Robert Kosara and Caroline Ziemkiewicz, "Do Mechanical Turks dream of square pie charts?", presented at the Proceedings of the 3rd BELIV10 Workshop: BEyond time and errors: novel evaLuation methods for Information Visualization, Atlanta, Georgia, pp. 63-70, 2010.
27. Lars Kühne, Joachim Giesen, Zhiyuan Zhang, Sungsoo Ha, and Klaus Mueller, "A Data-Driven Approach to Hue-Preserving Color-Blending", IEEE Transaction on Visualization and Computer Graphics, 2012.
28. Anand P. Kulkarni, Matthew Can, and Bjoern Hartmann, "Turkomatic: automatic recursive task and workflow design for mechanical turk". presented at the Proceedings of the 2011 annual conference extended abstracts on Human factors in computing systems, Vancouver, BC, Canada, pp. 2053-2058, 2011.
29. Edith Law and Luis von Ahn, Human Computation: Morgan & Claypool Publishers, 2011.
30. Edith Law, Luis Von Ahn, Roger B. Dannenberg, and Mike Crawford. "TagATune: A Game for Music and Sound Annotation", ISMIR pp! 361-364,2007.
31. Greg Little, Lydia B. Chilton, Max Goldman, and Robert C Miller. "TurKit: tools for iterative tasks on mechanical Turk", presented at the Proceedings of the ACM SIGKDD Workshop on Human Computation, Paris, France, pp. 29-30, 2009.
32. Wang Lujin, J. Giesen, K. T. McDonnell, P. Zolliker, and K. Mueller. "Color Design for Illustrative Visualization", IEEE Transaction on Visualization and Computer Graphics, vol. 14, pp. 1739-1754, 2008.
33. Adam Marcus, Eugene Wu, David Karger, Samuel Madden, and Robert Miller, "Human-powered sorts and joins", Proc. VLDB Endow., vol. 5, pp. 13-24, 2011.
34. F. Mettali, "The perception of transparency", Scientific American, vol. 230, pp. 91-98, 1974.
35. Chan Ming-Yuen, Wu Yingcai, Mak Wai-Ho, Chen Wei, and Qu Huamin, "Perception-Based Transparency Optimization for Direct Volume Rendering", Visualization and Computer Graphics, IEEE Transactions on, vol. 15, pp. 1283-1290, 2009.
36. Catherine Plaisant, "The challenge of information visualization evaluation", presented at the Proceedings of the working conference on Advanced visual interfaces, Gallipoli, Italy, pp. 109-116, 2004.
37. Thomas Porter and Tom Duff, "Compositing digital images". SIGGRAPH Comput. Graph., vol. 18, pp. 253-259, 1984.
38. Joel Ross, Lilly Irani, M. Six Silberman, Andrew Zaldivar, and Bill Tomlinson, "Who are the crowdworkers? : shifting demographics in mechanical turk", presented at the Proceedings of the 28th of the international conference extended abstracts on Human factors in computing systems, Atlanta, Georgia, USA pp. 2863-2872, 2010.
39. Stuart Russel and Peter Norvig, "Intelligent Agents", in Artificial Intelligence: A Modern Approach, Third Edition ed: Prentice Hall, 2010, p. 54.
40. Petr Sereda, Anna Vilanova, and Frans A. Gerritsen, "Automating Transfer Function Design for Volume Rendering Using Hierarchical Clustering of Material Boundaries", Eurographics/IEEE VGTC Symposium on Visualization (EuroVis), pp. 243-250, 2006.
41. M. Singh and BL. Anderson, "Perceptual assignment of opacity to translucent surfaces: The role of image blur", Perception, vol. 31, pp. 531-552,2002.
42. M. Singh and BL. Anderson, "Toward a perceptual theory of transparency", Pschol Rev, vol. 109, pp. 492-519, 2002.
43. Iris Sprow, Zona Bara'nczuk, Tobias Stamm, and Peter Zolliker, "Web-based psychometric evaluation of image quality", Image Quality and System Performance VI (Proc. SPIE), vol. 7242, 2009.
44. D. G Stork, "The Open Mind Initiative", IEEE Intelligent Systems & Their Applications, vol. 14, pp. 19-20, 1999.
45. Jarke J. Van Wijk, "The value of visualization", IEEE Visualization, pp. 79-86, 2005.
46. Ziyi Zheng, Nafees Ahmed, and Klaus Mueller, "¡View: A Feature Clustering Framework for Suggesting Informative Views in Volume Visualization", Visualization and Computer Graphics, IEEE Transactions on, vol. 17, pp. 1959-1968,2011.