Identifying redundancy and exposing provenance in crowdsourced data analysis

IEEE Transactions on Visualization and Computer Graphics

Volumn 19, Issue 12, 2013, Pages 2198-2206

  Willett, Wesley ^a   Ginosar, Shiry ^b   Steinitz, Avital ^b   Hartmann, Bjorn ^b   Agrawala, Maneesh ^b  

^a INRIA   (France)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Crowdsourcing; Social Data Analysis

Indexed keywords

BROWSING BEHAVIOR; CLUSTERINGS; COLOR CLUSTERING; CROWDSOURCING; DOMAIN KNOWLEDGE; WORKERS';

COMPUTER GRAPHICS;

SOFTWARE ENGINEERING;

ALGORITHM; COMPUTER GRAPHICS; COMPUTER INTERFACE; DATA MINING; FACTUAL DATABASE; INFORMATION RETRIEVAL; INTERNET; PROCEDURES; ARTICLE; METHODOLOGY;

ALGORITHMS; COMPUTER GRAPHICS; DATA MINING; DATABASES, FACTUAL; INFORMATION STORAGE AND RETRIEVAL; INTERNET; USER-COMPUTER INTERFACE;

EID: 84886570509     PISSN: 10772626     EISSN: None     Source Type: Journal    
DOI: 10.1109/TVCG.2013.164     Document Type: Article

References (23)

1. Same Origin Policy. http://www.w3.org/Security/wiki/Same Origin Policy.
2. Amiǵo, E., Gonzalo, J., Artiles, J., and Verdejo, F. A comparison of extrinsic clustering evaluation metrics based on formal constraints. Information Retrieval 12, 4 (July 2008), 461-486.
3. Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D., and Panovich, K. Soylent: A Word Processor with a Crowd Inside. In Proceedings of UIST, ACM (2010), 313-322.
4. Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Miller, R., Tatarowicz, A., White, B., White, S., and Yeh, T. VizWiz: nearly realtime answers to visual questions. In Proc. of UIST (2010), 333-342.
5. Chilton, L., Little, G., Edge, D., Weld, D. S., and Landay, J. A. Cascade: Crowdsourcing Taxonomy Creation. In Proc. of CHI, ACM (2013).
6. Fisher, K., Counts, S., and Kittur, A. Distributed sensemaking: improving sensemaking by leveraging the efforts of previous users. In Proceedings of CHI, ACM (2012), 247-256.
7. Heer, J., Víegas, F. B., and Wattenberg, M. Voyagers and voyeurs: Supporting asynchronous collaborative visualization. Communications of the ACM 52, 1 (2009), 87-97.
8. Luther, K., Counts, S., Stecher, K. B., Hoff, A., and Johns, P. Pathfinder: An Online Collaboration Environment for Citizen Scientists. In Proceedings of CHI, ACM (2009), 239-248.
9. Manning, C. D., Raghavan, P., and Scḧutze, H. Introduction to Information Retrieval. Cambridge University Press, 2008.
10. Pirolli, P., and Card, S. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. Proceedings of International Conference on Intelligence Analysis 5 (2005).
11. Rzeszotarski, J. M., and Kittur, A. Instrumenting the crowd: using implicit behavioral measures to predict task performance. In Proceedings of UIST, ACM (2011), 13-22.
12. Stone, B., Dennis, S., and Kwantes, P. J. Comparing Methods for Single Paragraph Similarity Analysis. Topics in Cog. Sci. 3, 1 (2010), 92-122.
13. Surowiecki, J. The Wisdom of Crowds. Anchor, 2005.
14. Tamuz, O., Liu, C., Belongie, S., Shamir, O., and Kalai, A. T. Adaptively Learning the Crowd Kernel. arXiv.org (2011).
15. Tibshirani, R., Walther, G., and Hastie, T. Estimating the number of clusters in a data set via the gap statistic. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 63, 2 (2001), 411-423.
16. Trushkowsky, B., Kraska, T., Franklin, M. J., and Sarkar, P. Getting It All from the Crowd. arXiv.org (2012).
17. Víegas, F. B., Wattenberg, M., Ham, F. v., Kriss, J., and McKeon, M. Many Eyes: A Site for Visualization at Internet Scale . Trasactions on Visualization and Computer Graphics 13, 6 (2007), 1121-1128.
18. Víegas, F. B., Wattenberg, M., McKeon, M., Ham, F. V., and Kriss, J. Harry potter and the meat-filled freezer: A case study of spontaneous usage of visualization tools. In Proceedings of HICSS (2008).
19. Willett, W., Heer, J., and Agrawala, M. Strategies for crowdsourcing social data analysis. In Proceedings of CHI, ACM (2012), 227-236.
20. Willett, W., Heer, J., Hellerstein, J. M., and Agrawala, M. CommentSpace: Structured Support for Collaborative Visual Analysis. In Proceedings of CHI, ACM (2011).
21. Wu, M., and Marian, A. Corroborating answers from multiple web sources. In Proceedings of WebDB (2007).
22. Yi, J., Jin, R., Jain, A. K., and Jain, S. Crowdclustering with Sparse Pairwise Labels: A Matrix Completion Approach. In Workshops at the Conference on Artificial Intelligence, AAAI (2012).
23. Zhang, H., Law, E., Miller, R., Gajos, K., Parkes, D., and Horvitz, E. Human Computation Tasks with Global Constraints. In Proceedings of CHI, ACM (2012), 217-226.