CLAMShell: Speeding up crowds for low-latency data labeling

Proceedings of the VLDB Endowment

Volumn 9, Issue 4, 2016, Pages 372-383

  Haas, Daniel ^a   Wang, Jiannan ^b   Wu, Eugene ^c   Franklin, Michael J ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b SIMON FRASER UNIVERSITY   (Canada)

^c Columbia University ^*  (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ACTIVE LEARNING; AMAZON'S MECHANICAL TURKS; DATA LABELING; INTERACTIVE SYSTEM; LABELING STRATEGY; LOW LATENCY; NOVEL TECHNIQUES;

ARTIFICIAL INTELLIGENCE;

EID: 84976515556     PISSN: None     EISSN: 21508097     Source Type: Conference Proceeding    
DOI: None     Document Type: Chapter

References (57)

1. Abouzied, D. J. Abadi, and A. Silberschatz. Invisible loading: access-driven data transfer from raw files into database systems. EDBT, 2013.
2. Agarwal et al. Sentiment analysis of Twitter data. LASM, 2011.
3. Ananthanarayanan et al. Reining in the Outliers in Map- Reduce Clusters using Mantri. OSDI, 2010.
4. Ananthanarayanan et al. Effective Straggler Mitigation: Attack of the Clones. NSDI, 2013.
5. S. Bernstein, J. Brandt, R. C. Miller, and D. R. Karger. Crowds in two seconds: enabling realtime crowd-powered interfaces. UIST, 2011.
6. S. Bernstein et al. Soylent: a word processor with a crowd inside. UIST, 2010.
7. S. Bernstein, D. R. Karger, R. C. Miller, and J. Brandt. Analytic Methods for Optimizing Realtime Crowdsourcing. Col- lective Intelligence, 2012.
8. P. Bigham et al. VizWiz: nearly real-time answers to visual questions. UIST, 2010.
9. Callison-Burch. Fast, cheap, and creative: evaluating translation quality using Amazon's Mechanical Turk. EMNLP, 2009.
10. S. Chakraborty et al. Adaptive Batch Mode Active Learning. Trans. Neural Netw. Learning Sys., 2015.
11. D. A. Cohn, Z. Ghahramani, and M. I. Jordan. Active Learning with Statistical Models. JAIR, 1996.
12. A. Das Sarma et al. Crowd-powered find algorithms. ICDE, 2014.
13. J. Dean and S. Ghemawat. MapReduce: simplified data processing on large clusters. Communications of the ACM, 2008.
14. C. Diaconu et al. Hekaton: SQL server's memory-optimized OLTP engine. SIGMOD, 2013.
15. M. J. Franklin et al. CrowdDB: answering queries with crowdsourcing. SIGMOD, 2011.
16. Y. Gao and A. G. Parameswaran. Finish them!: Pricing algorithms for human computation. PVLDB, 7(14):1965-1976, 2014.
17. A. Ghoting et al. SystemML: Declarative machine learning on MapReduce. ICDE, 2011.
18. C. Gokhale et al. Corleone: hands-off crowdsourcing for entity matching. SIGMOD, 2014.
19. J. E. Gonzalez et al. PowerGraph: Distributed Graph-Parallel Computation on Natural Graphs. OSDI, 2012.
20. I. Guyon. Design of experiments for the NIPS 2003 variable selection benchmark, 2003.
21. D. Haas, S. Krishnan, J. Wang, M. J. Franklin, and E. Wu. Wisteria: Nurturing Scalable Data Cleaning Infrastructure. VLDB, 2015.
22. Hadoop. http://hadoop.apache.org/.
23. P. Ipeirotis and J. Horton. Visualizations of the oDesk "oConomy": Exploring Our World of Work. https://www.upwork.com/ blog/2012/07/visualizations-of-odesk-oconomy/, 2012.
24. P. G. Ipeirotis. Analyzing the Amazon Mechanical Turk marketplace. ACM Crossroads, 2010.
25. P. G. Ipeirotis, F. Provost, and J. Wang. Quality management on Amazon Mechanical Turk. SIGKDD, 2010.
26. M. I. Jordan and T. M. Mitchell. Machine learning: Trends, perspectives, and prospects. Science, 2015.
27. U. Jugel, Z. Jerzak, G. Hackenbroich, and V. Markl. M4: A Visualization-Oriented Time Series Data Aggregation. VLDB, 2014.
28. S. Kandel, A. Paepcke, J. M. Hellerstein, and J. Heer. Wrangler: interactive visual specification of data transformation scripts. CHI, 2011.
29. D. R. Karger, S. Oh, and D. Shah. Iterative Learning for Reliable Crowdsourcing Systems. Advances in neural information processing systems (NIPS), 2011.
30. Keystone ML. http://keystone-ml.org/.
31. A. Kittur, E. H. Chi, and B. Suh. Crowdsourcing user studies with Mechanical Turk. CHI, 2008.
32. A. Krizhevsky. Learning multiple layers of features from tiny images, 2009.
33. G. P. Krueger. Sustained work, fatigue, sleep loss and performance: A review of the issues. Work and Stress, 2007.
34. Y. LeCun et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.
35. Z. Liu, B. Jiang, and J. Heer. imMens: Real-time Visual Querying of Big Data. EuroVis, 2013.
36. A. Marcus, D. Karger, S. Madden, R. Miller, and S. Oh. Counting with the crowd. VLDB, 2012.
37. A. Marcus and A. Parameswaran. Crowdsourced data management industry and academic perspectives. Foundations and Trends in Databases, 2015.
38. A. Marcus, E.Wu, D. Karger, S. Madden, and R. Miller. Humanpowered sorts and joins. VLDB, 2011.
39. S. Melnik et al. Dremel: interactive analysis of web-scale datasets. VLDB, 2010.
40. X. Meng et al. MLlib: Machine Learning in Apache Spark. arXiv.org, 2015.
41. B. Mozafari et al. Scaling up crowd-sourcing to very large datasets: a case for active learning. VLDB, 2014.
42. Amazon Mechanical Turk. https://www.mturk.com/.
43. T. Mühlbauer et al. Instant loading for main memory databases. VLDB, 2013.
44. A. G. Parameswaran et al. DataSift: An Expressive and Accurate Crowd-Powered Search Toolkit. HCOMP, 2013.
45. F. Pedregosa et al. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
46. A. Ramesh et al. Identifying Reliable Workers Swiftly. Technical report, Stanford University, 2012.
47. B. Settles. Active learning literature survey. Technical report, University of Wisconsin-Madison, 2010.
48. M. Stonebraker et al. C-store: a column-oriented DBMS. VLDB, 2005.
49. M. Stonebraker et al. Data Curation at Scale: The Data Tamer System. CIDR, 2013.
50. B. Trushkowsky, T. Kraska, M. J. Franklin, and P. Sarkar. Crowdsourced enumeration queries. ICDE, 2013.
51. J. Wang, T. Kraska, M. J. Franklin, and J. Feng. CrowdER: Crowdsourcing Entity Resolution. VLDB, 2012.
52. J. Wang, S. Krishnan, M. J. Franklin, K. Goldberg, T. Kraska, and T. Milo. A sample-and-clean framework for fast and accurate query processing on dirty data. SIGMOD, 2014.
53. H. Wickham. Bin-summarise-smooth: a framework for visualising large data. Technical report, RStudio, 2013.
54. E. Wu, L. Battle, and S. R. Madden. The Case for Data Visualization Management Systems. VLDB, 2014.
55. M. Zaharia et al. Improving MapReduce Performance in Heterogeneous Environments. OSDI, 2008.
56. M. Zaharia et al. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. NSDI, 2012.
57. M. Zukowski, M. van de Wiel, and P. A. Boncz. Vectorwise: A Vectorized Analytical DBMS. ICDE, 2012.