Scaling up crowdsourcing to very large datasets: A case for active learning

Proceedings of the VLDB Endowment

Volumn 8, Issue 2 2, 2014, Pages 125-136

  Mozafari, Barzan ^a   Sarkar, Purna ^b   Franklin, M Michael ^b   Jordan, Michael ^b   Madden, Samuel ^c  

^a UNIVERSITY OF MICHIGAN   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; CROWDSOURCING; LEARNING ALGORITHMS; LEARNING SYSTEMS; SENTIMENT ANALYSIS;

ACTIVE LEARNING; ACTIVE-LEARNING ALGORITHM; CROWD SOURCING; ENTITY RESOLUTIONS; IMAGE TAGGING; LABELED DATA; LARGE DATASETS; MACHINE-LEARNING; RESOLUTION ANALYSIS; SCALING-UP;

DATABASE SYSTEMS;

EID: 84938053293     PISSN: None     EISSN: 21508097     Source Type: Journal    
DOI: 10.14778/2735471.2735474     Document Type: Conference Paper

References (40)

1. D. N. A. Asuncion. UCI machine learning repository, 2007.
2. A. Agarwal, L. Bottou, M. Dudík, and J. Langford. Para-active learning. CoRR, abs/1310.8243, 2013.
3. A. Arasu, M. Götz, and R. Kaushik. On active learning of record matching packages. In SIGMOD, 2010.
4. K. Bellare et al. Active sampling for entity matching. In KDD, 2012.
5. A. Beygelzimer, S. Dasgupta, and J. Langford. Importance weighted active learning. In ICML, 2009.
6. A. Beygelzimer, D. Hsu, J. Langford, and T. Zhang. Agnostic active learning without constraints. In NIPS, 2010.
7. C. M. Bishop. Pattern Recognition and Machine Learning. Springer-Verlag New York, Inc., 2006.
8. A. Bosch, A. Zisserman, and X. Muoz. Image classification using random forests and ferns. In ICCV, 2007.
9. A. Brew, D. Greene, and P. Cunningham. Using crowdsourcing and active learning to track sentiment in online media. In ECAI, 2010.
10. A. Chatterjee and S. N. Lahiri. Bootstrapping lasso estimators. Journal of the American Statistical Association, 106(494):608-625, 2011.
11. D. A. Cohn, Z. Ghahramani, and M. I. Jordan. Active learning with statistical models. J. Artif. Int. Res., 4, 1996.
12. S. Dasgupta, D. Hsu, and C. Monteleoni. A general agnostic active learning algorithm. In ISAIM, 2008.
13. A. Dawid and A. Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Applied Statistics, 1979.
14. P. Donmez, J. G. Carbonell, and J. Schneider. Efficiently learning the accuracy of labeling sources for selective sampling. In KDD, 2009.
15. B. Efron and R. J. Tibshirani. An Introduction to the Bootstrap. Chapman & Hall, 1993.
16. L. Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples. InWGMBV, 2004.
17. M. J. Franklin, D. Kossmann, T. Kraska, S. Ramesh, and R. Xin. Crowddb: answering queries with crowdsourcing. In SIGMOD, 2011.
18. S. Hoi, R. Jin, J. Zhu, and M. Lyu. Batch mode active learning and its application to medical image classification. In ICML, 2006.
19. A. Kittur, B. Smus, S. Khamkar, and R. E. Kraut. Crowdforge:crowdsourcing complex work. In UIST, 2011.
20. A. Kleiner et al.The big data bootstrap. In ICML, 2012.
21. R. Kohavi and D. H. Wolpert. Bias plus variance decomposition for zero-one loss functions. In ICML, 1996.
22. S. Lahiri. On bootstrapping m-estimators. Sankhyā. Series A. Methods and Techniques, 54(2), 1992.
23. F. Laws, C. Scheible, and H. Schütze. Active learning with amazon mechanical turk. In EMNLP, 2011.
24. A. Marcus, D. R. Karger, S. Madden, R. Miller, and S. Oh. Counting with the crowd. PVLDB, 2012.
25. A. Marcus, E.Wu, D. Karger, S. Madden, and R. Miller. Human-powered sorts and joins. PVLDB, 5, 2011.
26. B. Mozafari, P. Sarkar, M. J. Franklin, M. I. Jordan, and S. Madden. Active learning for crowd-sourced databases. CoRR, 2012.
27. A. Pak and P. Paroubek. Twitter as a corpus for sentiment analysis and opinion mining. In Proceedings of LREC, 2010.
28. A. G. Parameswaran et al. Crowdscreen: algorithms for filtering data with humans. In SIGMOD, 2012.
29. M. Saar-Tsechansky and F. Provost. Active sampling for class probability estimation and ranking. Mach. Learn., 54, 2004.
30. S. Sarawagi and A. Bhamidipaty. Interactive deduplication using active learning. In SIGKDD, 2002.
31. B. Settles. Active learning literature survey. Computer Sciences Technical Report 1648, University ofWisconsin-Madison, 2010.
32. V. Sheng, F. Provost, and P. Ipeirotis. Get another label? In KDD, 2008.
33. B. Trushkowsky, T. Kraska, M. J. Franklin, and P. Sarkar. Crowdsourced enumeration queries. In ICDE, 2013.
34. M.-H. Tsai et al. Active learning strategies using svms. In IJCNN, 2010.
35. S. Vijayanarasimhan and K. Grauman. Cost-sensitive active visual category learning. Int. J. Comput. Vision, 91, 2011.
36. A. Vlachos. A stopping criterion for active learning. Comput. Speech Lang., 22(3), 2008.
37. J.Wang, T. Kraska, M. J. Franklin, and J. Feng. CrowdER:Crowdsourcing entity resolution. PVLDB, 5, 2012.
38. K. Zeng, S. Gao, J. Gu, B. Mozafari, and C. Zaniolo. ABS: a system for scalable approximate queries with accuracy guarantees. In SIGMOD, 2014.
39. K. Zeng, S. Gao, B.Mozafari, and C. Zaniolo.The analytical bootstrap:a new method for fast error estimation in approximate query processing. In SIGMOD, 2014.
40. D. Zhou, S. Basu, Y.Mao, and J. C. Platt. Learning from thewisdomof crowds by minimax entropy. In Advances in NIPS, 2012.