Combining Crowd and Expert Labels Using Decision Theoretic Active Learning

Proceedings of the 3rd AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2015

Volumn , Issue , 2015, Pages 120-129

  Nguyen, An T ^a   Wallace, Byron C ^b   Lease, Matthew ^b  

^a University of Texas at Austin   (United States)

^b UNIVERSITY OF TEXAS AT AUSTIN   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; LEARNING SYSTEMS;

ACTIVE LEARNING; AUTOMATIC MACHINES; DECISION-THEORETIC; DOMAIN EXPERTS; HIGH-ACCURACY; HUMAN LABELLING; HUMAN-MACHINE; MACHINE-LEARNING; MINIMAL COST; WORKERS';

BUDGET CONTROL;

EID: 85134819655     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1609/hcomp.v3i1.13225     Document Type: Conference Paper

References (29)

1. Barza, M.; Trikalinos, T. A.; and Lau, J. 2009. Statistical considerations in meta-analysis. Infectious disease clinics of North America.
2. Bastian, H.; Glasziou, P.; and Chalmers, I. 2010. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS medicine 7(9):e1000326.
3. Beygelzimer, A.; Dasgupta, S.; and Langford, J. 2009. Importance weighted active learning. In Proceedings of the 26th Annual International Conference on Machine Learning, 49–56. ACM.
4. Bousquet, O., and Bottou, L. 2008. The tradeoffs of large scale learning. In Advances in neural information processing systems, 161–168.
5. Cohen, A. M.; Hersh, W. R.; Peterson, K.; and Yen, P.-Y. 2006. Reducing workload in systematic review preparation using automated citation classification. Journal of the American Medical Informatics Association 13(2):206–219.
6. Cohn, D. A.; Ghahramani, Z.; and Jordan, M. I. 1996. Active learning with statistical models. Journal of artificial intelligence research.
7. Donmez, P., and Carbonell, J. G. 2008. Proactive learning: cost-sensitive active learning with multiple imperfect oracles. In Proceedings of the 17th ACM conference on Information and knowledge management.
8. Horvitz, D. G., and Thompson, D. J. 1952. A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association 47(260):663–685.
9. Jung, H. J., and Lease, M. 2015. Modeling Temporal Crowd Work Quality with Limited Supervision. In Proceedings of the 3rd AAAI Conference on Human Computation (HCOMP).
10. Kapoor, A.; Horvitz, E.; and Basu, S. 2007. Selective supervision: Guiding supervised learning with decision-theoretic active learning. In IJCAI.
11. Laws, F.; Scheible, C.; and Schütze, H. 2011. Active learning with amazon mechanical turk. In Proceedings of the conference on empirical methods in natural language processing, 1546–1556. Association for Computational Linguistics.
12. Lease, M., and Alonso, O. 2014. Crowdsourcing and human computation, introduction. Encyclopedia of Social Network Analysis and Mining (ESNAM) 304–315.
13. Lease, M. 2011. On Quality Control and Machine Learning in Crowdsourcing. In Proceedings of the 3rd Human Computation Workshop (HCOMP) at AAAI, 97–102.
14. Lewis, D. D., and Gale, W. A. 1994. A sequential algorithm for training text classifiers. In Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, 3–12. Springer-Verlag New York, Inc.
15. Mateen, F. J.; Oh, J.; Tergas, A. I.; Bhayani, N. H.; and Kamdar, B. B. 2013. s versus titles and abstracts for initial screening of articles for systematic reviews. Clinical epidemiology 5:89–95.
16. Mortensen, M. L.; Wallace, B. C.; Adam, G. P.; Trikalinos, T. A.; and Kraska, T. 2015. Crowdsourcing citation screening for systematic reviews. In submission.
17. Mozafari, B.; Sarkar, P.; Franklin, M.; Jordan, M.; and Madden, S. 2014. Scaling up crowd-sourcing to very large datasets: A case for active learning. Proceedings of the VLDB Endowment 8(2).
18. Patterson, G.; Horn, G. V.; Belongie, S.; Perona, P.; and Hays, J. 2013. Bootstrapping fine-grained classifiers: Active learning with a crowd in the loop. In NIPS Workshop on Crowdsourcing.
19. Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; Vanderplas, J.; Passos, A.; Cournapeau, D.; Brucher, M.; Perrot, M.; and Duchesnay, E. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12:2825–2830.
20. Roy, N., and McCallum, A. 2001. Toward optimal active learning through sampling estimation of error reduction. In In Proc. 18th International Conf. on Machine Learning.
21. Settles, B. 2012. Active learning. Synthesis Lectures on Artificial Intelligence and Machine Learning.
22. Sheshadri, A., and Lease, M. 2013. Square: A benchmark for research on computing crowd consensus. In First AAAI Conference on Human Computation and Crowdsourcing.
23. Snow, R.; O’Connor, B.; Jurafsky, D.; and Ng, A. Y. 2008. Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of the conference on empirical methods in natural language processing.
24. Wallace, B. C.; Small, K.; Brodley, C. E.; and Trikalinos, T. A. 2010a. Active learning for biomedical citation screening. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining, 173–182. ACM.
25. Wallace, B. C.; Trikalinos, T. A.; Lau, J.; Brodley, C.; and Schmid, C. H. 2010b. Semi-automated screening of biomedical citations for systematic reviews. BMC bioinformatics 11(1):55.
26. Wallace, B. C.; Small, K.; Brodley, C. E.; and Trikalinos, T. A. 2011. Who should label what? instance allocation in multiple expert active learning. In SDM, 176–187. SIAM.
27. Wallace, B. C.; Dahabreh, I. J.;; Moran, K. H.; Brodley, C. E.; and Trikalinos, T. A. 2013a. Active literature discovery for scoping evidence reviews: How many needles are there? In KDD workshop on data mining for healthcare (KDD-DMH).
28. Wallace, B. C.; Dahabreh, I. J.; Schmid, C. H.; Lau, J.; and Trikalinos, T. A. 2013b. Modernizing the systematic review process to inform comparative effectiveness: tools and methods. Journal of comparative effectiveness research 2(3):273–282.
29. Yan, Y.; Fung, G. M.; Rosales, R.; and Dy, J. G. 2011. Active learning from crowds. In Proceedings of the 28th international conference on machine learning (ICML-11), 1161–1168.