Who should label what? instance allocation in multiple expert active learning

Proceedings of the 11th SIAM International Conference on Data Mining, SDM 2011

Volumn , Issue , 2011, Pages 176-187

  Wallace, Byron C ^a,b   Small, Kevin ^a   Brodley, Carla E ^a   Trikalinos, Thomas A ^b  

^a Tufts University   (United States)

^b TUFTS MEDICAL CENTER   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; SENTIMENT ANALYSIS;

ACTIVE LEARNING; BIOMEDICAL CITATIONS; FIXED COST; METACOGNITIVES; MULTIPLE DOMAINS; NOVEL ALGORITHM; PREDICTIVE MODELING; REAL-WORLD;

DATA MINING;

EID: 84874086932     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1137/1.9781611972818.16     Document Type: Conference Paper

References (23)

1. V. Ambati, S. Vogel, and J. Carbonell. Active learning and crowd-sourcing for machine translation. In Language Resources and Evaluation (LREC), 2010.
2. S. Arora, E. Nyberg, and C. Rosé. Estimating annotation cost for active learning in a multi-annotator environment. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL) Workshop on Active Learning for Natural Language Processing, pages 18-26. Association for Computational Linguistics, 2009.
3. J. Attenberg and F. Provost. Why label when you can search?: alternatives to active learning for applying human resources to build classification models under extreme class imbalance. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 423-432. ACM, 2010.
4. B. Beigman Klebanov and E. Beigman. From anno-tator agreement to noise models. Computational Linguistics, 35(4):495-503, 2009.
5. P. Castaldi, M. Cho, M. Cohn, F. Langerman, S. Moran, N. Tarragona, H. Moukhachen, R. Venugopal, D. Hasimja, E. Kao, B. Wallace, C. Hersh, S. Bagade, L. Bertram, E. Silverman, and T. Trikalinos. The COPD genetic association compendium: a comprehensive online database of COPD genetic associations. Human Molecular Genetics, 2009.
6. C. Cole, G. Binney, P. Casey, J. Fiascone, J. Hagadorn, C. Kim, C. Wang, D. Devine, K. Miller, and J. Lau. Criteria for determining disability in infants and children: Low birth weight. Evidence Report/Technology Assessment No. 70. Prepared by New England Medical Center Evidence-based Practice Center under Contract No. 290-97-0019, 2002.
7. P. Donmez, J. Carbonell, and J. Schneider. Efficiently learning the accuracy of labeling sources for selective sampling. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge Discovery and Data Mining, pages 259-268. ACM, 2009.
8. P. Donmez and J. G. Carbonell. Proactive learning: Cost-sensitive active learning with multiple imperfect oracles. In Conference on Information and Knowledge Management (CIKM), pages 619-628, 2008.
9. L. Hunter and K. B. Cohen. Biomedical language processing: What's beyond pubmed? Mol Cell, 21(5):589-594, March 2006.
10. J. Kruger and D. Dunning. Unskilled and unaware of it: How difficulties in recognizing ones own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77(6):1121-1134, 1999.
11. B. Pang and L. Lee. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the ACL, pages 271-278, 2004.
12. J. Platt. Probabilistic outputs for SVMs and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers, pages 61-74, 1999.
13. V. Raykar, S. Yu, L. Zhao, A. Jerebko, C. Florin, G. Valadez, L. Bogoni, and L. Moy. Supervised Learning from Multiple Experts: Whom to trust when everyone lies a bit. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 889-896. ACM, 2009.
14. I. Rehbein, J. Ruppenhofer, and A. Palmer. Bringing Active Learning to Life. In Proceedings of the 23rd International Conference on Computational Linguistics (COLING), 2010.
15. B. Settles. Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin-Madison, 2009.
16. V. S. Sheng, F. Provost, and P. G. Ipeirotis. Get another label? Improving data quality and data mining using multiple, noisy labelers. In Knowledge Discovery and Data mining (KDD), pages 614-622, 2008.
17. T. Terasawa, T. Dvorak, S. Ip, G. Raman, J. Lau, and T. A. Trikalinos. Charged Particle Radiation Therapy for Cancer: A Systematic Review. Ann. Intern. Med., pages 556-565, 2009.
18. S. Tong and D. Koller. Support vector machine active learning with applications to text classification. In Journal of Machine Learning Research, pages 999-1006, 2000.
19. A. J. Vickers and E. B. Elkin. Decision curve analysis: A novel method for evaluating prediction models. Medical Decision Making, 26:565-574, 2006.
20. B. C. Wallace, K. Small, C. E. Brodley, and T. A. Trikalinos. Active learning for biomedical citation screening. In Knowledge Discovery and Data Mining (KDD), pages 173-182, 2010.
21. B. C. Wallace, T. A. Trikalinos, J. Lau, C. E. Brodley, and C. H. Schmid. Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinformatics, 11:55-67, 2010.
22. P. Wheeler, E. Balk, K. Bresnahan, B. Shephard, J. Lau, D. DeVine, M. Chung, and K. Miller. Criteria for determining disability in infants and children: short stature. Evidence Report/Technology Assessment No. 73. Prepared by New England Medical Center Evidence-based Practice Center under Contract No. 290-97-001, Mar 2003.
23. J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. Movellan. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Proceedings of the 2009 Neural Information Processing Systems (NIPS) Conference, pages 2035-2043, 2009.