How well does active learning actually work? Time-based evaluation of cost-reduction strategies for language documentation

EMNLP 2009 - Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: A Meeting of SIGDAT, a Special Interest Group of ACL, Held in Conjunction with ACL-IJCNLP 2009

Volumn , Issue , 2009, Pages 296-305

  Baldridge, Jason ^a   Palmer, Alexis ^b  

^a UNIVERSITY OF TEXAS AT AUSTIN   (United States)

^b SAARLAND UNIVERSITY   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

ACTIVE LEARNING; COST-SENSITIVE LEARNING; EXAMPLE SELECTION; LABEL SUGGESTION; RANDOM SELECTION;

COMPUTATIONAL LINGUISTICS; NATURAL LANGUAGE PROCESSING SYSTEMS;

COST REDUCTION;

EID: 78650965441     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (21)

1. Shilpa Arora, Eric Nyberg, and Carolyn P. Rosé. 2009. Estimating annotation cost for active learning in a multi-annotator environment. In Proceedings of the NAACL HLT Workshop on Active Learning for Natural Language Processing, pages 18-26, Boulder, CO.
2. Jason Baldridge and Miles Osborne. 2008. Active learning and logarithmic opinion pools for HPSG parse selection. Natural Language Engineering, 14(2):199-222.
3. Sabine Buchholz and Erwin Marsi. 2006. CoNLL-X Shared Task on Multilingual Dependency Parsing. In Proceedings of the Tenth Conference on Computational Natural Language Learning (CoNLL-X), pages 149-164, New York City, June. Association for Computational Linguistics.
4. David A. Cohn, Zoubin Ghahramani, and Michael I. Jordan. 1995. Active learning with statistical models. In G. Tesauro, D. Touretzky, and T. Leen, editors, Advances in Neural Information Processing Systems, volume 7, pages 705-712. The MIT Press.
5. David Crystal. 2000. Language Death. Cambridge University Press, Cambridge.
6. James R. Curran and Stephen Clark. 2003. Investigating GIS and smoothing for maximum entropy taggers. In Proceedings of the 10th Conference of the European Association for Computational Linguistics,pages 91-98.
7. Pinar Donmez and Jaime G. Carbonell. 2008. Proactive learning: Cost-sensitive active learning with multiple imperfect oracles. In Proceedings of CIKM08, Napa Valley, CA.
8. Ben Hachey, Beatrice Alex, and Markus Becker. 2005. Investigating the effects of selective sampling on the annotation task. In Proceedings of the 9th Conference on Computational Natural Language Learning,Ann Arbor, MI.
9. Robbie A. Haertel, Kevin D. Seppi, Eric K. Ringger, and James L. Carroll. 2008. Return on investment for active earning. In Proceedings of the NIPS Workshop on Cost-Sensitive Learning.
10. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: the Penn Treebank. Computational linguistics, 9:313-330.
11. Prem Melville and Raymond J. Mooney. 2004. Diverse ensembles for active learning. In Proceedings of the 21st International Conference on Machine Learning, pages 584-591, Banff, Canada.
12. Grace Ngai and David Yarowsky. 2000. Rule writing or annotation: cost-efficient resource usage for base noun hrase chunking. In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics, pages 117-125, Hong Kong.
13. Alexis Palmer, Taesun Moon, and Jason Baldridge. 2009. Evaluating automation strategies in language documentation. In Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing, pages 36-44, Boulder, CO.
14. Telma Can Pixabaj, Miguel Angel Vicente Méndez, María VicenteMéndez, and Oswaldo Ajcot Damián. 2007. Text Collections in Four Mayan Languages. Archived in The Archive of the Indigenous Languages of Latin America.
15. Adwait Ratnaparkhi. 1998. Maximum Entropy Models for Natural Language Ambiguity Resolution. Ph.D. thesis, University of Pennsylvania, Philadelphia, PA.
16. Christian Ritz and Jens Carl Streibig. 2008. Nonlinear Regression with R. Springer.
17. Burr Settles, Mark Craven, and Lewis Friedland. 2008.Active learning with real annotation costs. In Proceedings of the NIPS Workshop on Cost-Sensitive Learning.
18. Burr Settles. 2009. Active learning literature survey. Technical Report Computer Sciences Technical Report 1648, University of Wisconsin-Madison.
19. Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast - but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of EMNLP 2008, pages 254-263.
20. Katrin Tomanek and Fredrik Olsson. 2009. A Web Survey on the Use of Active learning to support annotation of text ata. In Proceedings of the NAACL HLT Workshop on Active Learning for Natural Language Processing, pages 45-48, Boulder, CO.
21. Sudheendra Vijayanarasimhan and Kristen Grauman. 2008. Multi-level active prediction of useful image annotations for recognition. In Proceedings of NIPS08, Vancouver, Canada.