Estimating Annotation Cost for Active Learning in a Multi-Annotator Environment

NAACL HLT 2009 - Active Learning for Natural Language Processing, ALNLP 2009 - Proceedings of the Workshop

Volumn , Issue , 2009, Pages 18-26

  Arora, Shilpa ^a   Nyberg, Eric ^a   Rosé, Carolyn P ^a  

^a Carnegie Mellon University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; CLASSIFICATION (OF INFORMATION); COST BENEFIT ANALYSIS; COST ESTIMATING;

ACTIVE LEARNING; CLASSIFICATION TASKS; CORRELATION COEFFICIENT; COST ESTIMATIONS; EMPIRICAL INVESTIGATION; MOVIE REVIEWS; ROOT MEAN SQUARE ERRORS; SAMPLING STRATEGIES; SELECTIVE SAMPLING; TASK CHARACTERISTICS;

MEAN SQUARE ERROR;

EID: 85012906154     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (16)

1. Shilpa Arora and Eric Nyberg. 2009. Interactive Annotation Learning with Indirect Feature Voting. In Proceedings of NAACL-HLT 2009 (Student Research Workshop).
2. Xavier Carreras and Lluìs Márquez. 2004. Introduction to the CoNLL-2004 Shared Task: Semantic Role Labeling. http://www.lsi.upc.edu/srlconll/st04/st04.html.
3. Gahgene Gweon, Carolyn Penstein Ros’e, Joerg Wittwer and Matthias Nueckles. 2005. Supporting Efficient and Reliable Content Analysis Using Automatic Text Processing Technology. In proceedings of INTERACT 2005: 1112-1115.
4. Robbie A. Haertel, Kevin D. Seppi, Eric K. Ringger and Janes L. Cattoll. 2008. Return on Investment for Active Learning. In proceedings of NIPS Workshop on Cost Sensitive Learning.
5. Rebecca Hwa. 2000. Sample Selection for Statistical Grammar Induction. In proceedings of joint SIGDAT conference on Empirical Methods in NLP and Very Large Corpora.
6. Ashish Kapoor, Eric Horvitz and Sumit Basu. 2007. Selective supervision:Guiding supervised learning with decision-theoretic active learning. In proceedings of IJCAI, pages 877-882.
7. Ross D. King, Kenneth E. Whelan, Ffion M. Jones, Philip G. K. Reiser, Christopher H. Bryant, Stephen H. Muggleton, Douglas B. Kell and Stephen G. Oliver. 2004. Functional Genomics hypothesis generation and experimentation by a robot scientist. In proceedings of Nature, 427(6971):247-52.
8. Trausti Kristjansson, Aron Culotta, Paul Viola and Andrew Mccallum. 2004. Interactive Information Extraction with Constrained Conditional Random Fields. In proceedings of AAAI.
9. Karl Pearson. 1895. Correlation Coefficient. Royal Society Proceedings, 58, 214.
10. Eric Ringger, Marc Carmen, Robbie Haertel, Kevin Seppi, Deryle Lonsdale, Peter McClanahan, Janes L. Cattoll and Noel Ellison. 2008. Assessing the Costs of Machine-Assisted Corpus Annotation through a User Study. In proceedings of LREC.
11. Burr Settles, Mark Craven and Lewis Friedland. 2008. Active Learning with Real Annotation Costs. In proceedings of NIPS Workshop on Cost Sensitive Learning.
12. Alex J. Smola and Bernhard Scholkopf 1998. A Tutorial on Support Vector Regression. NeuroCOLT2 Technical Report Series - NC2-TR-1998-030.
13. Katrin Tomanek, Joachim Wermter and Udo Hahn. 2007. An approach to text corpus construction which cuts annotation costs and maintains reusability of annotated data. In proceedings of EMNLP-CoNLL, pp. 486-495.
14. Theresa Wilson, Janyce Wiebe and Paul Hoffmann. 2005. Recognizing Contextual Polarity in Phrase-Level Sentiment Analysis. In proceedings of HLT/EMNLP, Vancouver, Canada.
15. Ian H. Witten and Eibe Frank. 2005. Data Mining: Practical machine learning tools and techniques. 2nd Edition, Morgan Kaufmann, San Francisco.
16. Omar Zaidan, Jason Eisner and Christine Piatko. 2007. Using”annotator rationales” to improve machine learning for text categorization. In Proceedings of NAACL-HLT, pp. 260-267, Rochester, NY.