Confidence-based stopping criteria for active learning for data annotation

ACM Transactions on Speech and Language Processing

Volumn 6, Issue 3, 2010, Pages

  Zhu, Jingbo ^a   Wang, Huizhen ^a   Hovy, Eduard ^b   Ma, Matthew ^c  

^a NORTHEASTERN UNIVERSITY   (China)

^b UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^c Scientific Works   (United States)

Author keywords

Active learning; Confidence estimation; Stopping criterion; Text classification; Uncertainty sampling; Word sense disambiguation

Indexed keywords

ACTIVE LEARNING; CONFIDENCE ESTIMATION; STOPPING CRITERIA; TEXT CLASSIFICATION; WORD SENSE DISAMBIGUATION;

CLASSIFICATION (OF INFORMATION); COMPUTATIONAL LINGUISTICS; COST REDUCTION; ESTIMATION; SUPERVISED LEARNING; TEXT PROCESSING;

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 77953755634     PISSN: 15504875     EISSN: 15504883     Source Type: Journal    
DOI: 10.1145/1753783.1753784     Document Type: Article

References (47)

1. ANGLIUN, D. 1988. Queries and concept learning. Mach. Learn. 2, 3, 319-342.
2. BARAM, Y., EL-YANIV, R., AND LUZ, K. 2004. Online choice of active learning algorithms. J. Mach. Learn. Res. 5, 255-291.
3. BECKER, M. AND OSBORNE, M. 2005. A two-stage method for active learning of statistical grammars. In Proceedings of the 19th International Joint Conference on Artificial Intelligence. 991-996.
4. BERGER, A. L., DELLA PIETRA, S. A., AND DELLA PIETRA, V. J. 1996. A maximum entropy approach to natural language processing. Comput. Linguist. 22, 1, 39-71.
5. BRUCE, R. AND WIEBE, J. 1994. Word-Sense disambiguation using decomposable models. In Proceedings of the 32nd Annual Meeting of the Association for Computational Linguistics. 139-146.
6. CAMPBELL, C., CRISTIANINI, N., AND SMOLA, A. 2000. Query learning with large margin classifiers. In Proceedings of the International Conference on Machine Learning. 111-118.
7. CHAN, Y. S. AND NG, H. T. 2007. Domain adaptation with active learning for word sense disambiguation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics. 49-56.
8. CHEN, J., SCHEIN, A.,UNGAR, L., AND PALMER, M. 2006. An empirical study of the behavior of active learning for word sense disambiguation. In Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL. 120-127.
9. COHN, D. A., ATLAS, L., AND LADNER, R. E. 1994. Improving generalization with active learning. Mach. Learn. 15, 201-221.
10. COHN, D. A., GHAHRAMANI, Z., AND JORDAN, M. I. 1996. Active learning with statistical models. J. Artif. Intell. Res. 4, 129-145.
11. CULOTTA, A. AND MCCALLUM, A. 2005. Reducing labeling effort for structured prediction tasks. In Proceedings of the 20th National Conference on Artificial Intelligence (AAAI-05). 746-751.
12. DIMISTAKAKIS, C. AND SAVU-KROHN, C. 2008. Cost-Minimizing strategies for data labeling: Optimal stopping and active learning. In Proceedings of the 5th International Symposium on Foundations of Information and Knowledge Systems (FoIKS). 96-111.
13. DONMEZ P., CARBONELL, J. G., AND BENNETT, P. N. 2007. Dual strategy active learning. In Proceedings of the European Conference on Machine Learning (ECML). 1-12.
14. DUDA, R. O. AND HART, P. E. 1973. Pattern Classification and Scene Analysis. Wiley, New York.
15. DAGAN, I. AND ENGELSON, S. P. 1995. Committee-Based sampling for training probabilistic classifiers. In Proceedings of the International Conference on Machine Learning. 150-157.
16. FREUND, Y., SEUNG, H. S., SHAMIR, E., AND TISHBY, N. 1997. Selective sampling using the query by committee algorithm. Mach. Learn. 28, 2, 133-168.
17. HOVY, E. H., MARCUS, M., PALMER, M.,RAMSHAW, L., ANDWEISCHEDEL, R. 2006. Ontonotes: The 90% solution. In Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL. 57-60.
18. HWA, R. 2000. Sample selection for statistical grammar induction. In Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora. 45-52.
19. KORNER, C. AND WROBEL, S. 2006. Multi-Class ensemble-based active learning. In Proceedings of the European Conference on Machine Learning (ECML). 687-694.
20. KIM, S. M. 2006. Identification, classification, and analysis of opinions on the Web. Ph.D. thesis, University of Southern California.
21. JONES, R. 2005. Learning to extract entities from labeled and unlabeled text. Ph.D. thesis, Carnegie Mellon University.
22. LAWS, F. AND SCḦUTZE, H. 2008. Stopping criteria for active learning of named entity recognition. In Proceedings of the 22nd International Conference on Computational Linguistics. 465-472.
23. LEE, Y. AND NG, H. 2002. An empirical evaluation of knowledge sources and learning algorithm forword sense disambiguation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 41-48.
24. LEWIS, D. AND GALE, W. 1994. A sequential algorithm for training text classifiers. In Proceedings of 17th ACM International Conference on Research and Development in Information Retrieval. 3-12.
25. LI, M. AND SETHI, I. K. 2006. Confidence-Based active learning. IEEE Trans. Patt. Anal. Mach. Intell. 28, 8, 1251-1261.
26. MARCUS, M., SANTORINI, B., AND MARCINKIEWICZ, M. 1993. Building a large annotated corpus of English: the Penn treebank. Comput. Linguist. 19, 2, 313-330.
27. MCCALLUM, A. AND NIGRAM, K. 1998a. Employing EM in pool-based active learning for text classification. In Proceedings of 15th International Conference on Machine Learning. 350-358.
28. MCCALLUM, A. AND NIGRAM, K. 1998b. A comparison of event models for näIve Bayes text classification. In Proceedings of AAAI-98 Workshop on Learning for Text Categorization.
29. NG, H. AND LEE, H. 1996. Integrating multiple knowledge sources to disambiguate word sense: An exemplar-based approach. In Proceedings of the 34th Annual Meeting of the Association of Computational Linguistics. 40-47.
30. NGAI, G. AND YAROWSKY, D. 2000. Rule writing or annotation: Cost-Efficient resource usage for based noun phrase chunking. In Proceedings of the 38th Annual Meeting of the Association of Computational Linguistics. 117-125.
31. PHILPOT, A., HOVY, E. H., AND PANTEL, P. 2005. The Omega ontology. In Proceedings of OntoLex Conference on Ontologies and Lexical Resources. 59-66.
32. PLATT, J. 1999. Probabilistic outputs for support vector machines and comparison to regularized likelihood methods. In Advances in Large Classifiers. 61-74.
33. ROTH, D. AND SMALL, K. 2008. Active learning for pipeline models. In Proceedings of the National Conference on Artificial Intelligence. 683-688.
34. ROY, N. AND MCCALLUM, A. 2001. Toward optimal active learning through sampling estimation of error reduction. In Proceedings of the 18th International Conference on Machine Learning. 441-448.
35. SCHEIN, A. AND UNGAR, L. 2007. Active learning for logistic regression: An evaluation. Mach. Learn. 68, 235-265.
36. SCHOHN, G. AND COHN, D. 2000. Less is more: Active learning with support vector machines. In Proceedings of the 17th International Conference on Machine Learning. 839-846.
37. SEUNG, H. S., OPPER, M., AND SOMPOLINSKY, H. 1992. Query by committee. In Proceedings of the 5th Annual ACM Conference on Computational Learning Theory. 287-294.
38. SHEN, D., ZHANG, J., SU, J., ZHOU, G., AND TAN, C. 2004. Multi-Criteria-Based active learning for named entity recognition. In Proceedings of the 42th Annual Meeting of the Association of Computational Linguistics.
39. TANG, M., LUO, X., AND ROUKOS, S. 2002. Active learning for statistical natural language parsing. In Proceedings of the 40th Annual Meeting of the Association of Computational Linguistics. 120-127.
40. TONG, S. AND KOLLER, D. 2002. Support vector machine active learning with applications to text classification. J. Mach. Learn. Res. 2, 45-66.
41. TOMANEK, K.,WERMTER, J., ANDHAHN,U. 2007. An approach to text corpus construction which cuts annotation costs and maintains reusability of annotated data. In Proceedings of the Joint Meeting of the Conference on Empirical Methods on Natural Language Processing and the Conference on Natural Language Learning. 486-495.
42. THOMPSON, C. A.,CALIFF, M. E., ANDMOONEY, R. J. 1999. Active learning for natural language parsing and information extraction. In Proceedings of the 16th International Conference on Machine Learning. 406-414.
43. VLACHOS, A. 2008. A stopping criterion for active learning. Comput. Speech Lang. 22, 3, 295-312.
44. WIEBE, J., BRECK, E., BUCKLEY, C., CARDIE, C., DAVIS, P., ET AL. 2003. Recognizing and organizing opinions expressed in the world press. In Proceedings of the AAAI Spring Symposium on New Directions in Question Answering.
45. ZHU, J. AND HOVY, E. H. 2007. Active learning for word sense disambiguation with methods for addressing the class imbalance problem. In Proceedings of the Joint Meeting of the Conference on Empirical Methods on Natural Language Processing and the Conference on Natural Language Learning. 783-790.
46. ZHU, J., WANG, H., AND HOVY, E. H. 2008a. Learning a stopping criterion for active learning for word sense disambiguation and text classification. In Proceedings of the 3rd International Joint Conference on Natural Language Processing. 366-372.
47. ZHU, J.,WANG, H., AND HOVY, E. H. 2008b. Multi-Criteria-Based strategy to stop active learning for data annotation. In Proceedings of the 22nd International Conference on Computational Linguistics. 1129-1136.