Uncertainty-based active learning with instability estimation for text classification

ACM Transactions on Speech and Language Processing

Volumn 8, Issue 4, 2012, Pages

  Zhu, Jingbo ^a   Ma, Matthew ^b  

^a NORTHEASTERN UNIVERSITY   (China)

^b Scientific Works   (United States)

Author keywords

Active learning; Data annotation; Instability estimation; Text classification; Uncertainty sampling

Indexed keywords

ACTIVE LEARNING; DATA ANNOTATION; DATA SETS; DECISION BOUNDARY; DENSITY METHODS; DIRECTLY MODEL; ENTROPY-BASED; LEARNING PROCESS; POOL-BASED; POSTERIOR PROBABILITY; RANDOM SAMPLING; SAMPLING METHOD; SELECTED EXAMPLES; SELECTIVE SAMPLING; SIGMOIDAL FUNCTIONS; TEXT CLASSIFICATION;

HEURISTIC METHODS; LEARNING ALGORITHMS; STABILITY; UNCERTAINTY ANALYSIS;

LEARNING SYSTEMS;

EID: 84863267844     PISSN: 15504875     EISSN: 15504883     Source Type: Journal    
DOI: 10.1145/2093153.2093154     Document Type: Article

References (48)

1. YORAM, B., EL-YANIV, R., AND LUZ, K. 2004. Online choice of active learning algorithms. J. Mach. Learn. Res., 5, 255-291.
2. BALDRIDGE, J. AND PALMER, A. 2009. How well does active learning actually work? Time-Based evaluation of cost-reduction strategies for language documentation. In Proceedings of the ACL'09 Conference.
3. BECKER, M. AND OSBORNE, M. 2005. A two-stage method for active learning of statistical grammars. In Proceedings of the IJCAI'05 Conference. 991-996
4. BERGER, A. L., DELLA PIETRA, V. J., AND DELLA PIETRA, S. A. 1996. A maximum entropy approach to natural language processing. Comput. Linguistics. 22, 1, 39-71.
5. BLOODGOOD, M. AND VIJAY-SHANKER, K. 2009. Amethod for stopping active learning based on stabilizing predictions and the need for user-adjustable stopping. In Proceedings of the 13th Conference on Computational Natural Language Learning (CoNLL), 39-47.
6. BRUCE, R. AND WIEBE, J. 1994. Word-Sense disambiguation using decomposable models. In Proceedings of the 32nd Annual Meeting of the Association for Computational Linguistics. 139-146.
7. CHAN, Y. S. AND NG, H. T. 2007. Domain adaptation with active learning for word sense disambiguation. In Proceedings of the 45th Annual Meeting on Association for Computational Linguistics. 49-56.
8. CHEN, J., SCHEIN, A., UNGAR, L., AND PALMER, M. 2006. An empirical study of the behavior of active learning for word sense disambiguation. In Proceedings of the Main Conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics. 120-127.
9. COHN, D., ATLAS, L., AND LADNER, R. 1994. Improving generalization with active learning. Mach. Learn. 15, 2, 201-221.
10. COHN, D. A., GHAHRAMANI, Z., AND JORDAN, M. I. 1996. Active learning with statistical models. J. Artif. Intell. Res. 4, 129-145. (Pubitemid 126646151)
11. CULOTTA, A. AND MCCALLUM, A. 2005. Reducing labeling effort for structured prediction tasks. In Proceedings of AAAI'05 Conference. 746-751 (Pubitemid 43006698)
12. HAERTEL, R., RINGGER, E., SEPPI, K., CARROLL, J., AND MCCLANAHAN, P. 2008a. Assessing the costs of sampling methods in active learning for annotation. In Proceedings of the ACL08 Conference. 65-68
13. HAERTEL, R. A.,SEPPI, K. D.,RINGGER, E. K., AND CARROLL, J. L. 2008b. Return on investment for active learning. In Proceedings of NIPS'08 Workshop on Cost-Sensitive Learning.
14. HAFFARI, G. AND SARKAR, A. 2009. Active learning for multilingual statistical machine translation. In Proceedings of the ACL'09 Conference. 181-189.
15. HWA, R. 2000. Sample selection for statistical grammar induction. In Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora. 45-52.
16. JONES, R. 2005. Learning to extract entities from labeled and unlabeled text. PhD thesis, Carnegie Mellon University.
17. KIM, S. M. 2006. Identification, classification, and analysis of opinions on the web. PhD thesis, University of Southern California.
18. LEACOCK, C., TOWELL, G., AND VOORHEES, E. 1993. Corpus-Based statistical sense resolution. In Proceedings of the ARPA Workshop on Human Language Technology. 260-265.
19. LEE, Y. AND NG, H. 2002. An empirical evaluation of knowledge sources and learning algorithm for word sense disambiguation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 41-48.
20. LEWIS, D. D. AND GALE, W. A. 1994. A sequential algorithm for training text classifiers. In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 3-12.
21. MCCALLUM, A. AND NIGAM, K. 1998a. Employing EM in pool-based active learning for text classification. In Proceedings of 15th International Conference on Machine Learning. 350-358.
22. MCCALLUM, A. AND NIGAM, K. 1998b. A comparison of event models for naïve bayes text classification. In Proceedings of the AAAI'98 Workshop on Learning for Text Categorization.
23. MILGRAM, J., CHERIET, M., AND SABOURIN, R. 2006. One against one or one against all: Which one is better for handwriting recognition with SVMs? In Proceedings of the 10th International Workshop on Frontiers in Handwriting Recognition.
24. NG, H. AND LEE, H. 1996. Integrating multiple knowledge sources to disambiguate word sense: An exemplarbased approach. In Proceedings of the 34th Annual Meeting of the Association of Computational Linguistics. 40-47.
25. NGAI, G. AND YAROWSKY, D. 2000. Rule writing or annotation: Cost-Efficient resource usage for based noun phrase chunking. In Proceedings of the 38th Annual Meeting of the Association of Computational Linguistics.
26. NGUYEN, H. AND SMEULDERS, A. 2004. Active learning with pre-clustering. In Proceedings of the International Conference on Machine Learning (ICML'04). 623-630.
27. PLATT, J. C. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers. MIT Press.
28. RINGGER, E., MCCLANAHAN, P., HAERTEL, R., BUSBY, G., CARMEN, M., CARROLL, J., SEPPI, K., AND LONSDALE, D. 2007. Active learning for part-of-speech tagging: Accelerating corpus annotation. In Proceedings of the ACL'07 Linguistic Annotation Workshop (LAW).
29. ROTH, D. AND SMALL, K. 2006. Margin-Based active learning for structured output spaces. In Proceedings of the ECML'06 Conference.
30. NICHOLAS, R. AND MCCALLUM, A. 2001. Toward optimal active learning through sampling estimation of error reduction. In Proceedings of the 18th International Conference on Machine Learning. 441-448.
31. SCHAPIRE, R. E., FREUND, Y.,BARTLETT, P., AND LEE,W. S. 1997. Boosting the margin: A new explanation for the effectiveness of voting methods. In Proceedings of the 14th International Conference Machine Learning. 322-330.
32. SCHEIN, A. I. AND UNGAR, L. H. 2007. Active learning for logistic regression: An evaluation. Mach. Learn. 68, 3, 235-265. (Pubitemid 47312489)
33. SETTLES, B. 2009. Active learning literature survey. Computer Science Tech. rep. 1648, University of Wisconsin-Madison.
34. SETTLES, B. AND CRAVEN, M. 2008. An analysis of active learning strategies for sequence labeling tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 1070-1079.
35. SEUNG, H. S., OPPER, M., AND SOMPOLINSKY, H. 1992. Query by committee. In Proceedings of the 5th Annual ACM Conference on Computational Learning Theory. 287-294. (Pubitemid 23615454)
36. SHEN, D.,ZHANG, J.,SU, J.,ZHOU, G., AND TAN, C.-L. 2004. Multi-Criteria-Based active learning for named entity recognition. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics. 589-596.
37. TANG, M., LUO, X., AND ROUKOS, S. 2002. Active learning for statistical natural language parsing. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. 120-127.
38. THOMPSON, C. A., CALIFF, M. E., AND MOONEY, R. J. 1999. Active learning for natural language parsing and information extraction. In Proceedings of the 16th International Conference on Machine Learning. 406-414.
39. TOMANEK, K.,WERMTER, J., AND HAHN, U. 2007. An approach to text corpus construction which cuts annotation costs and maintains reusability of annotated data. In Proceedings of the Joint Meeting of the Conference on Empirical Methods on Natural Language Processing and the Conference on Natural Language Learning. 486-495.
40. TONG S. AND KOLLER, D. 2001. Support vector machine active learning with applications to text classification. J. Mach. Learn. Res. 45-66.
41. VLACHOS, A. 2008. A stopping criterion for active learning. Comput. Speech Lang. 22, 3, 295-312. (Pubitemid 351329454)
42. WIEBE, J., BRECK, E., AND BUCKLEY, C. 2003. Recognizing and organizing opinions expressed in the world press. In Proceedings of the AAAI Spring Symposium on New Directions in Question Answering.
43. YU, D.,VARADARAJAN, B.,DENG, L., AND ACERO, A. 2010. Active learning and semi-supervised learning for speech recognition: A unified framework using the global entropy reduction maximization criterion. Comput. Speech Lang., 24, 3, 433-444.
44. ZHU, J. AND HOVY, E. 2007. Active learning for word sense disambiguation with methods for addressing the class imbalance problem. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. 783-790.
45. ZHU, J.,WANG, H., YAO, T., AND TSOU, B. 2008a. Active learning with sampling by uncertainty and density for word sense disambiguation and text classification. In Proceedings of the 22nd International Conference on Computational Linguistics. 1137-1144.
46. ZHU, J.,WANG, H., AND HOVY, E. 2008b. Multi-Criteria-Based strategy to stop active learning for data annotation. In Proceedings of the 22nd International Conference on Computational Linguistics. 1129-1136.
47. ZHU, J.,WANG, H.,HOVY, E., AND MA, M. 2010. Confidence-Based stopping criteria for active learning for data annotation. ACM Trans. Speech Lang. Process. 6, 3, 1-24.
48. ZHU, J., WANG, H., AND TSOU, B. K. 2009. A density-based re-ranking technique for active learning for data annotation. In Proceedings of the 22nd International Conference on the Computer Processing of Oriental Languages (ICCPOL). 1-10.