Why label when you can search? Alternatives to active learning for applying human resources to build classification models under extreme class imbalance

Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Volumn , Issue , 2010, Pages 423-432

  Attenberg, Josh ^a   Provost, Foster ^b  

^a POLYTECHNIC UNIVERSITY   (United States)

^b NEW YORK UNIVERSITY   (United States)

Author keywords

Active learning; Class imbalance; Human resources; Machine learning; Micro outsourcing; On line advertising

Indexed keywords

ACTIVE LEARNING; CLASS IMBALANCE; HUMAN RESOURCES; MACHINE-LEARNING; MICRO-OUTSOURCING; ONLINE ADVERTISING;

CLASSIFIERS; LEARNING SYSTEMS; OUTSOURCING;

COST BENEFIT ANALYSIS;

EID: 77956208411     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1835804.1835859     Document Type: Conference Paper

References (32)

1. http://www.dmoz.org/.
2. https://www.mturk.com/mturk/welcome.
3. Content category definitions & rating guidelines. New York, NY, USA, January. AdSafe Media, Inc. http://www.adsafemedia.com/adsafe-rating-guide.php.
4. Beyond the grey areas: Transparency, brand safety and the future of online advertising. New York, NY, USA, April 2010. Winterberry Group LLC.
5. A. Asuncion and D. Newman. UCI machine learning repository, 2007.
6. J. Atteberg and F. Provost. Toward optimal allocation of human resources for active learning with application to safe advertising. In Winter Conf. on Business Intel., 2010.
7. J. Attenberg and F. Provost. Toward optimal allocation of human resources for active learning with application to safe on-line advertising. Stern working paper ceder-09-07, 2009.
8. M. Bloodgood and K. V. Shanker. Taking into account the differences between actively and passively acquired data: the case of active learning with support vector machines for imbalanced datasets. In NAACL '09, 2009.
9. N. V. Chawla, K. W. Bowyer, and P. W. Kegelmeyer. Smote: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16:321-357, 2002.
10. P. Domingos. Metacost: A general method for making classifiers cost-sensitive. In In Proc. of the Fifth Intl. Conf. on Knowledge Discovery and Data Mining, 1999.
11. P. Donmez and J. Carbonell. Paired Sampling in Density-Sensitive Active Learning. In Proc. of the 10 Intl. Symp. on Artificial Intelligence and Mathematics, 2008.
12. P. Donmez, J. G. Carbonell, and P. N. Bennett. Dual strategy active learning. In ECML '07, 2007.
13. S. Ertekin, J. Huang, L. Bottou, and L. Giles. Learning on the border: active learning in imbalanced data classification. In CIKM '07: Proc. of the sixteenth ACM Conf. on information and knowledge management, 2007.
14. T. Fawcett. ROC graphs: Notes and practical considerations for data mining researchers. Technical report hpl-2003-4, 2003.
15. D. D. Lewis and W. A. Gale. A sequential algorithm for training text classifiers. In SIGIR '94: Proc. of the 17th Annual Intl. ACM SIGIR Conf on Research and development in Info. Retreival, 1994.
16. X. Y. Liu, J. Wu, and Z. H. Zhou. Exploratory undersampling for class-imbalance learning. Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Trans. on, 39, 2009.
17. R. Lomasky, C. E. Brodley, M. Aernecke, D. Walt, and M. Friedl. Active class selection. In ECML '07: Proc. of the 18th European conference on Machine Learning, pages 640-647, Berlin, Heidelberg, 2007. Springer-Verlag.
18. J. Ma, L. K. Saul, S. Savage, and G. M. Voelker. Beyond blacklists: learning to detect malicious web sites from suspicious urls. In KDD '09, 2009.
19. H. T. Nguyen and A. Smeulders. Active learning using pre-clustering. In ICML '04: Proc. of the 21st Intl. Conf. on Machine learning, New York, NY, USA, 2004.
20. F. J. Provost, D. Jensen, and T. Oates. Efficient progressive sampling. In KDD'99, 1999.
21. S. Rajan, D. Yankov, S. Gaffney, and A. Ratnaparkhi. A large-scale active learning system for topical categorization on the web. In WWW, 2010.
22. B. Settles. Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin-Madison, 2009.
23. H. S. Seung, M. Opper, and H. Sompolinsky. Query by committee. In COLT '92, 1992.
24. V. S. Sheng, F. Provost, and P. G. Ipeirotis. Get another label? improving data quality and data mining using multiple, noisy labelers. In KDD '08: Proc. of the 14th ACM SIGKDD Intl. Conf. on Knowledge discovery and data mining, 2008.
25. K. Tomanek and U. Hahn. Reducing class imbalance during active learning for named entity annotation. In K-CAP '09: Proc. of the 5th Intl. Conf. on Knowledge capture, 2009.
26. S. Tong and D. Koller. Support vector machine active learning with applications to text classification. J. Mach. Learn. Res., 2:45-66, 2002.
27. K. Weinberger, A. Dasgupta, J. Attenberg, J. Langford, and A. Smola. Feature hashing for large scale multitask learning. In ICML '09, 2009.
28. G. Weiss and F. Provost. The effect of class distribution on classifier learning. Rutgers technical report ml-tr-44, 2001.
29. G. M. Weiss. The impact of small disjuncts on classifier learning. In Data Mining, volume 8, pages 193-226. 2010.
30. G. M. Weiss and F. Provost. Learning when training data are costly: the effect of class distribution on tree induction. J. Artif. Int. Res., 19(1):315-354, 2003.
31. J. Zhu and E. Hovy. Active learning for word sense disambiguation with methods for addressing the class imbalance problem. In EMNLP-CoNLL '07.
32. J. Zhu, H. Wang, T. Yao, and B. K. Tsou. Active learning with sampling by uncertainty and density for word sense disambiguation and text classification. In COLING '08, 2008.