Imbalanced multiple noisy labeling

IEEE Transactions on Knowledge and Data Engineering

Volumn 27, Issue 2, 2015, Pages 489-503

  Zhang, Jing ^a   Wu, Xindong ^a,b   Sheng, Victor S ^c  

^a HEFEI UNIVERSITY OF TECHNOLOGY   (China)

^b University of Vermont   (United States)

^c UNIVERSITY OF CENTRAL ARKANSAS   (United States)

Author keywords

crowdsourcing; imbalanced learning; Imbalanced noisy labeling; repeated labeling

Indexed keywords

INFERENCE ENGINES; INTERNET; VIRTUAL REALITY;

AMAZON MECHANICAL TURKS; CLASS DISTRIBUTIONS; CROWDSOURCING; FREQUENCY THRESHOLD; IMBALANCED LEARNING; INFERENCE ALGORITHM; INTERNET BASED; PERSONAL PREFERENCES;

CLASSIFICATION (OF INFORMATION);

EID: 84920118410     PISSN: 10414347     EISSN: None     Source Type: Journal    
DOI: 10.1109/TKDE.2014.2327039     Document Type: Article

References (34)

1. C. L. Black and C. J. Merz. UCI repository of machine learning database [Online]. Available: http://archive.ics.uci.edu/ml/, 1998.
2. N. V. Chawla, L. O. Hall, K. W. Bowyer, and W. P. Kegelmeyer, "SMOTE: Synthetic minority oversampling technique," J. Artif. Intell. Res., vol. 16, pp. 321-357, 2002.
3. A. P. Dawid and A. M. Skene, "Maximum likelihood estimation of observer error-rates using the EM algorithm," Appl. Stat., vol. 28, no. 1, pp. 20-28, 1979.
4. P. Donmez, J. G. Carbonell, and J. Schneider, "Efficiently learning the accuracy of labeling sources for selective sampling," in Proc. 15th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2009, pp. 259-268.
5. A. Estabrooks, T. Jo, and N. Japkowicz, "A multiple resampling method for learning from imbalanced data sets," Comput. Intell., vol. 20, no. 1, pp. 18-36, 2004.
6. A. Gelman, J. B. Carlin, H. S. Stern, and D. B. Rubin, Bayesian Data Analysis, 2nd ed. Boca Raton, FL, USA: CRC, 2003.
7. Y. Fu, X. Zhu, and B. Li, "A survey on instance selection for active learning," Knowl. Inform. Syst., vol. 35, no. 2, pp. 249-283, 2013.
8. H. He and E. A. Garcia, "Learning from imbalanced data," IEEE Trans. Knowl. Data Eng., vol. 21, no. 9, pp. 1263-1284, Sep. 2009.
9. H. Huang, Q. He, K. Chiew, F. Qian, and L. Ma, "CLOVER: A faster prior-free approach to rare-category detection," Knowl. Inform. Syst., vol. 35, no. 3, pp. 713-736, 2013.
10. P. Ipeirotis, F. Provost, and J. Wang, "Quality management on amazon mechanical turk," in Proc. 16th ACM SIGKDD Workshop Human Comput., 2010, pp. 64-67.
11. H., Kajino, Y. Tsuboi, and H. Kashima, "A convex formulation for learning from crowds," in Proc. 26th AAAI Conf. Artif. Intell., 2012, pp. 73-79.
12. A. Kumar and M. Lease, "Modeling annotator accuracies for supervised learning," in Proc. 4th ACM WSDM Workshop Crowdsourcing Search Data Mining, 2011, pp. 19-22.
13. C. X. Ling and V. S. Sheng, "Cost-sensitive learning," in Encyclopedia of Machine Learning. Berlin, Germany: Springer, 2010, pp. 231-235.
14. X. Y. Liu, J. Wu, and Z. H. Zhou, "Exploratory under sampling for class imbalance learning," in Proc. IEEE 6th Int. Conf. Data Mining, 2006, pp. 965-969.
15. H. Y. Lo, S. D. Lin, and H. M. Wang, "Generalized k-labelsets ensemble for multi-label and cost-sensitive classification," IEEE Trans. Knowl. Data Eng., vol. 26, no. 7, pp. 1679-1691, 2014.
16. H. Y. Lo, J. C. Wang, H. M., Wang, and S. D., Lin, "Cost-sensitive multi-label learning for audio tag annotation and retrieval," IEEE Trans. Multimedia, vol. 13, no. 3, pp. 518-529, Jun. 2011.
17. M. I. Mandel and D. P. W. Ellis, "A web-based game for collecting music metadata," J. New Music Res., vol. 37, no. 2, pp. 151-165, 2008.
18. C. Parker, "On measuring the performance of binary classifiers," Knowl. Inform. Syst., vol. 35, no. 1, pp. 131-152, 2013.
19. J. R. Quinlan, C4.5: Programs for Machine Learning. San Mateo, CA, USA: Morgan Kaufmann, 1992.
20. V. C. Raykar, S. Yu, L. H. Zhao, C. Florin, G. H. Valadez, L. Bogoni, and L. Moy, "Learning from crowds," J. Mach. Learn. Res., vol. 11, pp. 1297-1322, 2010.
21. V. S. Sheng, "Simple multiple noisy label utilization strategies," in Proc. IEEE 11th Int. Conf. Data Mining, 2011, pp. 635-644.
22. V. S. Sheng, F. Provost, and P. Ipeirotis, "Get another label? Improving data quality and data mining using multiple, nosiy labeler," in Proc. 14th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2008, pp. 614-662.
23. P. Smyth, M. C. Burl, U. M. Fayyad, and P. Perona, "Knowledge discovery in large image database: Dealing with uncertainties in ground truth," in Proc. Knowl. Discovery Database: Papers AAAI Workshop, 1994, pp. 109-120.
24. P. Smyth, M. C. Burl, U. M. Fayyad, P. Perona, and P. Baldi, "Inferring ground truth from subjective labeling of venus images," Adv. Neural Inform. Process. Syst., vol. 8, pp. 1085-1092, 1995.
25. R. Snow, B. O'Connor, D. Jurafsky, and A. Ng, "Cheap and fast-But is it good?" in Proc. Conf. Empirical Methods Natural Lang. Process., 2008, pp. 254-263.
26. C. Strapparava and R. Mihalcea, "SemEval-2007 Task 14: Affective text," in Proc. 4th Int. Workshop Semantic Eval., 2007, pp. 70-74.
27. K. M. Ting, "An instance-weighting method to induce costsensitive trees," IEEE Trans. Knowl. Data Eng., vol. 14, no. 3, pp. 659-665, May 2002.
28. L. Von Ahn and L. Dabbish, "Labeling images with a computer game," in Proc. ACM Conf. Human Factors Comput. Syst., 2004, pp. 319-326.
29. P. Welinder and P. Perona, "Online crowdsourcing: Rating annotators and obtaining cost-effective labels," in Proc. Workshop Adv. Comput. Vis. Humans Loop, 2010, pp. 25-32.
30. J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. Movellan, "Whose vote should count more: Optimal integration of labels from labelers of unknown expertise," in Proc. Adv. Neural Info. Process. Syst. 22, 2009, pp. 2035-2043.
31. I. H. Witten and E. Frank, Data Mining: Practical Machine Learning Tools and Techniques, 2nd ed. San Mateo, CA, USA: Morgan Kaufman, Jun. 2005.
32. Y. Yan, R. Rosales, G. Fung, and J. G. Dy, "Modeling multiple annotator expertise in the semi-supervised learning scenario," in Proc. 26th Conf. Uncertainity Artif. Intell., 2010, pp. 674-682.
33. Y. Yan, R. Rosales, G. Fung, and J. G. Dy, "Active learning from crowds," in Proc. 28th Int. Conf. Mach. Learn., 2011, pp. 1161-1168.
34. B. Zadrozny, J. Langford, and N. Abe, "Cost-sensitive learning by cost-proportionate example weighting," in Proc. 3rd IEEE Int. Conf. Data Mining, 2003, pp. 435-442.