Crowdsourcing Multi-Label Classification for Taxonomy Creation

Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2013

Volumn , Issue , 2013, Pages 25-33

  Bragg, Jonathan ^a   Mausam, ^a   Weld, Daniel S ^a  

^a University of Washington   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; DECISION THEORY; LEARNING SYSTEMS; TAXONOMIES;

CROWD LABORS; HUMAN EXPERT; INFORMATION GAIN; MACHINE-LEARNING; MICROTASKS; MULTI-LABEL CLASSIFICATIONS; WORK-FLOWS;

CROWDSOURCING;

EID: 85114255699     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (23)

1. Chi, E. H., and Mytkowicz, T. 2008. Understanding the efficiency of social tagging systems using information theory. In Proceedings of the Nineteenth ACM Conference on Hypertext and Hypermedia (HT’08), 81–88.
2. Chilton, L. B.; Little, G.; Edge, D.; Weld, D. S.; and Landay, J. A. 2013. Cascade: Crowdsourcing taxonomy creation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’13).
3. Dai, P.; Lin, C. H.; Mausam; and Weld, D. S. 2013. POMDP-based control of workflows for crowdsourcing. Artificial Intelligence 202(0):52–85.
4. Donmez, P.; Carbonell, J. G.; and Schneider, J. 2010. A probabilistic framework to learn from multiple annotators with time-varying accuracy. In Proceedings of the SIAM International Conference on Data Mining (SDM’10), 826–837.
5. Golder, S. A., and Huberman, B. A. 2006. Usage patterns of collaborative tagging systems. Journal of Information Science 32(2):198–208.
6. Johnson, N. L., and Kotz, S. 1977. Urn models and their application: an approach to modern discrete probability theory. Wiley New York.
7. Kamar, E.; Hacker, S.; and Horvitz, E. 2012. Combining human and machine intelligence in large-scale crowdsourcing. In Proceedings of the Eleventh International Conference on Autonomous Agents and Multiagent Systems (AAMAS’12).
8. Krause, A., and Guestrin, C. 2005. Near-optimal nonmyopic value of information in graphical models. In Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence (UAI’05).
9. Lin, C.; Mausam; and Weld, D. S. 2012a. Dynamically switching between synergistic workflows for crowdsourcing. In Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence (AAAI’12).
10. Lin, C. H.; Mausam; and Weld, D. S. 2012b. Crowdsourcing control: Moving beyond multiple choice. In Proceedings of the Twenty-Eighth Conference on Uncertainty in Artificial Intelligence (UAI’12).
11. Ling, X., and Weld, D. S. 2012. Fine-grained entity resolution. In Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence (AAAI’12).
12. Little, G.; Chilton, L. B.; Goldman, M.; and Miller, R. C. 2009. TurKit: Tools for iterative tasks on mechanical turk. In Human Computation Workshop (HCOMP’09).
13. Miller, G. 1991. WordNet: An on-line lexical database. International Journal of Lexicography 3(4):235–312.
14. Nemhauser, G. L.; Wolsey, L. A.; and Fisher, M. L. 1978. An analysis of approximations for maximizing submodular set functions. Mathematical Programming 14(1):265–294.
15. Parameswaran, A. G.; Sarma, A. D.; Garcia-Molina, H.; Polyzotis, N.; and Widom, J. 2011. Human-assisted graph search: It’s okay to ask questions. Proceedings of the VLDB Endowment 4(5):267–278.
16. Raykar, V. C.; Yu, S.; Zhao, L. H.; and Valadez, G. 2010. Learning from crowds. Journal of Machine Learning Research 11:1297–1322.
17. Shahaf, D., and Horvitz, E. 2010. Generalized task markets for human and machine computation. In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence (AAAI’10).
18. Sheng, V. S.; Provost, F.; and Ipeirotis, P. G. 2008. Get another label? Improving data quality and data mining using multiple, noisy labelers. In Proceedings of the Fourteenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’08).
19. Tran-Thanh, L.; Stein, S.; Rogers, A.; and Jennings, N. R. 2012. Efficient crowdsourcing of unknown experts using multi-armed bandits. In Proceedings of the Twentieth European Conference on Artificial Intelligence (ECAI’12), 768–773.
20. Wauthier, F., and Jordan, M. 2011. Bayesian bias mitigation for crowdsourcing. In Advances in Neural Information Processing Systems 24 (NIPS’11). 1800–1808.
21. Weld, D. S.; Mausam; and Dai, P. 2011. Human Intelligence Needs Artificial Intelligence. In Human Computation Workshop (HCOMP’11).
22. Welinder, P.; Branson, S.; Belongie, S.; and Perona, P. 2010. The multidimensional wisdom of crowds. In Advances in Neural Information Processing Systems 23 (NIPS’10). 2424–2432.
23. Whitehill, J.; Ruvolo, P.; Wu, T.; Bergsma, J.; and Movellan, J. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems 22 (NIPS’09). 2035–2043.