Collecting high quality overlapping labels at low cost

SIGIR 2010 Proceedings - 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2010, Pages 459-466

  Yang, Hui ^a   Mityagin, Anton ^b   Svore, Krysta M ^c   Markov, Sergey ^b  

^a Carnegie Mellon University   (United States)

^b MICROSOFT   (United States)

^c MICROSOFT RESEARCH   (United States)

Author keywords

Learning to rank; Overlapping labels; Relevance labels

Indexed keywords

HIGH QUALITY; IS PERFORMANCE; LABELING SCHEME; LEARNING TO RANK; LOW COSTS; MAJORITY VOTE; OVERLAPPING LABELS; REAL-WORLD TESTS; RELEVANCE LABELS; RETRIEVAL ACCURACY; SELECTIVE LABELING; TRAINING SAMPLE; WEB SEARCHES;

SEARCH ENGINES; WORLD WIDE WEB;

INFORMATION RETRIEVAL;

EID: 77956037060     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1835449.1835526     Document Type: Conference Paper

References (11)

1. A. Bernstein and J. Li. From active towards interactive learning: Using consideration information to improve labeling correctness. University of Zurich, Dynamic and distributed information systems group working paper. www.ifi.uzh.ch/ddis/nc/publications.
2. C. J. C. Burges, R. Ragno, and Q. V. Le. Learning to rank with nonsmooth cost functions. NIPS'06.
3. Kalervo Järvelin and Jaana Kekäläinen. Ir evaluation methods for retrieving highly relevant documents. SIGIR'00.
4. NAACL-HLT 2010 Workshop on Amazon Mechanical Turk. http://sites.google. com/site/amtworkshop2010/.
5. V. S. Sheng, F. Provost, and P. G. Lpeirotis. Get another label? Improving data quality and data mining using multiple noisy labelers. KDD'08.
6. P. Smyth, U. M. Fayyad, M. C. Burl, P. Perona, and P. Baldi. Inferring ground truth from subjective labeling of Venus images. NIPS'94.
7. P. Smyth, M. C. Burl, U. M. Fayyad, P. Perona. Knowledge discovery in large image databases: Dealing with uncertainties in ground truth. AAAI Knowledge Discovery in Databases Workshop of KDD'94.
8. P. Smyth. Bounds on the mean classification error rate of multiple experts. Pattern Recognition Letters 17, 12. 1996.
9. R. Snow, B. O'Connor, D. Jurafsky and A. Y. Ng. 2008. Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. EMNLP'08.
10. E. M. Voorhees. Evaluating by highly relevant documents. SIGIR'01.
11. Q. Wu, C. J. C. Burges, K. M. Svore and J. Gao. Ranking, Boosting, and Model Adaptation. Microsoft Research Technical Report MSR-TR-2008-109.