Relation between agreement measures on human labeling and machine learning performance: Results from an art history domain

Proceedings of the 6th International Conference on Language Resources and Evaluation, LREC 2008

Volumn , Issue , 2008, Pages 2841-2848

  Passonneau, Rebecca J ^a   Lippincott, Tom ^a   Yano, Tae ^b   Klavans, Judith ^c  

^a Och Spine at New York Presbyterian Hospitals   (United States)

^b CARNEGIE MELLON UNIVERSITY   (United States)

^c UNIVERSITY OF MARYLAND   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; SEMANTICS;

ART HISTORY; BINARY CLASSIFIERS; CORPUS ANNOTATIONS; FUNCTIONAL SEMANTICS; LABELED DATA; LEARNING PERFORMANCE; SEMANTIC LABELING; TEST DATA;

LEARNING SYSTEMS;

EID: 84880379979     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (20)

1. Argamon, S., Whitelaw, C., Chase, P., Hota, S. R., Garg, N., and Levitan, S. (2007) Stylistic text classification using functional lexical features: Research Articles. J. Am. Soc. Inf. Sci. Technol. 58, 6(Apr. 2007), 802-822.
2. Artstein, R. and M. Poesio. (2005) Kappa3 = Alpha (or Beta). Technical Report NLE Technote 2005-01, University of Essex, Essex, 2005.
3. Baca, M. (2003) Practical Issues in Applying Metadata Schemas and Controlled Vocabularies to Cultural Heritage Information. The Haworth Press, Inc.
4. Chen, H. (2001) An analysis of image queries in the field of art history. Journal of the American Society for Information Science and Technology, pages 260-273.
5. Hachey, B. and C. Grover. (2004) Sentence classification experiments for legal text summarisation. In Proceedings of the 17th Annual Conference on Legal Knowledge and Information Systems (Jurix).
6. Forman, G. (2003) An extensive empirical study of feature selection metrics for text classification. Journal of Machine Learning. Research 3(Mar. 2003), 1289-1305.
7. Giral, A. and A. Taylor. (1993) Indexing overlap and consistency between the Avery Index to Architectural Periodicals and the Architectural Periodicals Index. Library Resources and Technical Services 37(1):19-44.
8. Jaccard, P. (1908). Nouvelles recherches sur la distribution florale. Bulletin de la Societe Vaudoise des Sciences Naturelles 44:223-270.
9. Klavans, J.; Sidhu, T.; Sheffield, C.; Soergel, D.; Lin, J.; Abels, E.; Passonneau, R. (2008) Computational Linguistics for Metadata Building (CLiMB) Text Mining for the Automatic Extraction of Subject Terms for Image Metadata. International Conference on Computer Vision Theory and Applications, Workshop 3: Metadata Mining for Image Understanding, Funchal, Portugal.
10. Krippendorff, K. (1980). Content Analysis: An Introduction to Its Methodology. Sage Publications, Beverly Hills, CA.
11. Layne, S. S. (1994) Some issues in the indexing of images. Journal of the American Society for Information Science, pages 583-8.
12. Markey, K. (1984) Interindexer consistency tests: a literature review and report of a test of consistency in indexing visual materials. Library and Information Science Research, pages 155-177.
13. Passonneau, R; Yano, T.; Lippincott, T.; Klavans, J. (2008). Functional semantic categories for art history text: human labeling and preliminary machine learning. International Conference on Computer Vision Theory and Applications, Workshop 3: Metadata Mining for Image Understanding, Funchal, Portugal.
14. Passonneau, R. (2006). Measuring agreement on setvalued items (MASI) for semantic and pragmatic annotation. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC). Portugal.
15. Passonneau, R.; Habash, N.; Rambow, O. (2006) Interannotatator agreement on a multilingual semantic annotation task. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC). Portugal.
16. Riedsma, D. and Carletta, J. (Forthcoming) Reliability measurement: there's no safe limit To appear in Computational Linguistics.
17. Sebastiani, F. (2002) Machine learning in automated text categorization. ACM Computing Surveys, 34(1):1-47, March 2002.
18. Teufel, S. and M. Moens. (2002) Summarising scientific articles - experiments with relevance and rhetorical status. Computational Linguistics, pages 409-445.
19. Witten, I. H. and E. Frank (2005) Data Mining: Practical Machine Learning Tools and Techniques, 2nd Edition. Morgan Kaufmann, San Francisco, 2005.
20. th International Conference on Machine Learning (ICML), pp. 412-420.