Social book search: Comparing topical relevance judgements and book suggestions for evaluation

ACM International Conference Proceeding Series

Volumn , Issue , 2012, Pages 185-194

  Koolen, Marijn ^a   Kamps, Jaap ^a   Kazai, Gabriella ^b  

^a UNIVERSITY OF AMSTERDAM   (Netherlands)

^b MICROSOFT RESEARCH   (United Kingdom)

Author keywords

book search; evaluation; user generated content

Indexed keywords

BOOK SEARCH; CLASSICAL MODEL; DISCUSSION FORUM; EVALUATION; EXPERIMENTAL EVALUATION; INFORMATION NEED; MECHANICAL TURKS; RELEVANCE JUDGEMENT; RETRIEVAL SYSTEMS; SEARCH SYSTEM; SEARCH TASKS; SOCIAL MEDIA; SOCIAL SEARCH; SYSTEM EVALUATION; SYSTEM RANKINGS; USER-GENERATED CONTENT; WEALTH OF INFORMATION;

KNOWLEDGE MANAGEMENT; METADATA;

SEARCH ENGINES;

EID: 84871084096     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2396761.2396788     Document Type: Conference Paper

References (29)

1. O. Alonso and R. A. Baeza-Yates. Design and Implementation of Relevance Assessments Using Crowdsourcing. In ECIR 2011, volume 6611 of LNCS, pages 153-164. Springer, 2011.
2. M. J. Bates. Task Force Recommendation 2.3 Research and Design Review: Improving user access to library catalog and portal information. In Library of Congress Bicentennial Conference on Bibliographic Control for the New Millennium, 2003.
3. T. Beckers, N. Fuhr, N. Pharo, R. Nordlie, and K. N. Fachry. Overview and Results of the INEX 2009 Interactive Track. In ECDL, volume 6273 of LNCS, pages 409-412. Springer, 2010.
4. M. Buckland. Vocabulary as a Central Concept in Library and Information Science. In Digital Libraries: Interdisciplinary Concepts, Challenges, and Opportunities. CoLIS3, 1999.
5. C. Buckley, D. Dimmick, I. Soboroff, and E. Voorhees. Bias and the limits of pooling for large collections. Inf. Retr., 10(6):491-508, 2007.
6. C. L. Clarke, M. Kolla, G. V. Cormack, O. Vechtomova, A. Ashkan, S. Büttcher, and I. MacKinnon. Novelty and diversity in information retrieval evaluation. In SIGIR '08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pages 659-666, 2008. ACM.
7. C. W. Cleverdon. The Cranfield tests on index language devices. Aslib, 19:173-192, 1967.
8. S. A. Golder and B. A. Huberman. Usage patterns of collaborative tagging systems. Journal of Information Science, 32(2):198-208, 2006.
9. C. Grady and M. Lease. Crowdsourcing document relevance assessment with mechanical turk. In CSLDAMT '10, pages 172-179, 2010.
10. D. Hawking and N. Craswell. Very large scale retrieval and web search. In TREC: Experiment and Evaluation in Information Retrieval, chapter 9. MIT Press, 2005.
11. G. Kazai. In Search of Quality in Crowdsourcing for Search Engine Evaluation. In ECIR 2011, volume 6611 of LNCS, pages 165-176. Springer, 2011.
12. G. Kazai and N. Milic-Frayling. Effects of Social Approval Votes on Search Performance. Information Technology: New Generations, Third International Conference on, 0:1554-1559, 2009.
13. G. Kazai, J. Kamps, M. Koolen, and N. Milic-Frayling. Crowdsourcing for Book Search Evaluation: Impact of HIT Design on Comparative System Ranking. In SIGIR. ACM Press, New York NY, 2011.
14. F. W. Lancaster. Vocabulary control for information retrieval. Information Resources Press, Arlington VA, second edition, 1986.
15. J. Le, A. Edmonds, V. Hester, and L. Biewald. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation, pages 21-26, 2010.
16. C. Lu, P. Jung-ran, and X. Hu. User tags versus expert-assigned subject terms: A comparison of LibraryThing tags and Library of Congress Subject Headings. Journal of Information Science, 36(6): 763-779, 2010.
17. A. Mathes. Folksonomies - Cooperative Classification and Communication Through Shared Metadata, December 2004.
18. I. Ounis, C. Macdonald, M. de Rijke, G. Mishne, and I. Soboroff. Overview of the TREC 2006 Blog Track. In TREC, volume Special Publication 500-272. NIST, 2006.
19. L. Page, S. Brin, R. Motwani, and T. Winograd. The PageRank Citation Ranking: Bringing Order to the Web. Technical report, Stanford Digital Library Technologies Project, 1998.
20. I. Peters, L. Schumann, J. Terliesner, and W. G. Stock. Retrieval Effectiveness of Tagging Systems. In Proceedings of the 74rd ASIS&T Annual Meeting, volume 48, 2011.
21. K. Reuter. Assessing aesthetic relevance: Children's book selection in a digital library. JASIST, 58(12):1745-1763, 2007.
22. C. S. Ross. Finding without seeking: the information encounter in the context of reading for pleasure. Information Processing & Management, 35(6):783-799, 1999.
23. E. Svenonius. Unanswered questions in the design of controlled vocabularies. JASIS, 37(5):331-340, 1986.
24. E. M. Voorhees. The Philosophy of Information Retrieval Evaluation. In CLEF '01, pages 355-370, 2002. Springer-Verlag.
25. J. Voss. Tagging, folksonomy & co - renaissance of manual indexing? CoRR, abs/cs/0701072, 2007.
26. W. Weerkamp and M. de Rijke. Credibility improves topical blog post retrieval. In Proceedings of ACL-08: HLT, pages 923-931, June 2008. Association for Computational Linguistics.
27. K. Yi. A semantic similarity approach to predicting Library of Congress Subject Headings for social tags. JASIST, 61(8): 1658-1672, 2010.
28. K. Yi and L. M. Chan. Linking folksonomy to Library of Congress Subject Headings: an exploratory study. Journal of Documentation, 65(6):872-900, 2009.
29. E. Yilmaz, J. A. Aslam, and S. Robertson. A new rank correlation coefficient for information retrieval. In SIGIR, pages 587-594. ACM, 2008.