Constructing test collections by inferring document relevance via extracted relevant information

ACM International Conference Proceeding Series

Volumn , Issue , 2012, Pages 145-154

  Rajput, Shahzad ^a   Ekstrand Abueg, Matthew ^a   Pavlu, Virgil ^a   Aslam, Javed A ^a  

^a Northeastern University   (United States)

Author keywords

evaluation; information retrieval; nuggets; relevance assessment

Indexed keywords

EVALUATION; GENERATING SET; INFORMATION NEED; NUGGETS; RELEVANCE ASSESSMENTS; RELEVANT DOCUMENTS; SEARCH SPACES; SYSTEM EVALUATION; TEST COLLECTION;

ALGORITHMS; INFORMATION RETRIEVAL; KNOWLEDGE MANAGEMENT;

INFORMATION RETRIEVAL SYSTEMS;

EID: 84871062543     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2396761.2396783     Document Type: Conference Paper

References (42)

1. The Eighth Text REtrieval Conference (TREC-8). U.S. Government Printing Office, 2000.
2. 33rd ACM SIGIR Workshop on Crowdsourcing for Search Evaluation, Geneva, Switzerland, 2010.
3. E. Amitay, D. Carmel, R. Lempel, and A. Soffer. Scaling IR-system evaluation using term relevance sets. SIGIR '04.
4. J. Aslam. Improving algorithms for boosting. In N. Cesa-Bianchi and S. Goldman, editors, Proceedings of the Thirteenth Annual Conference on Computational Learning Theory, July 2000.
5. J. A. Aslam and V. Pavlu. A practical sampling strategy for efficient retrieval evaluation, technical report.
6. J. A. Aslam, V. Pavlu, and R. Savell. A unified model for metasearch and the efficient evaluation of retrieval systems via the hedge algorithm. SIGIR '03.
7. Y. Baram, R. El-Yaniv, and K. Luz. Online choice of active learning algorithms. J. Mach. Learn. Res., Volume 5, December 2004.
8. L. Bentivogli, P. Clark, I. Dagan, H. T. Dang, and D. Giampiccolo. The sixth PASCAL recognizing textual entailment challenge. In The Text Analysis Conference (TAC 2010), 2010.
9. C. M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics). 2006.
10. A. Z. Broder. Identifying and filtering near-duplicate documents. COM '00, London, UK, 2000.
11. A. Z. Broder, S. C. Glassman, M. S. Manasse, and G. Zweig. Syntactic clustering of the web. WWW'97.
12. B. Carterette, J. Allan, and R. K. Sitaraman. Minimal test collections for retrieval evaluation. SIGIR'06.
13. B. Carterette and I. Soboroff. The effect of assessor error on IR system evaluation. SIGIR '10.
14. C. L. A. Clarke, N. Craswell, and I. Soboroff. Overview of the TREC 2009 web track, 2009.
15. C. L. A. Clarke, M. Kolla, G. V. Cormack, O. Vechtomova, A. Ashkan, S. Büttcher, and I. MacKinnon. Novelty and diversity in information retrieval evaluation. SIGIR'08.
16. G. V. Cormack, C. R. Palmer, and C. L. A. Clarke. Efficient construction of large test collections. SIGIR'98, pages 282-289, Aug. 1998.
17. H. T. Dang, J. J. Lin, and D. Kelly. Overview of the TREC 2006 question answering track 99. In TREC, 2006.
18. Y. Freund and R. E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. COLT '95.
19. D. Harman. Overview of the third text REtreival conference (TREC-3). In Overview of the Third Text REtrieval Conference (TREC-3). U.S. Government Printing Office, Apr. 1995.
20. M. Hosseini, I. J. Cox, N. Milic-Frayling, G. Kazai, and V. Vinay. On aggregating labels from multiple crowd workers to infer relevance of documents. In ECIR, volume 7224 of ECIR '12, 2012.
21. P. G. Ipeirotis. get-another-label: A set of tools for managing redundant answers collected from amazon. http://code.google.com/p/get-another-label/.
22. P. G. Ipeirotis, F. Provost, and J. Wang. Quality management on amazon mechanical turk. HCOMP '10.
23. K. Járvelin and J. Kekáláinen. Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst., Volume 20, 2002.
24. T. Joachims. Training linear svms in linear time. KDD '06, pages 217-226, New York, NY, USA, 2006. ACM.
25. S. Krenzel. Finding blurbs. Website. http://www.stevekrenzel.com/ articles/blurbs.
26. J. Lin and D. Demner-Fushman. Automatically evaluating answers to definition questions. HLT '05.
27. J. Lin and D. Demner-Fushman. Will pyramids built of nuggets topple over? HLT'06.
28. V. Pavlu, S. Rajput, P. B. Golbus, and J. A. Aslam. IR system evaluation using nugget-based test collections. WSDM '12.
29. F. Radlinski, M. Kurup, and T. Joachims. How does clickthrough data reflect retrieval quality? CIKM '08.
30. T. Sakai, M. P. Kato, and Y.-I. Song. Click the search button and be happy: evaluating direct and immediate information access. CIKM '11.
31. T. Sakai, M. P. Kato, and Y.-I. Song. Overview of NTCIR-9 1CLICK. 2011.
32. T. Sakai and C.-Y. Lin. Ranking Retrieval Systems without Relevance Assessments - Revisited. In the 3rd International Workshop on Evaluating Information Access (EVIA) - NTCIR-8, 2010.
33. T. Sakai and R. Song. Evaluating diversified search results using per-intent graded relevance. SIGIR '11.
34. R. E. Schapire, Y. Freund, P. Bartlett, and W. S. Lee. Boosting the Margin: A New Explanation for the Effectiveness of Voting Methods. The Annals of Statistics, 26(5), 1998.
35. H. S. Seung, M. Opper, and H. Sompolinsky. Query by committee. COLT '92.
36. I. Soboroff, C. Nicholas, and P. Cahan. Ranking retrieval systems without relevance judgments. SIGIR '01.
37. A. Spink. Term relevance feedback and query expansion: relation to design. SIGIR '94.
38. A. Spoerri. Using the structure of overlap between search results to rank retrieval systems without relevance judgments. Inf. Process. Manage., 43, 2007.
39. R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction (Adaptive Computation and Machine Learning). The MIT Press, Mar. 1998.
40. S. Thompson. Sampling. Wiley series in probability and mathematical statistics: Applied probability and statistics. 1992.
41. E. M. Voorhees. Overview of the TREC 2005 robust retrieval track. TREC 2005.
42. E. M. Voorhees. Question answering in TREC. CIKM '01.