Brute force and indexed approaches to pairwise document similarity comparisons with MapReduce

Proceedings - 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2009

Volumn , Issue , 2009, Pages 155-162

  Lin, Jimmy ^a  

^a UNIVERSITY OF MARYLAND   (United States)

Author keywords

Distributed algorithms; Hadoop

Indexed keywords

AD HOC RETRIEVAL; BRUTE FORCE; BRUTE FORCE ALGORITHMS; CARTESIAN PRODUCTS; DISTRIBUTED ALGORITHM; DOCUMENT COLLECTION; DOCUMENT SIMILARITY; EFFICIENCY GAIN; LIFE-SCIENCES;

INFORMATION RETRIEVAL; INFORMATION SERVICES; PARALLEL ALGORITHMS;

APPROXIMATION ALGORITHMS;

EID: 71749087578     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1571941.1571970     Document Type: Conference Paper

References (15)

1. A. Andoni and P. Indyk. Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions. CACM, 51(1):117-122, 2008.
2. V. Anh, O. de Kretser, and A. Moffat. Vector-space ranking with effective early termination. In SIGIR, 35-42, 2001.
3. R. Bayardo, Y. Ma, and R. Srikant. Scaling up all pairs similarity search. In WWW, 131-140, 2007.
4. J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. In OSDI, 137-150, 2004.
5. T. Elsayed, J. Lin, and D. Oard. Pairwise document similarity in large collections with MapReduce. In ACL, Companion Volume, 265-268, 2008.
6. S. Ghemawat, H. Gobioff, and S.-T. Leung. The Google File System. In SOSP, 29-43, 2003.
7. W. Hersh, A. Cohen, J. Yang, R. Bhupatiraju, P. Roberts, and M. Hearst. TREC 2005 Genomics Track overview. In TREC, 2005.
8. N. Lester, A. Moffat, W. Webber, and J. Zobel. Space-limited ranked query evaluation using adaptive pruning. In WISE, 470-477, 2005.
9. J. Lin and M. Smucker. How do users find things with PubMed? Towards automatic utility evaluation with user simulations. In SIGIR, 19-26, 2008.
10. J. Lin and W. J. Wilbur. PubMed related articles: A probabilistic topic-based model for content similarity. BMC Bioinformatics, 8:423, 2007.
11. A. Moffat and J. Zobel. Self-indexing inverted files for fast text retrieval. ACM TOIS, 14(4):349-379, 1996.
12. M. Persin, J. Zobel, and R. Sacks-Davis. Filtered document retrieval with frequency-sorted indexes. JASIS, 47(10):749-764, 1996.
13. T. Strohman and W. Croft. Efficient document retrieval in main memory. In SIGIR, 175-182, 2007.
14. I. Witten, A. Moffat, and T. Bell. Managing Gigabytes: Compressing and Indexing Documents and Images. Morgan Kaufmann, San Francisco, 1999.
15. C. Zhai and J. Lafferty. A study of smoothing methods for language models applied to ad hoc information retrieval. In SIGIR, 334-342, 2001.