Distributed text retrieval from overlapping collections

Conferences in Research and Practice in Information Technology Series

Volumn 63, Issue , 2007, Pages 141-150

  Shokouhi, Milad ^a   Zobel, Justin ^a   Bernstein, Yaniv ^a  

^a RMIT UNIVERSITY   (Australia)

Author keywords

Distributed information retrieval; Duplicate and near duplicate detection; Search engines; Similarity measurement

Indexed keywords

DISTRIBUTED INFORMATION RETRIEVAL; NEAR-DUPLICATE DETECTION; NEAR-DUPLICATES; RESULT MERGING; SERVER SELECTION; SIGNIFICANT IMPACTS; SIMILARITY MEASUREMENTS; SINGLE SERVER; TEST DATA; TEXT RETRIEVAL; TEXT RETRIEVAL SYSTEM;

QUERY PROCESSING; SEARCH ENGINES;

INFORMATION RETRIEVAL;

EID: 84872869103     PISSN: 14451336     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Article

References (54)

1. Allan et al, J. (2003), 'Challenges in information retrieval and language modeling: report of aworkshop held at the center for intelligent information retrieval, University of Massachusetts Amherst, september 2002', SIGIR Forum 37(1), 31-47.
2. Avrahami, T., Yau, L., Si, L. & Callan, J. (2006), 'The FedLe- mur: federated search in the real world', Journal of the American Society for Information Science and Technology 57(3), 347-358.
3. Bergman, M. (2001), 'The deep Web: Surfacing hidden value', Journal of Electronic Publishing 7(1).
4. Bernstein, Y., Shokouhi, M. & Zobel, J. (2006), Compact features for detection of near-duplicates in distributed retrieval, in 'Proceedings of String Processing and Information Retrieval Symposium (to appear)', Glasgow, Schotland.
5. Bernstein, Y. & Zobel, J. (2004), A scalable system for identifying co-derivative documents, in 'Proceedings of String Processing and Information Retrieval Symposium', Padova, Italy, pp. 55-67.
6. Bernstein, Y. & Zobel, J. (2005), Redundant documents and search effectiveness, in 'Proceedings of 14th ACM CIKM Conference on Information and Knowledge Management', Bremen, Germany, pp. 736-743.
7. Brin, S., Davis, J. & Garćia-Molina, H. (1995), Copy detection mechanisms for digital documents, in 'Proceedings of ACM SIGMOD international conference on Management of Data', San Jose, California, pp. 398-409.
8. Broder, A. Z., Glassman, S. C., Manasse, M. S. & Zweig, G. (1997), 'Syntactic clustering of the web', Computer Networks and ISDN Systems 29(8-13), 1157-1166.
9. Callan, J. & Connell, M. (2001), 'Query-based sampling of text databases', ACM Transactions on Information Systems 19(2), 97-130.
10. Callan, J., Connell, M. & Du, A. (1999), Automatic discovery of language models for text databases, in 'Proceedings of ACM SIGMOD International Conference on Management of Data', Philadelphia, Pennsylvania, pp. 479-490.
11. Callan, J., Croft, W. B. & Harding, S. M. (1992), The IN- QUERY retrieval system, in 'Proceedings of third International Conference on Database and Expert Systems Applications', Valencia, Spain, pp. 78-83.
12. Callan, J., Lu, Z. & Croft, W. B. (1995), Searching distributed collections with inference networks, in 'Proceedings of 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval', Seattle, Washington, pp. 21-28.
13. Chowdhury, A., Frieder, O., Grossman, D. & McCabe, M. C. (2002), 'Collection statistics for fast duplicate document detection', ACM Transactions on Information Systems 20(2), 171-191.
14. Conrad, J. G., Guo, X. S. & Schriber, C. P. (2003), Online duplicate document detection: Signature reliability in a dynamic retrieval environment, in 'Proceedings of 12th ACM CIKM Conference on Information and Knowledge Management', New Orleans, Louisiana, pp. 443-452.
15. Cooper, J. W., Coden, A. R. & Brown, E.W. (2002), Detecting similar documents using salient terms, in 'Proceedings of 11th ACM CIKM Conference on Information and Knowledge Management', McLean, Virginia, pp. 245-251.
16. Craswell, N., Bailey, P. & Hawking, D. (2000), Server selection on the World Wide Web, in 'Proceedings of Fifth ACM Conference on Digital Libraries', San Antonio, Texas, pp. 37-46.
17. Craswell, N. & Hawking, D. (2002), Overview of the TREC-2002 Web Track, in 'Proceedings of TREC-2002', Gaithersburg, Maryland.
18. Croft, B. (2000), 'Combining approaches to information retrieval', Advances in information retrieval, chapter 1 pp. 1-36.
19. D'Souza, D., Zobel, J. & Thom, J. (2004), Is CORI effective for collection selection? an exploration of parameers, queries, and data, in 'Proceedings of Australian Document Computing Symposium', Melbourne, Australia, pp. 41-46.
20. Fetterly, D., Manasse, M. & Najork, M. (2003), On the evo- lution of clusters of near-duplicate web pages, in 'Proceedings of first Latin American Web Congress', IEEE, pp. 37-45.
21. Fox, E. & Shaw, J. (1994), Combination of multiple searches, in 'Proceedings of TREC-1994', NIST Special Publication, Gaithersburg, Maryland, pp. 105-108.
22. Gauch, S., Wang, G. & Gomez, M. (1996), 'ProFusion: Intelligent fusion from multiple, distributed search engines', Journal Universal Computer Science 2(9), 637-649.
23. Gravano, L., Chang, C. K., Garcia-Molina, H. & Paepcke, A. (1997), STARTS: Stanford proposal for Internet metasearching, in 'Proceedings of ACM SIGMOD International Conference on Management of Data', Tucson, Arizona, pp. 207-218.
24. Gravano, L., Garcia-Molina, H. & Tomasic, A. (1999), 'GlOSS: text-source discovery over the Internet', ACM Transactions on Database Systems 24(2), 229-264.
25. Hawking, D. & Thomas, P. (2005), Server selection methods in hybrid portal search, in 'Proceedings of 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval', Salvador, Brazil, pp. 75-82.
26. Hernandez, T. & Kambhampati, S. (2005), Improving text collection selection with coverage and overlap statistics, in 'Proceedings of 14th International Conference on World Wide Web', Chiba, Japan.
27. Hoad, T. C. & Zobel, J. (2003), 'Methods for identifying versioned and plagiarised documents', Journal of the American Society for Information Science and Technology 54(3), 203-215.
28. Ilyinski, S., Kuzmin, M., Melkov, A. & Segalovich, I. (2002), An efficient method to detect duplicates of web documents with the use of inverted index, in 'Proceedings of 11th International Conference on World Wide Web', Honolulu, Hawaii.
29. Ipeirotis, P. G. & Gravano, L. (2002), Distributed search over the hidden Web: Hierarchical database sampling and selection., in 'Proceedings of 28th International Conference on Very Large Data Bases', Hong Kong, China, pp. 394-405.
30. Ipeirotis, P. G. & Gravano, L. (2004), When one sample is not enough: improving text database selection using shrinkage, in 'Proceedings of ACM SIGMOD International Conference on Management of Data', Paris, France, pp. 767-778.
31. Kirsch, T. (2003), 'Document retrieval over networks wherein ranking and relevance scores are computed at the client for multiple database documents', U.S. Patent 5,659,732.
32. Kolcz, A., Chowdhury, A. & Alspector, J. (2004), Improved robustness of signature-based near-replica detection via lexicon randomization, in 'Proceedings of ACM SIGKDD International Conference on Knowledge Discovery and Data Mining', Seattle, WA, pp. 605-610.
33. Lee, J. (1997), Analyses of multiple evidence combination, in 'Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval', Philadelphia, Pennsylvania, United States, pp. 267-276.
34. Lyon, C., Malcolm, J. & Dickerson, B. (2001), Detecting short passages of similar text in large document collections, in 'Proceedings of Conference on Empirical Methods in Natural Language Processing', Philadelphia, Pennsylvania.
35. Manber, U. (1994), Finding similar files in a large file system, in 'Proceedings of USENIXWinter Technical Conference', San Fransisco, CA, pp. 1-10.
36. Meng, W., Yu, C. & Liu, K. (2002), 'Building efficient and effective metasearch engines', ACM Computing Surveys 34(1), 48-89.
37. Nottelmann, H. & Fuhr, N. (2003), Evaluating different methods of estimating retrieval quality for resource selection, in 'Proceedings of 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval', Toronto, Canada, pp. 290-297.
38. Powell, A. L. & French, J. (2003), 'Comparing the performance of collection selection algorithms', ACM Transactions on Information Systems 21(4), 412-456.
39. Pugh, W. & Henzinger, M. H. (2003), 'Detecting duplicate and near-duplicate files (United States Patent 6,658,423)'.
40. Rasolofo, Y., Abbaci, F. & Savoy, J. (2001), Approaches to collection selection and results merging for distributed information retrieval, in 'Proceedings of 10th ACM CIKM International Conference on Information and knowledge management', Atlanta, Georgia, pp. 191-198.
41. Selberg, E. & Etzioni, O. (1997), 'The MetaCrawler architecture for resource aggregation on the web', IEEE Expert 12(1), 8-14.
42. Shokouhi, M., Scholer, F. & Zobel, J. (2006), Sample sizes for query probing in uncooperative distributed information retrieval, in 'Proceedings of of Eighth Asia Pacific Web Conference', Harbin, China, pp. 63-75.
43. Si, L. & Callan, J. (2003a), Relevant document distribution estimation method for resource selection, in 'Proceedings of 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval', Toronto, Canada, pp. 298-305.
44. Si, L. & Callan, J. (2003b), 'A semisupervised learning method to merge search engine results', ACM Transactions on Information Systems 21(4), 457-491.
45. Si, L. & Callan, J. (2004), Unified utility maximization frame- work for resource selection, in 'Proceedings of 13th ACM CIKM Conference on Information and Knowledge Management', Washington, D.C., pp. 32-41.
46. Si, L. & Callan, J. (2005), Modeling search engine effectiveness for federated search, in 'Proceedings of 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval', Salvador, Brazil.
47. Si, L., Jin, R., Callan, J. & Ogilvie, P. (2002), A language modeling framework for resource selection and results merging, in 'Proceedings of 11th ACM CIKM International Conference on Information and Knowledge Management', New York, NY, pp. 391-397.
48. Sugiura, A. & Etzioni, O. (2000), Query routing for web search engines: architectures and experiments, in 'Proceedings of the 9th international World Wide Web conference on Computer networks', North-Holland Publishing Co., Amsterdam, The Netherlands, pp. 417-429.
49. Wu, S. & McClean, S. (2006), 'Result merging methods in distributed information retrieval with overlapping databases', Journal of Information Retrieval (In press).
50. Xu, J. & Callan, J. (1998), Effective retrieval with distributed collections, in 'Proceedings of 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval', Melbourne, Australia, pp. 112-120.
51. Yuwono, B. & Lee, D. L. (1997), Server ranking for distributed text retrieval systems on the internet, in 'Proceedings of the Fifth International Conference on Database Systems for Advanced Applications (DASFAA)', World Scientific Press, Melbourne, Australia, pp. 41-50.
52. Zamir, O. & Etzioni, O. (1999), Grouper: a dynamic cluster- ing interface to web search results, in 'Proceedings of 8th International Conference on World Wide Web', Toronto, Canada, pp. 1361-1374.
53. Zobel, J. (1997), Collection selection via lexicon inspection, in P. Bruza, ed., 'Proceedings of the Australian Document Computing Symposium', pp. 74-80.
54. Zobel, J. & Bernstein, Y. (2006), The case of the duplicate documents: Measurement, search, and science, in 'Proceedings of of Eighth Asia Pacific Web Conference', Harbin, China, pp. 26-39.