String techniques for detecting duplicates in document databases

International Journal on Document Analysis and Recognition

Volumn 2, Issue 4, 2000, Pages 186-199

  Lopresti, Daniel P ^a  

^a LUCENT TECHNOLOGIES   (United States)

Author keywords

Approximate string matching; Document analysis; Duplicate detection; Information retrieval; Optical character recognition

Indexed keywords

CHARACTER RECOGNITION; INFORMATION RETRIEVAL;

APPROXIMATE STRING MATCHING; DOCUMENT ANALYSIS; DOCUMENT DATABASE; DOCUMENT IMAGE DATABASE; DUPLICATE DETECTION; ORDERS OF MAGNITUDE; PRINTED DOCUMENTS; REAL-WORLD NOISE;

OPTICAL CHARACTER RECOGNITION;

EID: 0042257703     PISSN: 14332833     EISSN: 14332825     Source Type: Journal    
DOI: 10.1007/s100320050005     Document Type: Article

References (34)

1. J. Aoe. Computer Algorithms: String Pattern Matching Strategies. IEEE Computer Society Press, Los Alamitos, CA, 1994
2. H. Bunke. A fast algorithm for finding the nearest neighbor of a word in a dictionary. In: Proc. 2nd Int. Conf. on Doc. Anal. and Recognition, pp. 632-637, Tsukuba Science City, Japan, October 1993
3. D. Doermann, H. Li, O. Kia. The detection of duplicates in document image databases. In: Proc. 4th Int. Conf. on Doc. Anal. and Recognition, pp. 314-318, Ulm, Germany, August 1997
4. M.-W. Du, S. C. Chang. An approach to designing very fast approximate string matching algorithms. IEEE Trans. on Knowl. and Data Eng. 6(4): 620-633 (1994)
5. J. Esakov, D. P. Lopresti, J. S. Sandberg. Classification and distribution of optical character recognition errors. In: Proc. Doc. Recognition I (IS&T/SPIE Electronic Imaging), pp. 204-216, San Jose, CA, February 1994
6. J. W. Fickett. Fast optimal alignment. Nucleic Acids Research 12(1): 175-179 (1984)
7. G. G. Gilmore. Former Army operations offcers assist DoD's search. Army Link News, December 1997. http://www.dtic.mil/armylink/news/Dec1997/ a19971209moredata.html
8. M. Gokhale, W. Holmes, A. Kopser, D. Lopresti, S. Lucas, R. Minnich, D. Sweely. Building and using a highly parallel programmable logic array. Computer, 24(1): 81-89, 1991
9. GulfLink. http://www.gulflink.osd.mil/
10. D. Gusfield. Algorithms on Strings, Trees, and Sequences. Cambridge University Press, Cambridge, UK, 1997
11. J. J. Hull. Document image similarity and equivalence detection. Int. J. Doc. Anal. and Recognition 1(1): 37-42 (1998)
12. J. J. Hull, J. Cullen, M. Peairs. Document image matching and retrieval techniques. In: Proc. Symp. on Doc. Image Understanding Technol., pp. 31-35, Annapolis, MD, April-May 1997
13. D.-S. Lee, J. J. Hull. Duplicate detection for symbolically compressed documents. In: Proc. 5th Int. Conf. on Doc. Anal. and Recognition, pp. 305-308, Bangalore, India, September 1999
14. V. I. Levenshtein. Binary codes capable of correcting deletions, insertions, and reversals. Cybernetics and Control Theory 10(8): 707-710 (1966)
15. R. J. Lipton, D. P. Lopresti. A systolic array for rapid string comparison. In: Proc. 1985 Chapel Hill Conf. on Very Large Scale Integration, pp. 363-376. Computer Science Press, 1985
16. D. P. Lopresti. Models and algorithms for duplicate document detection. In: Proc. 5th Int. Conf. on Doc. Anal. and Recognition, pp. 297-300, Bangalore, India, September 1999
17. D. P. Lopresti. String techniques for duplicate document detection. In: Proc. Symp. on Doc. Image Understanding Technol., pp. 101-112, Annapolis, MD, April 1999
18. D. P. Lopresti. A comparison of text-based methods for detecting duplication in document image databases. In: Proc. Doc. Recognition and Retrieval VII (IS&T/SPIE Electronic Imaging), 3967: 210-221, San Jose, CA, January 2000
19. D. P. Lopresti, A. Tomkins. Block edit models for approximate string matching. Theoretical Computer Science (181): 159-179 (1997)
20. U. Manber. Finding similar files in a large file system. In: Proc. USENIX, pp. 1-10, San Francisco, CA, January 1994
21. S. B. Needleman, C. D. Wunsch. A general method applicable to the search for similarities in the amino-acid sequences of two proteins. J. Mol. Biol. 48:443-453 (1970)
22. F. Prokoski. Database partitioning and duplicate document detection based on optical correlation. In: Proc. Symp. on Doc. Image Understanding Technol., pp. 86-97, Annapolis, MD, April 1999
23. L. R. Rabiner, A. E. Rosenberg, S. E. Levinson. Considerations in dynamic time warping algorithms for discrete word recognition. IEEE Trans. on Acoust., Speech, and Signal Process. ASSP-26(6): 575-582 (1978)
24. R. Rogers, V. Chalana, G. Marchisio, T. Nguyen, A. Bruce. Duplicate document detection in DocBrowse. In: Proc. Symp. on Doc. Image Understanding Technol., pp. 119-127, Annapolis, MD, April 1999
25. D. Sankoff, J. B. Kruskal (eds.). Time Warps, String Edits, and Macromolecules: The Theory and Practice of Sequence Comparison. Addison-Wesley, Reading, MA, 1983
26. P. H. Sellers. The theory and computation of evolutionary distances: pattern recognition. J. Algorithms 1: 359-373 (1980)
27. N. Shivakumar, H. Garcia-Molina. SCAM: A copy detection mechanism for digital documents. In: Proc. 2nd Int. Conf. on Theory and Practice of Digital Libr., Austin, TX, 1995 http://www.csdl.tamu.edu/DL95/papers/shivakumar.ps
28. N. Shivakumar, H. Garcia-Molina. Finding near-replicas of documents on the Web. In: Proc. Workshop on Web Databases, March 1998
29. T. F. Smith, M. S. Waterman. identification of common molecular sequences. J. Mol. Biol. 147: 195-197 (1981)
30. A. L. Spitz. Duplicate document detection. In: Proc. Doc. Recognition IV (IS&T/SPIE Electronic Imaging), 3027: 88-94, San Jose, CA, February 1997
31. G. A. Stephen. String Searching Algorithms. World Scientific, Singapore, 1994
32. K. Taghva, J. Borsack, A. Condit, P. Inaparthy. Effects of OCR errors on short documents. In: Annual Report of UNLV Information Science Research Institute, pp. 99-105, Las Vegas, NV, 1995
33. E. Ukkonen. On approximate string matching. In: Proc. Int. Conf. on Foundations of Comput. Theory, LNCS 158, pp. 487-493. Springer, Berlin Heidelberg New York, 1983
34. R. A. Wagner, M. J. Fischer. The string-to-string correction problem. J. ACM 21: 168-173 (1974)