Recursive hashing functions for n-grams

ACM Transactions on Information Systems

Volumn 15, Issue 3, 1997, Pages 291-320

  Cohen, Jonathan D ^a  

^a NATIONAL SECURITY AGENCY   (United States)

Author keywords

E.2 Data : Data Storage Representations hash table representations; G.2.1 Discrete Mathematics : Combinatorics recurrences and difference equations; G.3 Mathematics of Computing : Probability And Statistics probabilistic algorithms

Indexed keywords

COMPUTATIONAL COMPLEXITY; DATA STRUCTURES; INDEXING (OF INFORMATION); INFORMATION RETRIEVAL;

RECURSIVE HASHING FUNCTIONS;

RECURSIVE FUNCTIONS;

EID: 0031177375     PISSN: 10468188     EISSN: None     Source Type: Journal    
DOI: 10.1145/256163.256168     Document Type: Article

References (54)

1. ANGELL, R. C., FREUND, G. E., AND WILLET, P. 1983. Automatic spelling correction using trigram similarity measure, Inf. Process. Manage. 19, 4, 255-261.
2. BARTON, I. J., CREASY, S. E., LYNCH, M. F., AND SNELL, M. J. 1974. An information-theoretic approach to text searching in direct access systems. Commun. ACM 17, 6 (June), 345-350.
3. BERLEKAMP, E. R. 1984. Algebraic Coding Theory. Aegean Park Press, Laguna Hills, Calif.
4. BICKEL, P. J. AND DOKSUM, K. A. 1977. Mathematical Statistics. Holden-Day, San Francisco, Calif.
5. BISHOP, Y. M. M., FIENBERG, S. E., AND HOLLAND, P. W. 1975. Discrete Multivariate Analysis: Theory and Practice. MIT Press, Cambridge, Mass.
6. BURNETT, J. E., COOPER, D., LYNCH, M. F., WILLETT, P., AND WYCHERLEY, M. 1979. Document retrieval experiments using indexing vocabularies of varying size. I. Variety generation symbols assigned to the fronts of index terms. J. Doc. 35, 3 (Sept.), 197-206.
7. CAVNAR, W. B. 1993. N-gram-based text filtering for TREC-2. In Proceedings of TREC-2: Text Retrieval Conference 2, D. Harman, Ed. National Bureau of Standards, Gaithersburg, Md.
8. COHEN, J. D. 1995. Highlights: Language- and domain-independent automatic indexing terms for abstracting. J. Am. Soc. Inf. Sci. 46 (Apr.), 162-174. See vol. 47, issue 3, p. 260 for erratum.
9. CORMEN, T. H., LEISERSON, C. E., AND RIVEST, R. L. 1990. Introduction to Algorithms. MIT Press, Cambridge, Mass.
10. CORNEW, R. W. 1968. A statistical method of spelling correction. Inf. Control 12, 79-93.
11. DAMASHEK, M. 1995. Gauging similarity with n-grams: Language-independent categorization of text. Science 267, 10 (Feb.), 843-848.
12. D'AMORE, R. J. AND MAH, C. P. 1985. One-time complete indexing of text: Theory and practice. 1985. In Proceedings of the 8th International ACM Conference on Research and Development in Information Retrieval. ACM, New York, 155-164.
13. DARRAGH, J. J., WITTEN, I. H., AND JAMES, M. L. 1990. The reactive keyboard: A predictive typing aid. Computer 23, 11 (Nov.), 41-49.
14. DE HEER, T. 1982. The application of the concept of homeosemy to natural language information retrieval. Inf. Process. Manage. 18, 5, 229-236.
15. DUMEY, A. I. 1956. Indexing for rapid random access memory systems. Comput. Autom. 5, 12 (Dec.), 6-9.
16. GONNET, G. H. AND BAEZA-YATES, R. A. 1990. An analysis of the Karp-Rabin string matching algorithm. Inf. Process. Lett. 34, 271-274.
17. GONNET, G. H. AND BAEZA-YATES, R. 1991. Handbook of Algorithms and Data Structures in C and Pascal. Addison-Wesley, Wokingham, U.K.
18. HANAN, M. AND PALERMO, F. P. 1963. An application of coding theory to a file address problem. IBM J. Res. Devel. 7 (Apr.), 127-129.
19. HANSON, A. R., RISEMAN, E. M., AND FISHER, E. 1976. Context in word recognition. Pattern Recog. 8, 35-45.
20. HARRISON, M. C. 1971. Implementation of the substring test for hashing. Commun. ACM 14, 12 (Dec.), 777-779.
21. HULL, J. J. AND SRIHARI, S. N. 1982. Experiments in text recognition with binary n-gram and Viterbi algorithms. IEEE Trans. Pattern Anal. Mach. Intell. PAMI-4, 5 (Sept.), 520-530.
22. HUSSAIN, A. B. S. AND DONALDSON, R. W. 1974. Suboptimal sequential decision schemes with on-line feature ordering. IEEE Trans. Comput. C-23, 6, 582-590.
23. KARP, R. M. AND RABIN, M. O. 1987. Efficient randomized pattern-matching algorithms. IBM J. Res. Devel. 31, 2 (Mar.), 249-260.
24. KIM, J. Y. AND SHAWE-TAYLOR, J. 1992. An approximate string-matching algorithm. Theoret. Comput. Sci. 92, 107-117.
25. KNUTH, D. E. 1973. The Art of Computer Programming. Vol. 3, Sorting and Searching. Addison-Wesley, Reading, Mass.
26. KOTAMARTI, U. AND THARP, A. L. 1990. Accelerated text searching through signature trees. J. Am. Soc. Inf. Sci. 41, 2 (Mar.), 79-86.
27. KUKICH, K. 1992. Techniques for automatically correcting words in text. ACM Comput. Surv. 24, 4 (Dec.), 377-439.
28. LUM, V. Y., YUEN, P. S. T., AND DODD, M. 1971. Key-to-address transform techniques: A fundamental performance study on large existing formatted files. Commun. ACM 14, 4 (Apr.), 228-239.
29. LYNCH, M. F. 1977. Variety generation - A reinterpretation of Shannon's mathematical theory of communication, and its implications for information science. J. Am. Soc. Inf. Sci. 28, 1 (Jan.), 19-25.
30. MCELWAIN, C. K. AND EVENS, M. B. 1962. The degarbler - A program for correcting machine-read morse code. Inf. Control 5, 368-384.
31. MCKENZIE, B. J., HARRIES, R., AND BELL, T. 1990. Selecting a hashing algorithm. Softw. Pract. Exper. 20, 2 (Feb.), 209-224.
32. MORRIS, R. AND CHERRY, L. L. 1975. Computer detection of typographical errors. IEEE Trans. Prof. Commun. PC-18 (Mar.), 54-64.
33. NAKAMURA, M. AND SHIKANO, K. 1989. A study of English word category prediction based on neural networks. In ICASSP-89: International Conference on Acoustics, Speech, and Signal Processing. Vol. 2. IEEE, New York, 731-734.
34. NEUHOFF, D. L. 1975. The Viterbi algorithm as an aid in text recognition. IEEE Trans. Inf. Theory IT-21 (Mar.), 222-226.
35. PAESELER, A. AND NEY, H. 1989. Continuous-speech recognition using a stochastic language model. In ICASSP-89: International Conference on Acoustics, Speech, and Signal Processing 2. IEEE, New York, 719-722.
36. PEARCE, C. E. 1994. A dynamic hypertext environment through n-gram analysis. Ph.D. dissertation, Univ. of Maryland Baltimore County, Baltimore, Md.
37. PETERSON, W. W. 1957. Addressing for random-access storage. IBM J. Res. Devel. 1 (Apr.), 130-146.
38. PETERSON, W. W. 1961. Error-Correcting Codes. John Wiley & Sons, New York.
39. PIETRA, S. D., PIETRA, V. D., MERCER, R. L., AND ROUKOS, S. 1992. Adaptive language modeling using minimum discriminant estimation. In ICASSP-92: International Conference on Acoustics, Speech, and Signal Processing. Vol. 1. IEEE, New York, 633-636.
40. PRYOR, D. V., THISTLE, M. R., AND SHIRAZI, N. 1993. Text searching on Splash 2. In Proceedings of the IEEE Workshop on FPGAs for Custom Computing Machines. IEEE Computer Society Press, Los Alamitos, Calif., 172-177.
41. SCHAY, G. AND RAVER, N. 1963. A method for key-to-address transformation. IBM J. Res. Devel. 7 (Apr.), 121-126.
42. SCHMITT, J. C. 1990. Trigram-based method of language identification. U.S. Patent No. 5,062,143. U.S. Patent Office, Washington, D.C.
43. SCHUEGRAF, E. J. AND HEAPS, H. S. 1973. Selection of equifrequent word fragments for information retrieval. Inf. Storage Retrieval. 9, 697-711.
44. SHANNON, C. E. 1951. Prediction and entropy of printed english. Bell System Tech. J. 30 (Jan.), 50-64.
45. SHINGHAL, R., ROSENBERG, D., AND TOUSSAINT, G. T. 1978. A simplified heuristic version of a recursive Bayes algorithm for using context in text recognition. IEEE Trans. Syst. Man Cyber. SMC-8, 5 (May), 412-414.
46. SUEN, C. Y. 1979. n-gram statistics for natural language understanding and text processing. IEEE Trans. Patt. Anal. Mach. Intell. PAMI-1, 2 (Apr.), 164-172.
47. THOMAS, R. B. AND KASSLER, M. 1967. Character recognition in context. Inf. Control 10, 43-64.
48. UKKONEN, E. 1992. Approximate string-matching with q-grams and maximal matches. Theor. Comput. Sci. 92, 191-211.
49. VOSSLER, C. M. AND BRANSTON, N. M. 1964. The use of context for correcting garbled English text. In Proceedings of the 19th National Conference of the ACM. ACM, New York, D2.4-1-D2.4-13.
50. WILLET, P. 1979. Document retrieval experiments using indexing vocabularies of varying size. II. Hashing, truncation, digram and trigram encoding of index terms. J. Doc. 35, 4 (Dec.), 296-305.
51. WISNIEWSKI, J. L. 1987. Effective text compression with simultaneous digram and trigram encoding. J. Inf. Sci. 13, 159-164.
52. WEIGHT, J. H., JONES, G. J. F., AND WRIGLEY, E. N. 1992. Hybrid grammar-bigram speech recognition system with first-order dependence model. In ICASSP-92: International Conference on Acoustics, Speech, and Signal Processing. Vol. 1. IEEE, New York, 169-172.
53. YANNAKOUDAKIS, E. J. AND HUTTON, P. J. 1992. An assessment of n-phoneme statistics in phoneme guessing algorithms which aim to incorporate phonotactic constraints. Speech Commun. 11, 6 (Dec.), 581-602.
54. ZAMORA, E. M., POLLOCK, J. J., AND ZAMORA, A. 1981. The use of trigram analysis for spelling error detection. Inf. Process. Manage. 17, 6, 305-316.