An efficient duplicate record detection using q-grams array inverted index

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Volumn 6263 LNCS, Issue , 2010, Pages 309-323

  Ferro, Alfredo ^a   Giugno, Rosalba ^a   Puglisi, Piera Laura ^a   Pulvirenti, Alfredo ^a  

^a UNIVERSITY OF CATANIA   (Italy)

Author keywords

bitmaps; clustering; Duplicate record detection; inverted index; q grams

Indexed keywords

BIT MAPS; CLUSTERING; DUPLICATE RECORD DETECTION; INVERTED INDICES; Q-GRAMS;

CLUSTER ANALYSIS; CLUSTERING ALGORITHMS; DATA STRUCTURES; EFFICIENCY; INDEXING (OF INFORMATION);

DATA WAREHOUSES;

EID: 78049385808     PISSN: 03029743     EISSN: 16113349     Source Type: Book Series    
DOI: 10.1007/978-3-642-15105-7_25     Document Type: Conference Paper

References (23)

1. Aizawa, A., Oyama, K.: A fast linkage detection scheme for multi-source information integration. In: International Workshop on Challenges in Web Information Retrieval and Integration, pp. 30-39 (2005)
2. Baxter, R., Christen, P., Churches, T.: A comparison of fast blocking methods for record linkage. In: ACM SIGKDD 2003 Workshop on Data Cleaning, Record Linkage, and Object Consolidation, pp. 25-27 (2003)
3. Chatterjee, A., Segev, A.: Data manipulation in heterogeneous databases. ACM SIGMOD Record 20, 64-68 (1991)
4. Chaudhuri, S., Ganjam, K., Ganti, V., Motwani, R.: Robust and efficient fuzzy match for online data cleaning. In: SIGMOD 2003, pp. 313-324 (2003)
5. Christen, P., Gayler, R.: Towards scalable real-time entity resolution using a similarity-aware inverted index approach. Proceedings of AusDM 2008, Glenelg, Adelaide 87, 30-39 (2008)
6. Christen, P., Churches, T.: Febrl: Freely extensible biomedical record linkage Manual (2002)
7. Cohen, W., Richman, J.: Learning to match and cluster large high-dimensional data sets for data integration. In: SIGKDD 2002 (2002)
8. Elmagarmid, A., Ipeirotis, P., Verykios, V.: Duplicate record detection: A survey. TKDE 19 (2007)
9. Gravano, L., Ipeirotis, P.G., Jagadish, H.V., Koudas, N., Muthukrishnan, S., Srivastava, D.: Approximate string joins in a database (almost) for free. In: VLDB 2001, pp. 491-500 (2001)
10. Han, J., Kamber, M.: The data warehouse ETL toolkit: Practical techniques for extracting, cleaning, conforming, and delivering data. John Wiley and Sons, Chichester (2004)
11. Jaro, M.A.: Unimatch: A record linkage system: User's manual. Technical report, U.S. Bureau of the Census, Washington, D.C (1976)
12. Jaro, M.A.: Advances in record linkage methodology as applied to matching the 1985 census of tampa, florida. Journal of the American Statistical Society 84, 414-420 (1989)
13. Levenshtein, V.I.: Binary codes capable of correcting deletions, insertions and reversals. Doklady Akademii Nauk SSSR 163, 845-848 (1965)
14. Hernandez, M., Stolfo, S.: The merge/purge problem for large databases. In: Proceedings of the ACM SIGMOD International Conference on Management of Data (1995)
15. Manber, U., Myers, G.: Suffix arrays: a new method for on-line string searches. SIAM Journal on Computing 22, 935-948 (1993)
16. McCallum, A., Nigam, K., Ungar, L.H.: Efficient clustering of high-dimensional data sets with application to reference matching. In: ACM SIGKDD, pp. 169-178 (2000)
17. Monge, A.E., Elkan, C.P.: An efficient domain-independent algorithm for detecting approximately duplicate database records. In: Proceedings of DMKD 1997, pp. 23-29 (1997)
18. Ramos, J.: Using tf-idf to determine word relevance in document queries. In: Proceedings of the First Instructional Conference on Machine Learning (2003)
19. Smith, T.F., Waterman, M.S.: Identification of common molecular subsequences. Journal of Molecular Biology 147, 195-197 (1981)
20. Sutinen, E., Tarhio, J.: On using q-gram locations in approximate string matching. In: Spirakis, P.G. (ed.) ESA 1995. LNCS, vol. 979, pp. 327-340. Springer, Heidelberg (1995)
21. Ukkonen, E.: Approximate string matching with q-grams and maximal matches. Theoretical Computer Science 92, 191-211 (1992)
22. Ullman, J.: A binary n-gram technique for automatic correction of substitution, deletion, insertion, and reversal errors in words. The Computer Journal 20, 141-147 (1977)
23. Winkler, W.E.: The state of record linkage and current research problems. In: Statistics of Income Division (1999)