Data de-duplication: A review

Studies in Computational Intelligence

Volumn 375, Issue , 2011, Pages 385-412

  Costa, Gianni ^a   Cuzzocrea, Alfredo ^a   Manco, Giuseppe ^a   Ortale, Riccardo ^a  

^a ICAR CNR   (Italy)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 80455127017     PISSN: 1860949X     EISSN: None     Source Type: Book Series    
DOI: 10.1007/978-3-642-22913-8_18     Document Type: Review

References (99)

1. Agichtein, E., Ganti, V.:Mining Reference Tables for Automatic Text Segmentation. Proc. of ACM SIGKDD Int. Conf. On Knowledge Discovery and Data Mining, Seattle, Washington, USA, pp. 20-29 (2004)
2. Ananthakrishna, R., Chaudhuri, S., Ganti, V.: Eliminating Fuzzy Duplicates in Data Warehouses. Proc. of Int. Conf. on Very Large Databases, Hong Kong, China, pp. 586-597 (2002)
3. Andoni, A., Indyk, P.: Near-Optimal Hashing Algorithms for Approximate Nearest Neighbor in High Dimensions. Proc. of IEEE Symposium on Foundations of Computer Science, Las Vegas, Nevada, USA, pp. 459-468 (2006)
4. Andoni, A., Indyk, P.: Near-optimal Hashing Algorithms for Approximate Nearest Neighbor in High Dimensions. Communications of the ACM 51(1), 117-122 (2008)
5. Arasu, A., Ganti, V., Kaushik, R.: Efficient Exact Set-Similarity Joins. Proc. of Int. Conf. on Very Large Databases, Seoul, Korea, pp. 918-929 (2006)
6. Axford, S.J., Newcombe, H.B., Kennedy, J.M., James, A.P.:Automatic Linkage of Vital Records. Science 130, 954-959 (1959)
7. Baeza-Yates, R., Ribeiro-Neto, B.: Modern Information Retrieval. Addison-Wesley, Reading (1999)
8. Bansal, N., Blum, A., Chawla, S.: Correlation Clustering. Machine Learning 56(1-3), 89-113 (2004)
9. Bawa, M., Tyson Condie, S., Ganesan, P.: LSH Forest: Self-Tuning Indexes for Similarity Search. Proc. of Int. Conf. on World Wide Web, Chiba, Japan, pp. 651-660 (2005)
10. Bayardo, R.J., Srikant, R., Ma, Y.: Scaling Up All Pairs Similarity Search. Proc. of Int. Conf. on World Wide Web, Banff, Alberta, Canada, pp. 131-140 (2007)
11. Benjelloun, O., Garcia-Molina, H., Menestrina, D., Su, Q., Whang, S.E., Widom, J.: Swoosh: a generic approach to entity resolution. VLDB Journal 18(1), 255-276 (2009)
12. Berson, T.A.: Differential Cryptanalysis Mod 232 with Applications to MD5. Proc. of Ann. Conf. on Theory and Applications of Cryptographic Techniques, pp. 71-80 (1992)
13. Bhattacharya, I., Getoor, L.: Collective Entity Resolution in Relational Data. ACM Trans. Knowl. Discovery from Data 1(1), 1-35 (2007)
14. Bhattacharya, I., Getoor, L., Licamele, Louis: QueryTime Entity Resolution. Proc. of ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, Philadelphia, Pennsylvania, USA, pp. 529-534 (2006)
15. Bilenko, M., Mooney, R.J.: Adaptive Duplicate Detection Using Learnable String Similarity Measures. Proc. of ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, Washington, DC, USA, pp. 39-48 (2003)
16. Christen, P.: Towards Parameter-free Blocking for Scalable Record Linkage. Tech. Rep. TR-CS-07-03, Australian National University, Canberra, Australia (2007)
17. Christen, P.: Febrl - An Open Source Data Cleaning, Deduplication and Record Linkage System with a Graphical User Interface. Proc. of ACM Int. Conf. on Knowledge Discovery and Data Mining, pp. 1065-1068 (2008)
18. Borkar, V.R., Deshmukh, K., Sarawagi, S.: Automatic Segmentation of Text into Structured Records. Proc. of ACM SIGMOD Int. Conf. on Management of Data, Santa Barbara, California, USA, pp. 175-186 (2001)
19. Broder, A., Charikar, M., Frieze, A.M., Mitzenmacher, M.: Minwise Independent Permutations. Proc. of ACMSymposium on Theory of Computing, Dallas, Texas, USA, pp. 327-336 (1998)
20. Broder, A., Glassman, S., Manasse, M., Zweig, G.: Syntactic Clustering on theWeb. Proc. of Int. Conf. on World Wide Web, Santa Clara, California, USA, pp. 1157-1166 (1997)
21. Cesario, E., Folino, F., Locane, A., Manco, G., Ortale, R.: Boosting Text Segmentation Via Progressive Classification. Knowl. and Inf. Syst. 15(3), 285-320 (2008)
22. Chaudhuri, S., Ganjam, K., Ganti, V., Motwani, R.: Robust and Efficient Fuzzy Match for Online Data Cleaning. Proc. of ACM SIGMOD Conf. on Management of Data, San Diego, California, USA, pp. 313-324 (2003)
23. Chaudhuri, S., Ganti, V., Motwani, R.: Robust Identification of Fuzzy Duplicates. Proc. of Int. Conf. on Data Engineering, Tokyo, Japan, pp. 865-876 (2005)
24. Chavez, E., Navarro, G., Baeza-Yates, R.,Marroquin, J.L.: Searching in Metric Spaces. ACM Comput. Surv. 33(3), 273-321 (2001)
25. Ciaccia, P., Patella, M., Zezula, P.: M-Tree: An Efficient Access Method for Similarity Search in Metric Spaces. Proc. of Int. Conf. on Very Large Databases, Athens, Greece, pp. 426-435 (1997)
26. Cochinwala, M., Dalal, S., Elmagarmid, A.K., Verykios, V.S.: Record Matching: Past, Present and Future. Technical Report, number CSD-TR #01-013. Department of Computer Sciences, Purdue University (2001)
27. Cochinwala, M., Kurien, V., Lalk, G., Shasha, D.: Efficient Data Reconciliation. Information Sciences 137(1-4), 1-15 (2001)
28. Cohen, W.W.: Data Integration using Similarity Joins and a Word-based Information Representation Language. ACM Trans. on Inf. Syst. 18(3), 228-321 (2000)
29. Cohen, W.W., Ravikumar, P., Fienberg, S.E.: A Comparison of String Distance Metrics for Name-Matching Tasks. Proc. of IJCAI Workshop on Information Integration on the Web, Acapulco, Mexico, pp. 73-78 (2003)
30. Cohen, W.W., Richman, J.: Learning to Match and Cluster Large High-Dimensional Data Sets for Data Integration. Proc. of ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, Edmonton, Alberta, Canada, pp. 475-480 (2002)
31. Cohn, D.A., Atlas, L., Ladner, R.E.: Improving Generalization with Active Learning. Machine Learning 15(2), 201-221 (1994)
32. Costa, G., Manco, G., Ortale, R.: An Incremental Clustering Scheme for Data Deduplication. Data Min. and Knowl. Discovery 20(1), 152-187 (2010)
33. Database Group Leipzig. Benchmark datasets for entity resolution, http://dbs.uni-leipzig.de/en/research/projects/objectmatching/fever/benchmark datasets for entity resolution
34. Dempster, A.P., Laird, N.M., Rubin, D.B.:Maximum Likelihood from Incomplete Data via the EM Algorithm. Journal of the Royal Statistical Society, Series B 39(1), 1-28 (2001)
35. Dittrich, J.-P., Seeger, B.: Data Redundancy and Duplicate Detection in Spatial Join Processing. Proc. of IEEE Int. Conf. on Data Engineering, pp. 535-546 (2000)
36. Elmagarmid, A.K., Ipeirotis, P.G., Verykios, V.S.: Duplicate Record Detection: A Survey. IEEE Transanctions on Knowledge and Data Engineering 19(1), 1-16 (2007)
37. Ester, M., Kriegel, H.P., Sander, J., Xu, X.: A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. Proc. of Int. Conf. on Knowledge Discovery and Data Mining, Portland, Oregon, USA, pp. 226-231 (1996)
38. Fayyad, U., Piatetsky-Shapiro, G., Smyth, P., Widener, T.: The KDD Process for Extracting Useful Knowledge from Volumes of Data. Communications of the ACM 39(11), 27-34 (1996)
39. Fellegi, I.P., Sunter, A.B.: A Theory for Record Linkage. Am. Stat. Assoc. 64, 1183-1210 (1969)
40. Ganti, V., Ramakrishnan, R., Gehrke, J., Powell, A.: Clustering Large Datasets in Arbitrary Metric Spaces. Proc. of Int. Conf. on Data Engineering, Sydney, Austrialia, pp. 502-511 (1999)
41. Garcia-Molina, H.: Entity resolution: Overview and challenges. In: Atzeni, P., Chu, W., Lu, H., Zhou, S., Ling, T.-W. (eds.) ER 2004. LNCS, vol. 3288, pp. 1-2. Springer, Heidelberg (2004)
42. Gilks,W.R., Richardson, S., Spiegelhalter, D.J.:Markov ChainMonte Carlo in Practice. Chapman and Hall, Boca Raton (1996)
43. Gionis, A., Indyk, P., Motwani, R.: Similarity Search in High Dimensions via Hashing. Proc. of Int. Conf. on Very Large Databases, Edinburgh, Scotland, pp. 518-529 (1999)
44. Goiser, K., Christen, P.: Towards Automated Record Linkage. Proc. of Australasian Data Mining Conf., pp. 23-31 (2006)
45. Grabmeier, J., Rudolph, A.: Techniques of Cluster Algorithms in Data Mining. Data Min. and Knowl. Discovery 6(4), 303-360 (2002)
46. Gravano, L., Ipeirotis, P.G., Jagadish, H.V., Koudas, N., Muthukrishnan, S., Srivastava, D.: Approximate String Joins in a Database (Almost) for Free. In: Proc of Int. Conf. On Very Large Databases, Rome, Italy, pp. 491-500 (2001)
47. Gu, L., Baxter, R.A., Vickers, D., Rainsford, C.: Record Linkage: Current Practice and Future Directions. Technical Report, number 03/83. CSIRO Mathematical and Information Sciences (2001)
48. Guha, S., Rastogi, R., Shim, K.: CURE: An Efficient Clustering Algorithm for Large Databases. Proc. of ACM SIGMOD Int. Conf. on Management of Data, Seattle, Washington, USA, pp. 73-84 (1998)
49. Guha, S., Rastogi, R., Shim, K.: ROCK: A Robust Clustering Algorithm for Categorical Attributes. Inf. Syst. 25(5), 345-366 (2001)
50. Gunsfield, D.: Algorithms on Strings, Trees and Sequences. Cambridge University Press, Davis (1997)
51. Hassanzadeh, O., Chiang, F., Lee, H.C.,Miller, R.J.: Framework for Evaluating Clustering Algorithms in Duplicate Detection. Proceedings of VLDB 2(1), 1282-1293 (2009)
52. Hassanzadeh, O., Miller, R.J.: Creating Probabilistic Databases from Duplicated Data. The VLDB Journal 18(5), 1141-1166 (2009)
53. Hernández, M.A., Stolfo, S.J.: The Merge/Purge Problem for Large Databases. Proc. of ACM SIGMOD Int. Conf. on Management of Data, San Jose, California, USA, pp. 127-138 (1995)
54. Hernández, M.A., Stolfo, J.: Real-world Data is Dirty: Data Cleansing and the Merge/Purge Problem. Data Min. and Knowl. Discovery 2(1), 9-37 (1998)
55. Herschel, M., Naumann, N.: Scaling up Duplicate Detection in Graph Data. Proc. of ACM Int. Conf. on Information and Knowledge Management, pp. 1325-1326 (2008)
56. Hjatason, G.R., Samet, H.: Index-Driven Similarity Search in Metric Spaces. ACM Trans. on Database Syst. 28(4), 517-518 (2003)
57. Indyk, P.,Motwani, R.: Approximate Nearest Neighbor - Towards Removing the Curse of Dimensionality. Proc. of Symposium on Theory of Computing, Dallas, Texas, USA, pp. 604-613 (1998)
58. Ipeirotis, P.G., Verykios, V.S., Elmagarmid, A.K.: Duplicate Record Detection: A urvey. IEEE Trans. Knowl. Data Eng. 19(1), 1-16 (2007)
59. Jain, A.K., Dubes, R.C.: Algorithms for Clustering Data. Prentice-Hall, Englewood Cliffs (1998)
60. Jain, A.K., Murty, M.N., Flynn, P.J.: Data Clustering: A Review. ACM Comput. Surv. 31(3), 264-323 (1999)
61. Jaro, M.A.: Advances in Record Linkage Methodology as Applied to Matching the 1985 Census of Tampa, Florida. Journal of the American Statistical Society 84, 420-424 (1989)
62. Kingsbury, N.R., et al.: Record Linkage and Privacy: Issues in Creating New Federal Research and Statistical Information. U.S. General Accounting Office (2001)
63. Kopcke, H., Rahm, E.: Frameworks for Entity Matching: A Comparison Data and Know. Engineering 69(2), 197-210 (2010) 64. Kopcke, H., Thor, A., Rahm, E.: Evaluation of entity resolution approaches on realworld match problems. Proc. of the VLDB Endowment 3(1), 484-493 (2010)
64. Kopcke, H., Thor, A., Rahm, E.: Evaluation of Learning-Based Approaches for Matching Web Data Entities. IEEE Internet Computing 14(4), 23-31 (2010)
65. McCallum, A.: MALLET: A Machine Learning for Language Toolkit, http://mallet.cs.umass.edu
66. Koudas, N., Sevcik, K.C.: Size Separation Spatial Join. Proc. of ACM Int. Conf. On Management of Data, pp. 324-335 (1997)
67. Lawrence, S., Bollacker, K., Giles, C.L.: Autonomous Citation Matching. Proc. of ACM Int. Conf. on Autonomous Agents, pp. 392-393 (1999)
68. Low, W.L., Lee, M.L., Ling, T.W.: A Knowledge-Based Approach for Duplicate Elimination in Data Cleaning. Information Systems 26(8), 585-606 (2001)
69. Lwin, T., Nyunt, T.T.S.: An Efficient Duplicate Detection System for XML Documents. Proc. of IEEE Int. Conf. on Computer Engineering and Applications, pp. 178-182 (2010)
70. McCallum, A., Freitag, D., Pereira, F.: Maximum Entropy Markov Models for Information Extraction and Segmentation. Proc. of Int. Conf. on Machine Learning, Standord, California, USA, pp. 591-598 (2000)
71. McCallum, A., Nigam, K., Ungar, L.: Efficient Clustering of High-Dimensional Data Sets with Application to Reference Matching. Proc. of ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, Boston, Massachusetts, USA, pp. 169-178 (2000)
72. Menestrina, D., Benjelloun, O., Garcia-Molina, H.: Generic Entity Resolution with Data Confidences. In: Int. VLDB Workshop on Clean Databases, Seoul, Korea (2006)
73. Mitchell, T.M.: Machine Learning. McGraw-Hill, New York (1997)
74. Monge, A.E., Elkan, C.P.: An Efficient Domain-Independent Algorithm For Detecting Approximately Duplicate Database Records. Proc. of SIGMOD Workshop on Research Issues on Data Mining and Knowledge Discovery, Tucson, Arizona, USA, pp. 23-29 (1997)
75. Monge, A.E., Elkan, C.P.: The Field Matching Problem: Algorithms and Applications. Proc. of Int. Conf. on Knowledge Discovery and Data Mining, Portland, Oregon, USA, pp. 267-270 (1996)
76. Mukherjee, S., Ramakrishnan, I.V.: Taming the Unstructured: Creating Structured Content from Partially Labeled Schematic Text Sequences. Proc. of CoopIS/DOA/ODBASE Int. Conf., Agia Napa, Cyprus, pp. 909-926 (2004)
77. Muse, A.G., Mikl, J., Smith, P.F.: Evaluating the quality of anonymous record linkage using deterministic procedures with the New York State AIDS registry and a hospital discharge file. Statistics in Medicine 14, 499-509 (1995)
78. Neiling, M., Jurk, S.: The Object Identification Framework. In: Proc. KDD Workshop on Data Cleaning, Record Linkage, and Object Consolidation, Washington, DC, USA, pp. 37-39 (2003)
79. Neutel, C.I.: Privacy Issues in Research Using Record Linkage. Pharmcoepidemiology and Drug Safety 6, 367-369 (1997)
80. Newcombe, H.B.: Record Linking: The Design of Efficient Systems for Linking Records into Individual and Family Histories. American Journal of Human Genetics 19, 335-359 (1967)
81. Newcombe, H.B., Kennedy, J.M., Axford, S.J., James, A.P.:Automatic Linkage of Vital Records. Science 130, 954-959 (1959)
82. Patel, J., DeWitt, D.J.: Partition Based Spatial-Merge Join. Proc. of ACM Int. Conf. on Management of Data, pp. 259-270 (1996)
83. Pasula, H., Marthi, B., Milch, B., Russell, S.J., Shpitser, I.: Identity Uncertainty and Citation Matching. Proc. of Ann. Conf. on Neural Information Processing Systems, pp. 1401-1408 (2002)
84. Sarawagi, S., Bhamidipaty, A.: Interactive Deduplication using Active Learning. Proc. of ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, Edmonton, Alberta, Canada, pp. 269-278 (2002)
85. Sarawagi, S., Kirpal, A.: Efficient set joins on similarity predicates. Proc. of SIGMOD Int. Conf. on Management of Data, Paris, France, pp. 743-754 (2004)
86. Shen, H., Zhang, Y.: Improved Approximate Detection of Duplicates for Data Streams over Sliding Windows. Journal of Computer Science and Technology 23(6), 973-987 (2008)
87. Singla, P., Domingos, P.: Multi-Relational Record Linkage. Proc. of ACM Int. Ws. on Multi-Relational Data Mining, pp. 31-38 (2004)
88. Smith, S., Waterman, M.S.: Identification of Common Molecular Subsequences. Journal of Molecular Biology 147(1), 195-197 (1981)
89. Statistical Linkage Key Working Group. Statistical Data Linkage in Community Services Data Collections (2002)
90. Tejada, S., Knoblock, C.A.,Minton, S.: Learning Domain-Independent String Transformation Weights for High Accuracy Object Identification. Proc. of ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, Edmonton, Alberta, Canada, pp. 350-359 (2002)
91. Tepping, J.B.: A Model for Optimum Linkage of Records. Journal of the American Statistical Association 63, 1321-1332 (1968)
92. Verykios, V.S., Elmagarmid, A.K., Houstis, E.N.: Automating the approximate recordmatching process. Inf. Sci. 126(1-4), 83-98 (2000)
93. Weber, R., Schek, H.J., Blott, S.: A Quantitative Analsysis and Performance Study for Similarity Search in High-Dimensional Spaces. Proc. of Int. Conf. on Very Large Databases, New York City, USA, pp. 194-205 (1998)
94. Weis,M., Naumann, N.: Detecting Duplicates in Complex XML Data. Proc. of IEEE Int. Conf. on Data Engineering, p. 109 (2006)
95. Weis, M., Naumann, N.: Space and Time Scalability of Duplicate Detection in Graph Data. Tech. Rep. 25, Hasso-Plattner Institut, Potsdam, Germany (2007)
96. Winkler,W.E.: String Comparator Metrics and Enhanced Decision Rules in the Fellegi- Sunter Model of Record Linkage. In: Proc. Section on Survey Research Methods, American Statistical Association, pp. 354-359 (1990)
97. Winkler, W.E.: Overview of Record Linkage and Current Research Directions. Technical Report. Statistical Research Division, U.S. Census Bureau (1999)
98. Winkler, W.E.: Methods for Record Linkage and Bayesian Networks. Tech. Rep. RRS2002/05, U.S. Bureau of the Census, Washington, D.C., USA (2002)
99. Zhang, Y., Lin, X., Yuan, Y., Kitsuregawa, M., Zhou, X., Yu, J.X.: Duplicate-insensitive Order Statistics Computation over Data Streams. IEEE Transanctions on Knowledge and Data Engineering 22(4), 493-507 (2010)