Real-world data is dirty: Data cleansing and the merge/purge problem

Data Mining and Knowledge Discovery

Volumn 2, Issue 1, 1998, Pages 9-37

  Hernández, Mauricio A ^a   Stolfo, Salvatore J ^a  

^a Columbia University ^*  (United States)

Author keywords

Data cleaning; Data cleansing; Duplicate elimination; Semantic integration

Indexed keywords

EID: 0013331361     PISSN: 13845810     EISSN: None     Source Type: Journal    
DOI: 10.1023/A:1009761603038     Document Type: Article

References (23)

1. ACM. SIGMOD record, December 1991.
2. Agrawal, R. and Jagadish, H. V. Multiprocessor Transitive Closure Algorithms. In Proc. Int'l Symp. on Databases in Parallel and Distributed Systems, pages 56-66, December 1988.
3. Batini, C., Lenzerini, M. and Navathe, S. A Comparative Analysis of Methodologies for Database Schema Integration. ACM Computing Surverys, 18(4):323-364, December 1986.
4. Bitton, D. and DeWitt, D. J. Duplicate Record Elimination in Large Data Files. ACM Transactions on Database Systems, 8(2):255-265, June 1983.
5. Buckles, B.P. and Petry, F. E. A fuzzy representation of data for relational databases. Fuzzy Sets and Systems, 7:213-226, 1982. Generally regarded as the paper that originated Fuzzy Databases.
6. Buckley, J. P. A Hierarchical Clustering Strategy for Very Large Fuzzy Databases. In Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, pages 3573-3578, 1995.
7. Church, K. W. and Gale, W. A. Probability Scoring for Spelling Correction. Statistics and Computing, 1:93-103, 1991.
8. Clark, T. K. Analyzing Foster Childrens' Foster Home Payments Database. In KDD Nuggets 95:7 (http://info.gte.com/kdd/nuggets/95/), Piatetsky-Shapiro, ed., 1995.
9. Dietterich, T. and Michalski, R. A Comparative Review of Selected Methods for Learning from Examples. In R. Michalski, J. Carbonell, and T. Mitchell, editors, Machine Learning, volume 1, pages 41-81. Morgan Kaufmann Publishers, Inc., 1983.
10. Dubes, R. and Jain, A. Clustering Techniques: The User's Dilema. Pattern Recognition, 8:247-260, 1976.
11. Fayyad, U., Piatetsky-Shapiro, G. and Smyth, P. From Data Mining to Knowledge Discovery in Databases. AI Magazine, 17(3), Fall 1996.
12. Fellegi, I. and Sunter, A. A Theory for Record Linkage. American Statistical Association Journal, pages 1183-1210, December 1969.
13. Forgy, C. L. OPS5 User's Manual. Technical Report CMU-CS-81-135, Carnegie Mellon University, July 1981.
14. George, R., Petry, F. E., Buckles, B. P. and Srikanth, R. Fuzzy Database Systems - Challenges and Opportunities of a New Era. International Journal of Intelligent Systems, 11:649-659, 1996.
15. Ghandeharizadeh, S. Physical Database Design in Multiprocessor Database Systems. PhD thesis, Department of Computer Science, University of Wisconsin - Madison, 1990.
16. Hernández, M. and Stolfo, S. The Merge/Purge Problem for Large Databases. In Proceedings of the 1995 ACM-SIGMOD Conference, May 1995.
17. Kukich, K. Techniques for Automatically Correcting Words in Text. ACM Computing Surveys, 24(4):377-439, 1992.
18. Lebowitz, M. Not the Path to Perdition: The Utility of Similarity-Based Learning. In Proceedings of 5th National Conference on Artificial Intelligence, pages 533-537, 1986.
19. Monge, A. and Elkan, C. An Efficient Domain-independent Algorithm for Detecting Approximate Duplicate Database Records. In Proceedings of the 1997 SIGMOD Workshop on Research Issues on DMKD, pages 23-29, 1997.
20. Nyberg, C., Barclay, T., Cvetanovic, Z, Gray, J. and Lomet, D. Alpha Sort: A RISC Machine Sort. In Proceedings of the 1994 ACM-SIGMOD Conference, pages 233-242, 1994.
21. Pollock, J. J. and Zamora, A. Automatic spelling correction in scientific and scholarly text. ACM Computing Surveys, 27(4):358-368, 1987.
22. Senator, T., Goldberg, H., Wooton, J., Cottini, A., Umar, A., Klinger, C., Llamas, W. Marrone, M. and Wong, R. The FinCEN Artificial Intelligence System: Identifying Potential Money Laundering from Reports of Large Cash Transactions. In Proceedings of the 7th Conference on Innovative Applications of AI, August 1995.
23. Wang, Y. R. and Madnick, S. E. The Inter-Database Instance Identification Problem in Integrating Autonomous Systems. In Proceedings of the Sixth International Conference on Data Engineering, February 1989.