BigDansing: A system for big data cleansing

Proceedings of the ACM SIGMOD International Conference on Management of Data

Volumn 2015-May, Issue , 2015, Pages 1215-1230

  Khayyaty, Zuhair ^c   Ilyasz, Ihab F ^d   Jindal, Alekh ^b   Madden, Samuel ^b   Ouzzani, Mourad ^a   Papotti, Paolo ^a   Quiané Ruiz, Jorge Arnulfo ^a   Tang, Nan ^a   Yin, Si ^a  

^a QATAR COMPUTING RESEARCH INSTITUTE   (Qatar)

^b MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^c KING ABDULLAH UNIVERSITY OF SCIENCE AND TECHNOLOGY   (Saudi Arabia)

^d UNIVERSITY OF WATERLOO   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

DATA HANDLING;

BASELINE SYSTEMS; DISTRIBUTED COMPUTATIONS; DISTRIBUTED PLATFORMS; ORDERS OF MAGNITUDE; PROCESSING PLATFORM; PROGRAMMING INTERFACE; REPAIR ALGORITHMS; USER DEFINED FUNCTIONS;

BIG DATA;

EID: 84949872769     PISSN: 07308078     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2723372.2747646     Document Type: Conference Paper

References (41)

1. Shark (Hive on Spark). https://github.com/amplab/shark.
2. TPC-H benchmark version 2.14.4. http://www.tpc.org/tpch/.
3. C. Batini and M. Scannapieco. Data Quality: Concepts, Methodologies and Techniques. Springer, 2006.
4. G. Beskales, I. F. Ilyas, and L. Golab. Sampling the repairs of functional dependency violations under hard constraints. PVLDB, 3:197-207, 2010.
5. P. Bohannon, M. Flaster, W. Fan, and R. Rastogi. A cost-based model and effective heuristic for repairing constraints by value modification. In SIGMOD, 2005.
6. X. Chu, I. F. Ilyas, and P. Papotti. Holistic Data Cleaning: Putting Violations into Context. In ICDE, 2013.
7. M. Dallachiesa, A. Ebaid, A. Eldawy, A. Elmagarmid, I. F. Ilyas, M. Ouzzani, and N. Tang. NADEEF: A Commodity Data Cleaning System. In SIGMOD, 2013.
8. J. Dean and S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. Communications of the ACM, 51(1):107-113, 2008.
9. D. J. DeWitt, J. F. Naughton, and D. A. Schneider. An evaluation of non-equijoin algorithms. In VLDB, 1991.
10. W. Fan and F. Geerts. Foundations of Data Quality Management. Morgan & Claypool, 2012.
11. W. Fan, F. Geerts, X. Jia, and A. Kementsietsidis. Conditional Functional Dependencies for Capturing Data Inconsistencies. ACM Transactions on Database Systems (TODS), 33(2):6:1-6:48, 2008.
12. W. Fan, F. Geerts, N. Tang, and W. Yu. Conict resolution with data currency and consistency. J. Data and Information Quality, 5(1-2):6, 2014.
13. W. Fan, J. Li, S. Ma, N. Tang, and W. Yu. Interaction between record matching and data repairing. In SIGMOD, 2011.
14. W. Fan, J. Li, N. Tang, and W. Yu. Incremental Detection of Inconsistencies in Distributed Data. In ICDE, 2012.
15. I. Fellegi and D. Holt. A systematic approach to automatic edit and imputation. J. American Statistical Association, 71(353), 1976.
16. T. Friedman. Magic quadrant for data quality tools. http://www.gartner.com/, 2013.
17. F. Geerts, G. Mecca, P. Papotti, and D. Santoro. The LLUNATIC Data-Cleaning Framework. PVLDB, 6(9):625-636, 2013.
18. F. Geerts, G. Mecca, P. Papotti, and D. Santoro. Mapping and Cleaning. In ICDE, 2014.
19. M. Interlandi and N. Tang. Proof positive and negative in data cleaning. In ICDE, 2015.
20. E. Jahani, M. J. Cafarella, and C. Ré. Automatic optimization for mapreduce programs. PVLDB, 4(6):385-396, 2011.
21. A. Jindal, J.-A. Quiané-Ruiz, and S. Madden. Cartilage: Adding Flexibility to the Hadoop Skeleton. In SIGMOD, 2013.
22. G. Karypis and V. Kumar. Multilevel K-way Hypergraph Partitioning. In Proceedings of the 36th Annual ACM/IEEE Design Automation Conference, DAC. ACM, 1999.
23. S. Kolahi and L. V. S. Lakshmanan. On Approximating Optimum Repairs for Functional Dependency Violations. In ICDT, 2009.
24. L. Kolb, A. Thor, and E. Rahm. Dedoop: Efficient Deduplication with Hadoop. PVLDB, 2012.
25. G. Malewicz, M. H. Austern, A. J. Bik, J. C. Dehnert, I. Horn, N. Leiser, and G. Czajkowski. Pregel: A System for Large-scale Graph Processing. In SIGMOD, 2010.
26. C. Mayfield, J. Neville, and S. Prabhakar. ERACER: a database approach for statistical inference and data cleaning. In SIGMOD, 2010.
27. A. Okcan and M. Riedewald. Processing theta-joins using mapreduce. In SIGMOD, 2011.
28. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig Latin: A Not-so-foreign Language for Data Processing. In SIGMOD, 2008.
29. E. Rahm and H. H. Do. Data cleaning: Problems and current approaches. IEEE Data Eng. Bull., 23(4):3-13, 2000.
30. G. Smith. PostgreSQL 9.0 High Performance: Accelerate your PostgreSQL System and Avoid the Common Pitfalls that Can Slow it Down. Packt Publishing, 2010.
31. N. Swartz. Gartner warns firms of 'dirty data'. Information Management Journal, 41(3), 2007.
32. A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, S. Anthony, H. Liu, P. Wyckoff, and R. Murthy. Hive: A Warehousing Solution over a Map-reduce Framework. PVLDB, 2(2):1626-1629, 2009.
33. Y. Tong, C. C. Cao, C. J. Zhang, Y. Li, and L. Chen. CrowdCleaner: Data cleaning for multi-version data on the web via crowdsourcing. In ICDE, 2014.
34. M. Volkovs, F. Chiang, J. Szlichta, and R. J. Miller. Continuous data cleaning. In ICDE, 2014.
35. J. Wang, S. Krishnan, M. J. Franklin, K. Goldberg, T. Kraska, and T. Milo. A Sample-and-Clean Framework for Fast and Accurate Query Processing on Dirty Data. In SIGMOD, 2014.
36. J. Wang and N. Tang. Towards dependable data repairing with fixing rules. In SIGMOD, 2014.
37. R. S. Xin, J. E. Gonzalez, M. J. Franklin, and I. Stoica. GraphX: A Resilient Distributed Graph System on Spark. In First International Workshop on Graph Data Management Experiences and Systems, GRADES. ACM, 2013.
38. R. S. Xin, J. Rosen, M. Zaharia, M. J. Franklin, S. Shenker, and I. Stoica. Shark: SQL and Rich Analytics at Scale. In SIGMOD, 2013.
39. M. Yakout, L. Berti-Equille, and A. K. Elmagarmid. Don't be SCAREd: use SCalable Automatic REpairing with maximal likelihood and bounded changes. In SIGMOD, 2013.
40. M. Yakout, A. K. Elmagarmid, J. Neville, M. Ouzzani, and I. F. Ilyas. Guided data repair. PVLDB, 2011.
41. M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica. Spark: Cluster computing with working sets. In HotCloud, 2010.