Big data validation and quality assurance - Issuses, challenges, and needs

Proceedings - 2016 IEEE Symposium on Service-Oriented System Engineering, SOSE 2016

Volumn , Issue , 2016, Pages 433-441

  Gao, Jerry ^a,b   Xie, Chunli ^c   Tao, Chuanqi ^d  

^a SAN JOSE STATE UNIVERSITY   (United States)

^b TAIYUAN UNIVERSITY OF TECHNOLOGY   (China)

^c JIANGSU NORMAL UNIVERSITY   (China)

^d NANJING UNIVERSITY OF SCIENCE AND TECHNOLOGY   (China)

Author keywords

Big data quality assurance; Big data validation; Data validation; Quality assurance

Indexed keywords

QUALITY ASSURANCE; SYSTEMS ENGINEERING;

ACADEMIC RESEARCH; DATA QUALITY; DATA TECHNOLOGIES; DATA VALIDATION; DATA VALIDATION TOOLS; GOVERNMENT SERVICES; RESEARCH AND APPLICATION; VALIDATION PROCESS;

BIG DATA;

EID: 84978961960     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/SOSE.2016.63     Document Type: Conference Paper

References (28)

1. Steve Sarsfield, "How a Small Data Error Becomes a Big Data Quality Problem", September 3, 2012. Retrievable at: https://www.linkedin.com/grp/post/103526-162333712.
2. M. R. Wigan and R. Clake, "Big Data's Big Unintended Consequences", IEEE Computer, Vol. 46, Issue 6, Pages:46-53, 2013.
3. D. M. Strong, Y. W. Lee, R. Y. Wang, "Data quality in context". Communications of the ACM, Vol.40, Issue 5, Pages: 103-110,1997.
4. L. L. Pipino, Y. W. Lee, R. Y. Wang, "Data quality assessment". Communications of the ACM, Vol.45, Issue 4, Pages: 211-218, 2002.
5. A. O. Mohammed, S. A. Talab, "Enhanced Extraction Clinical Data Technique to Improve Data Quality in Clinical Data Warehouse". International Journal of Database Theory and Application, Vol.8, Issue 3, Pages: 333-342, 2015.
6. M. I. Svanks, "Integrity analysis: Methods for automating data quality assurance". Information and Software Technology, Vol.30, Issue 10, Pages:595-605, 1988.
7. J. Alferes, P.Poirier, C. Lamaire-Chad, et al. "Data quality assurance in monitoring of wastewater quality: Univariate on-line and off-line methods". Proceedings of the11th IWA conference on instrumentation control and automation, 18-20, September,2013.
8. A.L.Ali and F.Schmid, "Data quality assurance for volunteered geographic information", Geographic Information Science. Springer International Publishing, Pages126-141,2014.
9. J. J. Gassman, et al. "Data quality assurance, monitoring, and reporting" Controlled clinical trials, Vol.16, Issue 2, Pages: 104-136,1995.
10. P. Woodall, et al. Classifying Data Quality Problems in Asset Management, Springer International Publishing, 2015.
11. N. Laranjeiro, et al, "A Survey on Data Quality: Classifying Poor Data", IEEE 21st Pacific Rim International Symposium on Dependable Computing (PRDC), Pages: 179-188, 2015.
12. Roger Clarke, "Quality Factors in Big Data and Big Data Analytics", Xamax Consultancy Pry Ltd. 2014.
13. EY,"Big Data: changing the way businesses operate", Retrive at: http://www.ey.com/Publication/vwLUAssets /EY-Big-data: changing- the way businesses-operate/ $FILE /EY-Insights-on-GRC-Big-data.pdf.
14. B. Razavi, Principles of data conversion system design, New York: IEEE press, 1995.
15. B. T. Hazen, et al. "Data quality for data science, predictive analytics, and big data in supply chain management: An introduction to the problem and suggestions for research and applications", International Journal of Production Economics, 154: 72-80. 2014.
16. SAS Institiute, "SaS: Big Data and Big Analytics for Product and Process Quality", White paper, October, 2011.
17. SAS Institue, "SaS: Understanding Big Data Quality for Maximum Infomration Usability", White paper.
18. W. W. Eckerson, "Data quality and the bottom line: Achieving business success through a commitment to high quality data". Data, Warehousing Institute, 2002.
19. J. W. Osbourne, "Notes on the Use of Data Transformation", Practical Assessment, Research & Evaluation, 2002, 8(6): n6.
20. I. Taleb, R. Dssouli R, and M. A. Serhani, "Big Data Pre-processing: A Quality Framework", IEEE International Congress on Big Data, Pages: 191.
21. A. Immonen, P. Paakkonen, and E. Ovaska, "Evaluating the Quality of Social Media Data in Big Data Architecture". IEEE Access, Volume 3, Pages: 2028-2043 October 16, 2015.
22. N. Meredith, W. Zozus, Ed Hammond, Beverly B. Green, and et al. "Assessing Data Quality for Healthcare Systems Data Used in Clinical Research (Version 1.0)".
23. M. Gudipati, S. Rao, N. D.Mohan and N. K. Gajja,, "Big Data: Testing Approach to Overcome Quality Challenges", Inforsys Labs Briefings, Vol. 11, No. 1, pp. 65-73, 2013.
24. The data quality benchmark report-Experian Data Quality, 2016.
25. New Jersey Department of Environmental Protection, Data Quality Assessment and Data Usability Evaluation Technical Guidance, April 2014.
26. David Loshin (Author), The Practitioner's Guide to Data Quality Improvement (The Morgan Kaufmann Series on Business Intelligence), 1st Edition, Morgan Kaufmann; 1 edition, October 29, 2010.
27. Katarina Grolinger et al, "Challenges for MapReduce in Big Data", IEEE World Congress on Services (SERVICES), Pages: 182-189, Anchorage, AK, June, 2014.
28. L.Cai and Y. Zhu, "The Challenges of Data Quality and Data Quality Assessment in the Big Data Era". Data Science Journal, Vol.14, Issue 2, Pages:97-181, 2015.