Automating Large-Scale Data Quality Verification

By Sebastian Schelter, Dustin Lange, Philipp Schmidt, Meltem Celikel, Felix Biessmann, Andreas Grafberger. Proceedings of the VLDB Endowment, 2018

296 citations recorded by the source corpus.

Modern companies and institutions rely on data to guide every single business process and decision. Missing or incorrect information seriously compromises any decision process downstream. Therefore, a crucial, but tedious task for everyone involved in data processing is to verify the quality of their data. We present a system for automating the verification of data quality at scale, which meets the requirements of production use cases. Our system provides a declarative API, which combines common quality constraints with user-defined validation code, and thereby enables 'unit tests' for data. We efficiently execute the resulting constraint validation workload by translating it to aggregation queries on Apache Spark. Our platform supports the incremental validation of data quality on growing datasets, and leverages machine learning, e.g., for enhancing constraint suggestions, for estimating the 'predictability' of a column, and for detecting anomalies in historic data quality time series. We discuss our design decisions, describe the resulting system architecture, and present an experimental evaluation on various datasets.

Subfields: Computer Networks and Communications, Management Information Systems, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/3606af912be399c789c40c1a1a5f16bd9bb84b6e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Management Science and Operations Research

Citations in the corpus

The effects of data quality on machine learning performance on tabular data (2022, unknown): “Furthermore, some studies presented a vision of data quality assessment tools [29], an automation of data quality verification [62] or a methodology to summarize the quality of a dataset as datasheets [24], nutritional labels [70], and data cards [71].”
Machine Learning Testing: Survey, Landscapes and Horizons (2019, background): “[181] focus on the automatic ‘unit’ testing of large-scale datasets.”
Towards Automated Data Cleaning Workflows (2019, unknown): “Current techniques require user-defined algorithm parameters, such as rules or thresholds, which are not straightforward to select by a data practitioner [26].”
Data collection and quality challenges in deep learning: a data-centric AI perspective (2021, background): “More recently, data validation systems are equipped with additional functionalities including declarative abstractions [55,115], automatic identification of error types [105], testing the impact of errors on models [116],”
The Eﬀects of Data Quality on Machine Learning Performance (2022, background): “Furthermore, some studies presented a vision of data quality assessment tools [30], an automation of data quality veriﬁcation [63] or a methodology to summarize the quality of a dataset as datasheets [25], nutritional labels [69], and data cards [70].”
Garbage in, garbage out?: do machine learning application papers in social computing report where human-labeled training data comes from? (2019, unknown): “There are also intersecting literatures on systems for capturing information in ML data flows and supply chains [17, 55, 60], as well as supporting data cleaning [32, 56].”
Data-centric Artificial Intelligence: A Survey (2023, unknown): “Furthermore, pipeline automation approaches have been developed to systematically curate data in multiple stages of the data pipeline, such as data integration and data cleaning [194, 220].”