The Challenges of Data Quality and Data Quality Assessment in the Big Data Era

By Li Cai, Yangyong Zhu. 2015

903 citations recorded by the source corpus.

High-quality data are the precondition for analyzing and using big data and for guaranteeing the value of the data. Currently, comprehensive analysis and research of quality standards and quality assessment methods for big data are lacking. First, this paper summarizes reviews of data quality research. Second, this paper analyzes the data characteristics of the big data environment, presents quality challenges faced by big data, and formulates a hierarchical data quality framework from the perspective of data users. This framework consists of big data quality dimensions, quality characteristics, and quality indexes. Finally, on the basis of this framework, this paper constructs a dynamic assessment process for data quality. This process has good expansibility and adaptability and can meet the needs of big data quality assessment. The research results enrich the theoretical scope of big data and lay a solid foundation for the future by establishing an assessment model and studying evaluation algorithms.

Subfields: Information Systems and Management, Management Information Systems, Management Science and Operations Research, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems and Management

1. Irene Solaiman
2. Charith Perera
3. Athanasios V. Vasilakos

In Management Information Systems

1. Irene Solaiman
2. Charith Perera
3. Mark A. Girolami

In Management Science and Operations Research

1. Irene Solaiman
2. Charith Perera
3. Mohamed Adel Serhani
4. Mark A. Girolami
5. Chris J. Oates
6. Theodore Papamarkou

In Evaluation & Benchmarks

1. Irene Solaiman
2. Athanasios V. Vasilakos
3. Mohamed Adel Serhani
4. Mark A. Girolami

Citations in the corpus

• The Gradient of Generative AI Release: Methods and Considerations (2023, background): “The size of the system and its components also determine auditability; the datasets that large generative AI systems are trained on are not only difficult to analyze at scale, but few tools exist to analyze large static datasets [19].”
• Machine learning and structural health monitoring overview with emerging technology and high-dimensional data source highlights (2021, background): “37,79 In the context of SHM and big data, fi ve distinct quality standard indicators, 80 namely: (1) availability, (2) usability, (3) reliability, (4) relevance, and (5) presentation-quality, must be taken into consideration before any SHM implementation.”
• Big Data Quality: A Quality Dimensions Evaluation (2016, background): “In [1], [4], [6], [7] Data quality assessment was discussed early in literature as in [8] where they divide data quality assessment into two main categories: subjective and objective.”
• Fog Computing for Sustainable Smart Cities (2017, background): “For example a temperature reading gathered from a trusted high quality sensor is likely very good, while a derived conclusion of the wereabouts of a person based on the last registration to a cell phone tower from a while ago is probably not too reliable [Cai and Zhu 2015].”