Data collection and quality challenges in deep learning: a data-centric AI perspective
By Steven Euijong Whang, Yuji Roh, Hwanjun Song, Jae-Gil Lee. 2021
559 citations recorded by the source corpus.
Data-centric AI is at the center of a fundamental shift in software engineering where machine learning becomes the new software, powered by big data and computing infrastructure. Here, software engineering needs to be re-thought where data become a first-class citizen on par with code. One striking observation is that a significant portion of the machine learning process is spent on data preparation. Without good data, even the best machine learning algorithms cannot perform well. As a result, data-centric AI practices are now becoming mainstream. Unfortunately, many datasets in the real world are small, dirty, biased, and even poisoned. In this survey, we study the research landscape for data collection and data quality primarily for deep learning applications. Data collection is important because there is lesser need for feature engineering for recent deep learning approaches, but instead more need for large amounts of data. For data quality, we study data validation, cleaning, and integration techniques. Even if the data cannot be fully cleaned, we can still cope with imperfect data during model training using robust model training techniques. In addition, while bias and fairness have been less studied in traditional data management research, these issues become essential topics in modern machine learning applications. We thus study fairness measures and unfairness mitigation techniques that can be applied before, during, or after model training. We believe that the data management community is well poised to solve these problems.
Subfields: Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Evaluation & Benchmarks
- Tristan Thrush
- Douwe Kiela
- Angelina McMillan-Major
- Yacine Jernite
- Alexandra Sasha Luccioni
- Margaret Mitchell
- Nazneen Rajani
- Mohit Bansal
Citations in the corpus
- Measuring Data (2022, background): “This has resulted in proposals for more systematic approaches to data collection (Whang et al., 2021), data-centric model training (Motamedi et al.”
- Data-centric Artificial Intelligence: A Survey (2023, unknown): “Recently, the role of data in AI has been significantly magnified, giving rise to the emerging concept of data-centric AI [108, 109, 183, 250, 268].”
- Data-Centric AI (2024, unknown): “In Whang et al. (2023), the authors analyze fairness measures and inequity mitigation techniques that can be applied before, during or after model training.”