A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective

By Yuji Roh, Geon Heo, Steven Euijong Whang. 2018

815 citations recorded by the source corpus.

Data collection is a major bottleneck in machine learning and an active research topic in multiple communities. There are largely two reasons data collection has recently become a critical issue. First, as machine learning is becoming more widely-used, we are seeing new applications that do not necessarily have enough labeled data. Second, unlike traditional machine learning, deep learning techniques automatically generate features, which saves feature engineering costs, but in return may require larger amounts of labeled data. Interestingly, recent research in data collection comes not only from the machine learning, natural language, and computer vision communities, but also from the data management community due to the importance of handling large amounts of data. In this survey, we perform a comprehensive study of data collection from a data management point of view. Data collection largely consists of data acquisition, data labeling, and improvement of existing data or models. We provide a research landscape of these operations, provide guidelines on which technique to use when, and identify interesting research challenges. The integration of machine learning and data management for data collection is part of a larger trend of Big data and Artificial Intelligence (AI) integration and opens many opportunities for new research.

Subfields: Artificial Intelligence, Computer Science Applications, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Evaluation & Benchmarks

Citations in the corpus

AutoML: A Survey of the State-of-the-Art (2019, methodology): “Roh et al. [61] have summarized the self-labeling methods into the following categories: self-training [62], [63], co-training [64], [65], and co-learning [66].”
Data collection and quality challenges in deep learning: a data-centric AI perspective (2021, background): “Our coverage of data collection originates from a survey [133] by two of the authors, so we keep it brief with new updates based on a tutorial [169].”
A Unifying Review of Deep and Shallow Anomaly Detection (2020, background): “Weak supervision or weakly supervised learning describes learning from imperfectly or scarcely labeled data [535]–[537].”
How We've Taught Algorithms to See Identity: Constructing Race and Gender in Image Databases for Facial Analysis (2020, background): “Increasingly, databases are being annotated through crowdsourcing or scraping associated image information on the web [142].”