SemDeDup: Data-efficient learning at web-scale through semantic deduplication

By Amro Abbas, Kushal Tirumala, Daniel Simig, Surya Ganguli, Ari S. Morcos. 2023

307 citations recorded by the source corpus.

Progress in machine learning has been driven in large part by massive increases in data. However, large web-scale datasets such as LAION are largely uncurated beyond searches for exact duplicates, potentially leaving much redundancy. Here, we introduce SemDeDup, a method which leverages embeddings from pre-trained models to identify and remove semantic duplicates: data pairs which are semantically similar, but not exactly identical. Removing semantic duplicates preserves performance and speeds up learning. Analyzing a subset of LAION, we show that SemDeDup can remove 50% of the data with minimal performance loss, effectively halving training time. Moreover, performance increases out of distribution. Also, analyzing language models trained on C4, a partially curated dataset, we show that SemDeDup improves over prior approaches while providing efficiency gains. SemDeDup provides an example of how simple ways of leveraging quality embeddings can be used to make models learn faster with less data.

Subfields: Artificial Intelligence, Management Science and Operations Research, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Natural Language Processing

Citations in the corpus

The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, background): “Recently, Abbas et al. (2023) has proposed to leverage embeddings from pretrained models to imbue semantic understanding in approximate matching algorithms.”
GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models (2025, unknown): “We additionally apply the SemDedup [1] pipeline to remove those similar webpages based on document embeddings.”
The MiniPile Challenge for Data-Efficient Language Models (2023, background): “Potential issues with such include token repetitions [33, 21, 1], misogyny, pornography, and malignant stereotypes [9, 7], benchmark data contamination [11, 14], spurious correlations [27, 37, 38], diluted robustness due to data mixing [45] and potentially sensitive information [20].”
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, unknown): “Additionally, SemDeDup (Abbas et al., 2023) makes use of embeddings from pre-trained models to identify semantic duplicates, which refers to data pairs with semantic similarities but not identical.”
OBELISC: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents (2023, background): “(Lee et al., 2022; Abbas et al., 2023) show that crawled data often contains a significant amount of duplication, and training on deduplicated data can improve performance.”
DataComp: In search of the next generation of multimodal datasets (2023, methodology): “[2] Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos.”
OLMo: Accelerating the Science of Language Models (2024, methodology): “…data composition and its match to the domains in Paloma (e.g., MPT trains on 27% non-Common Crawl data rather than 18% for LLaMA, 12.2% for RedPajama, and 11.2% for OLMo) as well as various data preprocessing decisions (e.g., MPT’s use of semantic deduplication by Abbas et al., 2023, on C4).”
An OpenMind for 3D Medical Vision Self-supervised Learning (2024, unknown): “When training on a large enough quantity of data, optimizing the dataset becomes an important aspect that can greatly improve the resulting model performance [2, 32, 33].”