Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks

By Nandan Thakur, Nils Reimers, Johannes Daxenberger, Iryna Gurevych. 2020

317 citations recorded by the source corpus.

There are two approaches for pairwise sentence scoring: Cross-encoders, which perform full-attention over the input pair, and Bi-encoders, which map each input independently to a dense vector space. While cross-encoders often achieve higher performance, they are too slow for many practical use cases. Bi-encoders, on the other hand, require substantial training data and fine-tuning over the target task to achieve competitive performance. We present a simple yet efficient data augmentation strategy called Augmented SBERT, where we use the cross-encoder to label a larger set of input pairs to augment the training data for the bi-encoder. We show that, in this process, selecting the sentence pairs is non-trivial and crucial for the success of the method. We evaluate our approach on multiple tasks (in-domain) as well as on a domain adaptation task. Augmented SBERT achieves an improvement of up to 6 points for in-domain and of up to 37 points for domain adaptation tasks compared to the original bi-encoder performance.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

A Survey of Data Augmentation Approaches for NLP (2021, methodology): “…paraphrasing (Gangal et al., 2021), augmenting misclassiﬁed examples (Dreossi et al., 2018), BERT cross-encoder labeling of new inputs (Thakur et al., 2021), guided generation us-ing large-scale generative language models (Liu et al., 2020b,c), and automated text augmentation (Hu et…”
COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining (2021, background): “Recent studies have shown the effectiveness of advanced data augmentation techniques in ﬁne-tuning language [16, 38, 51].”
BERTopic: Neural topic modeling with a class-based TF-IDF procedure (2022, background): “It achieves state-of-the-art performance on various sentence embedding tasks (Reimers and Gurevych, 2020; Thakur et al., 2020).”
Text Data Augmentation for Deep Learning (2021, background): “Thakur et al. [85] describe that “selecting the sentence pairs is non-trivial and crucial for the success of the method”.”
Right for the Right Reason: Evidence Extraction for Trustworthy Tabular Reasoning (2022, background): “(Reimers and Gurevych, 2020; Thakur et al., 2021; Wang et al., 2021a), which use Siamese neural networks (Koch et al.”
LVLM-EHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models (2023, unknown): “SentenceTransformer [77] computes the similarity between generated and ground-truth text as "Sim.”
Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders (2021, unknown): “, 2021), which could also be automatically mined (Su, 2020; Thakur et al., 2021).”
Data Augmentation Approaches in Natural Language Processing: A Survey (2021, methodology): “[31] first fine-tune BERT on the original data, then use the fine-tuned BERT to label unlabeled sentence pairs.”
Question Answering Infused Pre-training of General-Purpose Contextualized Representations (2021, methodology): “Thakur et al. (2021) train a sentence embedding biencoder to mimic the predictions of a cross-encoder model.”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “, knowledge distillation [38], [102] (Section 5.”
SGPT: GPT Sentence Embeddings for Semantic Search (2022, background): “Bi-Encoders encode query and document separately.”
Case-Based Abductive Natural Language Inference (2020, background): “, 2009), while the dense version employs Sentence-BERT (large) (Reimers and Gurevych, 2019; Thakur et al., 2020), which has demonstrated state-of-the-art performance in semantic textual similarity tasks.”
Advancing Semantic Textual Similarity Modeling: A Regression Framework with Translated ReLU and Smooth K2 Loss (2024, background): “…that are directly relevant to our work: Siamese Neural Network Architectures : These approaches (Reimers and Gurevych, 2019; Conneau et al., 2017; Thakur et al., 2021), proposed relatively earlier in the field, have been widely applied across various domains owing to their effectiveness on…”