Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders

By Fangyu Liu, Ivan Vulic, Anna Korhonen, Nigel Collier. 2021

137 citations recorded by the source corpus.

Previous work has indicated that pretrained Masked Language Models (MLMs) are not effective as universal lexical and sentence encoders off-the-shelf, i.e., without further task-specific fine-tuning on NLI, sentence similarity, or paraphrasing tasks using annotated task data. In this work, we demonstrate that it is possible to turn MLMs into effective lexical and sentence encoders even without any additional data, relying simply on self-supervision. We propose an extremely simple, fast, and effective contrastive learning technique, termed Mirror-BERT, which converts MLMs (e.g., BERT and RoBERTa) into such encoders in 20-30 seconds with no access to additional external knowledge. Mirror-BERT relies on identical and slightly modified string pairs as positive (i.e., synonymous) fine-tuning examples, and aims to maximise their similarity during “identity fine-tuning”. We report huge gains over off-the-shelf MLMs with Mirror-BERT both in lexical-level and in sentence-level tasks, across different domains and different languages. Notably, in sentence similarity (STS) and question-answer entailment (QNLI) tasks, our self-supervised Mirror-BERT model even matches the performance of the Sentence-BERT models from prior work which rely on annotated task data. Finally, we delve deeper into the inner workings of MLMs, and suggest some evidence on why this simple Mirror-BERT fine-tuning approach can yield effective universal lexical and sentence encoders.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Adrian Weller
2. Dani Yogatama
3. Zhen Liu
4. Weiyang Liu
5. Yan Wang
6. Lanqing Hong
7. L. Yu
8. Tianyang Hu

In Computer Vision and Pattern Recognition

1. Adrian Weller
2. Zhen Liu
3. Weiyang Liu
4. Lanqing Hong
5. L. Yu
6. Tianyang Hu
7. Dani Yogatama
8. Fangyu Liu

Citations in the corpus

• Continual Learning by Modeling Intra-Class Variation (2022, background): “Second, the representation space is more regularized, since it converges to the simplex equiangular tight frame (Papyan et al., 2020) which is also equivalent to a hyperspherically uniform space (Liu et al., 2018a; Lin et al., 2020; Liu et al., 2021c;b).”
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “Recently many approaches based on supervised learning [190], [191] or self-supervised learning [158], [159], [192]–[194], [194], [195] are proposed.”
• Language Models Can See: Plugging Visual Controls in Text Generation (2022, background): “In the field of NLP, numerous approaches have been proposed to learn better token-level [73] and sentence-level [22, 43] representations using contrastive learning.”