Pre-Training Transformers as Energy-Based Cloze Models

By Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning. 2020

86 citations recorded by the source corpus.

We introduce Electric, an energy-based cloze model for representation learning over text. Like BERT, it is a conditional generative model of tokens given their contexts. However, Electric does not use masking or output a full distribution over tokens that could occur in a context. Instead, it assigns a scalar energy score to each input token indicating how likely it is given its context. We train Electric using an algorithm based on noise-contrastive estimation and elucidate how this learning objective is closely related to the recently proposed ELECTRA pre-training method. Electric performs well when transferred to downstream tasks and is particularly effective at producing likelihood scores for text: it re-ranks speech recognition n-best lists better than language models and much faster than masked language models. Furthermore, it offers a clearer and more principled view of what ELECTRA learns during pre-training.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Furu Wei
3. Mohit Bansal
4. Ting-Wei Liu
5. Jianfeng Gao
6. Bing Qin
7. Li Dong
8. Jing Zhang

In Large Language Models

1. Furu Wei
2. Li Dong
3. Ulme Wennberg
4. Hrant Khachatrian
5. Dacheng Tao
6. Jonathan May
7. Jing Zhang
8. Bing Qin

In Natural Language Processing

1. Furu Wei
2. Li Dong
3. Ting-Wei Liu
4. Bing Qin
5. Ulme Wennberg
6. Hrant Khachatrian
7. Dacheng Tao
8. Jonathan May

Citations in the corpus

• WARP: Word-level Adversarial ReProgramming (2021, methodology): “Besides our best WARP models, we also include the human baselines, current state-of-the-art model (He et al., 2020), the regular fine-tuned pretrained model we use, and also include relatively small language models, including (Jiao et al., 2020), (Clark et al., 2020), (Houlsby et al., 2019).”
• Transforming Wikipedia Into Augmented Data for Query-Focused Summarization (2019, background): “Therefore, we can take the title and the body of the citation 2 as the document and treat the statement as the summary.”
• Step-unrolled Denoising Autoencoders for Text Generation (2021, background): “Electric (Clark et al., 2020b) does not perform masking, but instead uses noise-contrastive loss to learn a representation of text.”
• A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, background): “A series of subsequent works have demonstrated that task-agnostic self-supervised contrastive pre-training has been shown to improve language modeling [236], [237], [238].”
• FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark (2021, methodology): “With the burst of pre-trained models (BERT (De-vlin et al., 2019), RoBERTa (Liu et al., 2019) , ELECTRA (Clark et al., 2020), XL-Net (Zhilin Yang, 2020) and the evolution of uni-fied evaluation benchmarks (GLUE (Wang et al., 1 http://tcci.ccf.org.cn/conference/2021/ 2018) and SuperGLUE (Wang et al., ”
• The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models (2021, methodology): “This is true even for recent years’ state-of-the-art models such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), ALBERT (Lan et al., 2020), and ELECTRA (Clark et al., 2020), which require many positional parameters but still cannot handle arbitrary-length sequences.”