BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model

By Alex Wang, Kyunghyun Cho. 2019

389 citations recorded by the source corpus.

We show that BERT (Devlin et al., 2018) is a Markov random field language model. This formulation gives way to a natural procedure to sample sentences from BERT. We generate from BERT and find that it can produce high quality, fluent generations. Compared to the generations of a traditional left-to-right language model, BERT generates sentences that are more diverse but of slightly worse quality.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

Unified Language Model Pre-training for Natural Language Understanding and Generation (2019, background): “Although BERT signiﬁ-cantly improves the performance of a wide range of natural language understanding tasks (Devlin et al., 2018), its bidirectionality nature makes it difﬁcult to apply BERT to natural language generation tasks (Wang and Cho, 2019).”
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, background): “Further work could extend our analyses to toxicity to masked language models (Wang and Cho, 2019), among others.”
BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation (2021, methodology): “In our task, we use a pre-trained BERT model for filling in the next set of words given a prompt consisting of a set of seed words from Wikipedia [38].”
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (2019, methodology): “Generating text from BERT-style models is an open research area and we follow the sampling procedure from [37] – initializing the text stream with all MASK tokens and then sequentially resampling predicted output tokens in a Markov Chain Monte Carlo fashion.”
VideoBERT: A Joint Model for Video and Language Representation Learning (2019, methodology): “The BERT model, introduced in [6], can be thought of as a fully connected Markov Random Field (MRF) on a set of discrete tokens, which is trained to approximately maximize the pseudo log-likelihood, as explained in [32].”
Mask-Predict: Parallel Decoding of Conditional Masked Language Models (2019, methodology): “Generating from Masked Language Models One such approach for generating text from a masked language model casts BERT (Devlin et al., 2018), a non-conditional masked language model, as a Markov random field (Wang and Cho, 2019).”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, background): “Satellited around BERT, there are also studies on model compression (Sun et al., 2019c; Jiao et al., 2019; Shen et al., 2019) and extension from understanding to generation (Chen et al., 2019a; Clinchant et al., 2019; Wang and Cho, 2019).”
Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP (2021, methodology): “Nangia et al. (2020) use pseudo-log-likelihood (Wang and Cho, 2019; Salazar et al., 2020) to assign scores to sentences using MLMs. Bias in an MLM is then measured as the proportion of entries for which the MLM assigns a higher score to the more stereotypical sentence; an ideal model that does not…”
Masked Language Model Scoring (2019, methodology): “Our work extends the closest previous works (Wang and Cho, 2019; Shin et al., 2019) with regards to experiments and tasks, as outlined in Section 2.1.”
Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences (2019, methodology): “The models we have trained can be used to generate new sequences (Wang & Cho, 2019”
When Do You Need Billions of Words of Pretraining Data? (2020, methodology): “Since RoBERTa is a masked language model (MLM), we measure pseudo log-likelihood (Wang and Cho, 2019) to score sentences (Salazar et al.”
Justifying Recommendations using Distantly-Labeled Reviews and Fine-Grained Aspects (2019, methodology): “We follow the sampling strategy of Wang and Cho (2019) to generate justiﬁcations.”