Energy-Based Models for Text
By Anton Bakhtin, Yuntian Deng, Sam Gross, Myle Ott, Marc'Aurelio Ranzato, Arthur Szlam. 2020
13 citations recorded by the source corpus.
Current large-scale auto-regressive language models display impressive fluency and can generate convincing text. In this work we start by asking the question: Can the generations of these models be reliably distinguished from real text by statistical discriminators? We find experimentally that the answer is affirmative when we have access to the training data for the model, and guardedly affirmative even if we do not. This suggests that the auto-regressive models can be improved by incorporating the (globally normalized) discriminators into the generative process. We give a formalism for this using the Energy-Based Model framework, and show that it indeed improves the results of the generative models, measured both in terms of perplexity and in terms of human evaluation.
Subfields: Artificial Intelligence, Generative Models, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Tomasz Korbak
- Hady ElSahar
- Germán Kruszewski
- Marc Dymetman
- Kevin Gimpel
- Richard Yuanzhe Pang
- Sam Wiseman
- Lifu Tu
In Generative Models
- Tomasz Korbak
- Germán Kruszewski
- Hady ElSahar
- Marc Dymetman
- Kevin Gimpel
- Muhammad Khalifa
- Sam Wiseman
- Lifu Tu
In Natural Language Generation
Citations in the corpus
- A Distributional Approach to Controlled Text Generation (2020, background): “A recent survey of EBMs for text is provided in Bakhtin et al. (2020).”
- Energy-Based Models for Code Generation under Compilability Constraints (2021, background): “A recent surge of interest in EBMs (Du and Mordatch, 2019) has not left text generation unaffected (see (Bakhtin et al., 2020) for a survey).”
- ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation (2020, background): “Energy-based models have had limited application in text generation due to the computational challenges involved in learning and inference in extremely large search spaces (Bakhtin et al., 2020).”
- Automatic Detection of Machine Generated Text: A Critical Survey (2020, methodology): “…text generative models (TGMs) excel in producing text that approaches the style of human language, especially in terms of grammaticality, fluency, coherency, and usage of real world knowledge (Radford et al., 2019; Zellers et al., 2019; Keskar et al., 2019; Bakhtin et al., 2020; Brown et al., 2020).”