Towards Decoding as Continuous Optimisation in Neural Machine Translation

By Cong Duy Vu Hoang, Gholamreza Haffari, Trevor Cohn. 2017

44 citations recorded by the source corpus.

We propose a novel decoding approach for neural machine translation (NMT) based on continuous optimisation. We reformulate decoding, a discrete optimization problem, into a continuous problem, such that optimization can make use of efficient gradient-based techniques. Our powerful decoding framework allows for more accurate decoding for standard neural machine translation models, as well as enabling decoding in intractable models such as intersection of several different NMT models. Our empirical results show that our decoding framework is effective, and can leads to substantial improvements in translations, especially in situations where greedy search and beam search are not feasible. Finally, we show how the technique is highly competitive with, and complementary to, reranking.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

How Can We Know What Language Models Know? (2019, background): “…problem of finding text sequences that optimize some continuous objective has been studied both in the context of end-to-end sequence generation (Hoang et al., 2017), and in the context of making small changes to an existing input for adversarial attacks (Ebrahimi et al., 2018; Wallace et al.,…”
Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement (2018, methodology): “There is no known polynomial algorithm for solving it exactly, and practitioners have relied on approximate decoding algorithms (see, e.g., Cho, 2016; Hoang et al., 2017).”
Gradient-based Constrained Sampling from Language Models (2022, background): “On the other hand, considering the simplex itself (Kumar et al., 2021; Hoang et al., 2017) as soft-representations is not compatible with Gaussian noise and can lead to undesirable behavior (Pat-terson and Teh, 2013). be flexibly defined for most kinds of constraints.”
Controlled Text Generation as Continuous Optimization with Multiple Constraints (2021, methodology): “Gradient-descent based optimization to generate text has been explored in prior work for improving machine translation [19], paraphrasing [50] and generating adversarial examples [59].”
COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics (2022, unknown): “[26] extend [17] by imposing constraints with a Lagrangian method and optimizing for a single output with gradient descent.”
Neural Machine Translation: A Review (2019, background): “[177] proposed a very appealing theoretical framework to relax the discrete NMT optimization problem into a continuous optimization problem which allows to include both decoding directions.”
Learning Approximate Inference Networks for Structured Prediction (2018, background): “There are other settings in which gradient descent is used for inference, e.g., image generation applications like DeepDream (Mordvintsev et al., 2015) and neural style transfer (Gatys et al., 2015), as well as machine translation (Hoang et al., 2017).”
Differentiable Scheduled Sampling for Credit Assignment (2017, background): “Hoang et al. (2017) also propose continuous relaxations of decoders, but are focused on developing better inference procedures.”