Attention Forcing for Sequence-to-sequence Model Training
By Qingyun Dou, Yiting Lu, Joshua Efiong, Mark John Francis Gales. 2019
7 citations recorded by the source corpus.
Auto-regressive sequence-to-sequence models with attention mechanism have achieved state-of-the-art performance in many tasks such as machine translation and speech synthesis. These models can be difficult to train. The standard approach, teacher forcing, guides a model with reference output history during training. The problem is that the model is unlikely to recover from its mistakes during inference, where the reference output is replaced by generated output. Several approaches deal with this problem, largely by guiding the model with generated output history. To make training stable, these approaches often require a heuristic schedule or an auxiliary classifier. This paper introduces attention forcing, which guides the model with generated output history and reference attention. This approach can train the model to recover from its mistakes, in a stable fashion, without the need for a schedule or a classifier. In addition, it allows the model to generate output sequences aligned with the references, which can be important for cascaded systems like many speech synthesis systems. Experiments on speech synthesis show that attention forcing yields significant performance gain. Experiments on machine translation show that for tasks where various re-orderings of the output are valid, guiding the model with generated output history is challenging, while guiding the model with reference attention is beneficial.
Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Natural Language Processing
In Model Architectures
Citations in the corpus
- Teaching Autoregressive Language Models Complex Tasks By Demonstration (2021, methodology): “We refer to this approach as environment forcing by analogy to related concepts such as professor forcing[33] and attention forcing[34], as it allows the state of the environment to impose constraints on the generated text, with the goal of mitigating exposure bias.”