Learning What’s Easy: Fully Differentiable Neural Easy-First Taggers

By André F. T. Martins, Julia Kreutzer. 2017

31 citations recorded by the source corpus.

We introduce a novel neural easy-first decoder that learns to solve sequence tagging tasks in a flexible order. In contrast to previous easy-first decoders, our models are end-to-end differentiable. The decoder iteratively updates a “sketch” of the predictions over the sequence. At its core is an attention mechanism that controls which parts of the input are strategically the best to process next. We present a new constrained softmax transformation that ensures the same cumulative attention to every word, and show how to efficiently evaluate and backpropagate over it. Our models compare favourably to BILSTM taggers on three sequence tagging tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Efficient Content-Based Sparse Attention with Routing Transformers (2020, methodology): “(Martins and Kreutzer, 2017; Malaviya et al., 2018) propose to compute attention weights with variants of sparsemax.”
Learning Approximate Inference Networks for Structured Prediction (2018, methodology): “Like our use of inference networks, there have been efforts in designing differentiable approximations of combinatorial search procedures (Martins & Kreutzer, 2017; Goyal et al., 2018) and structured losses for training with them (Wiseman & Rush, 2016).”
SSN: Learning Sparse Switchable Normalization via SparsestMax (2019, background): “One of its variants is constrained softmax transformation (Martins and Kreutzer 2017) that is developed to learn to solve sequence tagging tasks in a flexible order.”
Attention in Natural Language Processing (2019, background): “In [46] and [51], the coverage information is exploited by a constrained distribution function, regulating the amount of attention that the same word receives over time.”
Gradient Estimation with Stochastic Softmax Tricks (2020, methodology): “Our SSTs use the following relaxations: euclidean [6] and categorical [56], binary [7], and exponential family [77] entropies.”
Sparse and Constrained Attention for Neural Machine Translation (2018, methodology): “The constrained soft-max transformation was recently proposed by Martins and Kreutzer (2017) in the context of easy-ﬁrst sequence tagging, being deﬁned as follows: where u is a vector of upper bounds, and KL ( . k .”