Pointer Networks

By Oriol Vinyals, Meire Fortunato, Navdeep Jaitly. 2015

3,513 citations recorded by the source corpus.

We introduce a new neural architecture to learn the conditional probability of an output sequence with elements that are discrete tokens corresponding to positions in an input sequence. Such problems cannot be trivially addressed by existent approaches such as sequence-to-sequence [1] and Neural Turing Machines [2], because the number of target classes in each step of the output depends on the length of the input, which is variable. Problems such as sorting variable sized sequences, and various combinatorial optimization problems belong to this class. Our model solves the problem of variable size output dictionaries using a recently proposed mechanism of neural attention. It differs from the previous attention attempts in that, instead of using attention to blend hidden units of an encoder to a context vector at each decoder step, it uses attention as a pointer to select a member of the input sequence as the output. We call this architecture a Pointer Net (Ptr-Net). We show Ptr-Nets can be used to learn approximate solutions to three challenging geometric problems - finding planar convex hulls, computing Delaunay triangulations, and the planar Travelling Salesman Problem - using training examples alone. Ptr-Nets not only improve over sequence-to-sequence with input attention, but also allow us to generalize to variable size output dictionaries. We show that the learnt models generalize beyond the maximum lengths they were trained on. We hope our results on these tasks will encourage a broader exploration of neural learning for discrete problems.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

A Deep Reinforced Model for Abstractive Summarization (2017, background): “One way to ﬁx this is to allow the decoder network to point back to some speciﬁc words or sub-sequences of the input and copy them onto the output sequence (Vinyals et al., 2015; Ling et al., 2016).”
Structured Attention Networks (2017, background): “…et al., 2015), speech recognition (Chorowski et al., 2015; Chan et al., 2015), question answering (Hermann et al., 2015; Sukhbaatar et al., 2015), and algorithm-learning (Graves et al., 2014; Vinyals et al., 2015), among many other applications (see Cho et al. (2015) for a comprehensive review).”
Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering (2018, background): “…2) attention layer in which hierarchical attention networks are designed to capture the relation between question and passage at different levels of granularity; 3) match layer where reﬁned question and passage are matched under a pointer-network (Vinyals et al., 2015) answer boundary predictor.”
U-Net: Machine Reading Comprehension with Unanswerable Questions (2018, methodology): “Finally, a pointer network (Vinyals, Fortunato, and Jaitly 2015) is used to predict the span boundaries of the answer.”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “…set of baselines for decaNLP that combine the basics of sequence-to-sequence learning [Sutskever et al., 2014, Bahdanau et al., 2014, Luong et al., 2015b] with pointer networks [Vinyals et al., 2015, Merity et al., 2017, Gülçehre et al., 2016, Gu et al., 2016, Nallapati et al., 2016], ad- Preprint.”
Matching Networks for One Shot Learning (2016, methodology): “We draw inspiration from models such as sequence to sequence (seq2seq) with attention [2], memory networks [29] and pointer networks [27].”
Adaptive Computation Time for Recurrent Neural Networks (2016, background): “ACT promises to be particularly interesting for recurrent architectures containing soft attention modules [2, 10, 34, 12], which it could enable to dynamically adapt the number of glances or internal operations they perform at each time-step.”
Get To The Point: Summarization with Pointer-Generator Networks (2017, methodology): “Our pointer-generator network is a hybrid between our baseline and a pointer network (Vinyals et al., 2015), as it allows both copying words via pointing, and generating words from a ﬁxed vocabulary.”
Pointer Sentinel Mixture Models (2016, background): “Attention-based pointer mechanisms were introduced in Vinyals et al. (2015) where the pointer network is able to select elements from the input as output.”
WikiLingua: A New Benchmark Dataset for Multilingual Abstractive Summarization (2020, background): “Nallapati et al. (2016) shows further improvements by incorporating embeddings for linguistic features such as part-of-speech tags and named-entity tags into their model, as well as a pointer network (Vinyals et al., 2015) to enable copying words from the source article.”
Rethinking Attention with Performers (2020, background): “…Transformer scope as a more scalable replacement for regular attention, which itself has a wide variety of uses in computer vision (Fu et al., 2019), reinforcement learning (Zambaldi et al., 2019), training with softmax cross entropy loss, and even combinatorial optimization (Vinyals et al., 2015).”
MLSUM: The Multilingual Summarization Corpus (2020, background): “Pointer-Generator See et al. (2017) proposed the addition of the copy mechanism (Vinyals et al., 2015) on top of a sequence to sequence LSTM model.”
SPoC: Search-based Pseudocode to Code (2019, methodology): “To generate candidate code lines, we use a standard seq2seq translation model with an LSTM encoder and decoder [19], attention-based copying mechanism [24, 37], and coverage vector [34].”
Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems (2017, background): “• If qi,j = COPY-INPUT, ai,j is obtained by copying an element from the input vector with a pointer network (Vinyals et al., 2015) over input words x1, .”
Discriminative Adversarial Search for Abstractive Summarization (2020, neutral): “Further, it has been pointed out how abstractive summarization systems tend to be too ex-tractive (Kry ´ sci ´ nski et al., 2018), mainly because of the copy mechanism (Vinyals et al., 2015).”
SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine (2017, background): “Attention Sum Reader Attention sum reader (ASR, Kadlec et al., 2016) is a variant of a pointer network (Vinyals et al., 2015) that was speciﬁ-cally constructed to solve a cloze-style question-answering task.”
Learning Recurrent Span Representations for Extractive Question Answering (2016, methodology): “Machine learned extractive question answering systems, such as the one presented here, learn a predictor function f ( q , p ) → a from a training dataset of (cid:104) q , p , a (cid:105) triples.”
Unbounded cache model for online language modeling with open vocabulary (2017, methodology): “This memory component also shares similarity with the parametric memory component of the pointer network introduced by Vinyals et al. [52] and extended by Merity et al. [43].”
Improving Neural Language Models with a Continuous Cache (2016, background): “Similarly, Vinyals et al. (2015) explores the use of this mechanism to reorder sequences of tokens.”