Pointing the Unknown Words

By Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, Yoshua Bengio. 2016

537 citations recorded by the source corpus.

The problem of rare and unknown words is an important issue that can potentially influence the performance of many NLP systems, including both the traditional count-based and the deep learning models. We propose a novel way to deal with the rare and unseen words for the neural network models using attention. Our model uses two softmax layers in order to predict the next word in conditional language models: one predicts the location of a word in the source sentence, and the other predicts a word in the shortlist vocabulary. At each time-step, the decision of which softmax layer to use choose adaptively made by an MLP which is conditioned on the context.~We motivate our work from a psychological evidence that humans naturally have a tendency to point towards objects in the context or the environment when the name of an object is not known.~We observe improvements on two tasks, neural machine translation on the Europarl English to French parallel corpora and text summarization on the Gigaword dataset using our proposed model.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

A Deep Reinforced Model for Abstractive Summarization (2017, methodology): “Gulcehre et al. (2016) and Merity et al. (2016) combine this pointer mechanism with the original word generation layer in the decoder to allow the model to use either method at each decoding step.”
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, methodology): “…is to simply copy rare words from source to target (as most rare words are names or numbers where the correct translation is just a copy), either based on the attention model [37], using an external alignment model [31], or even using a more complicated special purpose pointing network [18].”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “…set of baselines for decaNLP that combine the basics of sequence-to-sequence learning [Sutskever et al., 2014, Bahdanau et al., 2014, Luong et al., 2015b] with pointer networks [Vinyals et al., 2015, Merity et al., 2017, Gülçehre et al., 2016, Gu et al., 2016, Nallapati et al., 2016], ad- Preprint.”
Get To The Point: Summarization with Pointer-Generator Networks (2017, methodology): “The pointer network has been used to create hybrid approaches for NMT (Gul-cehre et al., 2016), language modeling (Merity et al., 2016), and summarization (Gu et al., 2016; Gulcehre et al., 2016; Miao and Blunsom, 2016; Nallapati et al., 2016; Zeng et al., 2016).”
Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond (2016, methodology): “For a more detailed treatment of this model, and experiments on multiple tasks, please refer to the parallel work published by some of the authors of this work (Gulcehre et al., 2016).”
Pointer Sentinel Mixture Models (2016, methodology): “Rather than relying on the RNN hidden state to decide when to use the pointer, as in the recent work of Gülçehre et al. (2016), we allow the pointer component itself to decide when to use the softmax vocabulary through a sentinel.”
Improving Neural Language Models with a Continuous Cache (2016, background): “Gulcehre et al. (2016) have shown that a similar mechanism called pointer softmax could be used in the context of machine translation, to decide which word to copy from the source to target.”
Towards VQA Models That Can Read (2019, methodology): “To generalize to arbitrary text, we take inspiration from pointer networks which allow pointing to OOV words in context [11, 39, 32, 12, 34].”
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (2020, methodology): “Our decoder consists of an embedding layer, a transformer block and a pointer softmax mechanism (Gulcehre et al., 2016).”
Semi-Supervised QA with Generative Domain-Adaptive Nets (2017, methodology): “We implement the generative model as a sequence-to-sequence model (Sutskever et al., 2014) with a copy mechanism (Gu et al., 2016; Gulcehre et al., 2016).”
Incorporating Copying Mechanism in Sequence-to-Sequence Learning (2016, methodology): “Gulcehre et al. (2016) proposed a pointing method to handle the OOV words for summarization and MT.”
Challenges in Data-to-Document Generation (2017, methodology): “Copying There has been a surge of recent work involving augmenting encoder-decoder models to copy words directly from the source material on which they condition (Gu et al., 2016; G¨ulc¸ehre et al., 2016; Merity et al., 2016; Jia and Liang, 2016; Yang et al., 2016).”
Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies (2018, methodology): “Pointer-Generator The pointer-generator model (See et al., 2017) uses abstractive token generation and extractive token copying using a pointer mechanism (Vinyals et al., 2015; G¨ulc¸ehre et al., 2016), keeping track of extractions using coverage (Tu et al., 2016).”
Neural Summarization by Extracting Sentences and Words (2016, background): “This effectively helps us sidestep the difﬁculty of searching for the next output symbol under a large vocabulary , with low-frequency words and named entities whose representations can be challenging to learn.”
Controlling Output Length in Neural Encoder-Decoders (2016, background): “Some studies address the other important phenomena frequently occurred in human-written summaries, such as copying from the source document (Gu et al., 2016; Gulcehre et al., 2016).”
SYSTRAN's Pure Neural Machine Translation Systems (2016, background): “In the recent literature (Gu et al., 2016; Gulcehre et al., 2016), researchers have attempted to directly model the unknown word handling within the attention and decoder networks.”
Key-Value Retrieval Networks for Task-Oriented Dialogue (2017, background): “Recent literature has shown that incorporating a copying mechanism into neural architectures improves performance on various sequence-to-sequence tasks (Jia and Liang, 2016; Gu et al., 2016; Ling et al., 2016; Gulcehre et al., 2016; Eric and Manning, 2017).”
Mem2Seq: Effectively Incorporating Knowledge Bases into End-to-End Task-Oriented Dialog Systems (2018, methodology): “…relevant information directly from the KBs. Copy mechanisms has also been used in question answering tasks (Dehghani et al., 2017; He et al., 2017), neural machine translation (Gulcehre et al., 2016; Gu et al., 2016), language modeling (Merity et al., 2017), and summarization (See et al., 2017).”