Learning Context-Sensitive Convolutional Filters for Text Processing

By Dinghan Shen, Martin Renqiang Min, Yitong Li, Lawrence Carin. 2017

23 citations recorded by the source corpus.

Convolutional neural networks (CNNs) have recently emerged as a popular building block for natural language processing (NLP). Despite their success, most existing CNN models employed in NLP share the same learned (and static) set of filters for all input sentences. In this paper, we consider an approach of using a small meta network to learn context-sensitive convolutional filters for text processing. The role of meta network is to abstract the contextual information of a sentence or document into a set of input-sensitive filters. We further generalize this framework to model sentence pairs, where a bidirectional filter generation mechanism is introduced to encapsulate co-dependent sentence representations. In our benchmarks on four different tasks, including ontology classification, sentiment analysis, answer sentence selection, and paraphrase identification, our proposed model, a modified CNN with context-sensitive filters, consistently outperforms the standard CNN and attention-based CNN baselines. By visualizing the learned context-sensitive filters, we further validate and rationalize the effectiveness of proposed framework.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Michael Auli
2. Angela Fan
3. Alexei Baevski
4. Jianfeng Gao
5. Lawrence Carin
6. Luke S. Zettlemoyer
7. Mohit Bansal
8. Kyunghyun Cho

In Natural Language Processing

1. Michael Auli
2. Angela Fan
3. Alexei Baevski
4. Yann Dauphin
5. Luke S. Zettlemoyer
6. Lawrence Carin
7. Michael Lewis
8. Jason Aaron Edward Weston

In Model Architectures

1. Angela Fan
2. Alexei Baevski
3. Michael Auli
4. Yann Dauphin
5. Felix Wu
6. Lawrence Carin
7. Ricardo Henao
8. Dinghan Shen

Citations in the corpus

• Pay Less Attention with Lightweight and Dynamic Convolutions (2019, methodology): “Shen et al. (2018a) and Gong et al. (2018) use input-dependent filters for text classification tasks.”
• Improved Semantic-Aware Network Embedding with Fine-Grained Word Alignment (2018, methodology): “We experiment with three variants for our WANE model: ( i ) WANE: where the word embeddings of each text sequence are simply average to obtain the sentence representations, similar to (Joulin et al., 2016; Shen et al., 2018c).”