Enhancing Sentence Embedding with Generalized Pooling

By Qian Chen, Zhenhua Ling, Xiao-Dan Zhu. 2018

76 citations recorded by the source corpus.

Pooling is an essential component of a wide variety of sentence representation and embedding models. This paper explores generalized pooling methods to enhance sentence embedding. We propose vector-based multi-head attention that includes the widely used max pooling, mean pooling, and scalar self-attention as special cases. The model benefits from properly designed penalization terms to reduce redundancy in multi-head attention. We evaluate the proposed model on three different tasks: natural language inference (NLI), author profiling, and sentiment classification. The experiments show that the proposed model achieves significant improvement over strong sentence-encoding-based methods, resulting in state-of-the-art performances on four datasets. The proposed approach can be easily implemented for more problems than we discuss in this paper.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Zhenhua Ling
3. Mohit Bansal
4. Guodong Long
5. Xiao-Dan Zhu
6. Erik Cambria
7. Jing Jiang
8. Qian Chen

In Natural Language Processing

1. Zhenhua Ling
2. Jianfeng Gao
3. Qian Chen
4. Erik Cambria
5. Xiao-Dan Zhu
6. Mohit Bansal
7. Wen Wang
8. William Yang Wang

Citations in the corpus

• Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering (2020, background): “While neural models have proven effective at learning deep semantic matching between text pairs (Bowman et al., 2015; Parikh et al., 2016; Chen et al., 2018; Devlin et al., 2019), they usually require computing question-dependent paragraph encodings (i.”
• COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning (2020, background): “There have been some attempts in other domains to design a context-aware feature fusion method [30, 31, 32, 33].”
• Exploring End-to-End Differentiable Natural Logic Modeling (2020, background): “Recently deep learning algorithms have been proposed (Bowman et al., 2015; Chen et al., 2017a; Chen et al., 2017b; Chen et al., 2017c; Chen et al., 2018; Peters et al., 2018; Yoon et al., 2018; Kiela et al., 2018; Talman et al., 2018; Yang et al., 2019; Devlin et al., 2019).”
• Sequential Attention-based Network for Noetic End-to-End Response Selection (2019, methodology): “Future work on multi-turn response selection includes exploring the efficacy of external knowledge (Chen et al. 2018), such as knowledge graph and user profile.”
• Attention in Natural Language Processing (2019, unknown): “with one another [95].”
• Fast Directional Self-Attention Mechanism (2018, methodology): “We evaluate all the models on the sentences involving compositional knowledge (SICK) dataset, where the similarity degree ranges from [1, 5].”
• Multi-Level Matching and Aggregation Network for Few-Shot Relation Classification (2019, methodology): “Some sentence matching methods mainly rely on sentence encoding (Mueller and Thyagarajan, 2016; Conneau et al., 2017; Chen et al., 2018), which encode a pair sentences independently and then transmit their embeddings into a classifier, such as a neural network, to decide the relationship between…”
• Deep Learning--based Text Classification (2020, methodology): “3 — — Generalized Pooling (600D) [93] 86.”