Semantic Sentence Matching with Densely-connected Recurrent and Co-attentive Information

By Seonhoon Kim, Jin-Hyuk Hong, Inho Kang, Nojun Kwak. 2018

226 citations recorded by the source corpus.

Sentence matching is widely used in various natural language tasks such as natural language inference, paraphrase identification, and question answering. For these tasks, understanding logical and semantic relationship between two sentences is required but it is yet challenging. Although attention mechanism is useful to capture the semantic relationship and to properly align the elements of two sentences, previous methods of attention mechanism simply use a summation operation which does not retain original features enough. Inspired by DenseNet, a densely connected convolutional network, we propose a densely-connected co-attentive recurrent neural network, each layer of which uses concatenated information of attentive features as well as hidden features of all the preceding recurrent layers. It enables preserving the original and the co-attentive feature information from the bottommost word embedding layer to the uppermost recurrent layer. To alleviate the problem of an ever-increasing size of feature vectors due to dense concatenation operations, we also propose to use an autoencoder after dense concatenation. We evaluate our proposed architecture on highly competitive benchmark datasets related to sentence matching. Experimental results show that our architecture, which retains recurrent and attentive features, achieves state-of-the-art performances for most of the tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “Fine-tuning a MQAN pretrained on decaNLP and training exactly as before on MultiNLI achieves an 87% test exact match score, which is a 2% increase over training from a random initialization and 2% from the state of the art (Kim et al., 2018).”
Non-entailed subsequences as a challenge for natural language inference (2018, background): “…of large sets of premises and hypotheses generated through crowdsourcing has made it possible to train neural networks without explicit logical representations to perform this task; such systems have reached considerable accuracy on these data sets (Radford et al., 2018; Kim et al., 2018).”
Knowledge-Grounded Dialogue Generation with Pre-trained Language Models (2020, background): “…2019; Ren et al., 2019) and thus is incompatible with the pre-trained language models, or it is learned with human annotations (Dinan et al., 2019; Kim et al., 2018) which are difﬁcult to obtain in practice (e.g., the dataset in (Zhou et al., 2018b) does not contain annotations for knowledge…”
Testing the Generalization Power of Neural Network Models across NLI Benchmarks (2018, background): “Both of them have been very successful, with the state of the art on the SNLI and MultiNLI datasets being 90.1% (Kim et al., 2018) and 86.7% (Devlin et al., 2018) respectively.”
Named Entity Recognition for Social Media Texts with Semantic Augmentation (2020, methodology): “To further improve NER performance, we also attempt multiple types of pre-trained word embeddings for feature extraction, which has been demonstrated to be effective in previous studies (Akbik et al., 2018; Jie and Lu, 2019; Kasai et al., 2019; Kim et al., 2019; Yan et al., 2019).”
Deep Learning--based Text Classification (2020, methodology): “t sequences and labels via cosine similarity, as shown in Fig.8(b). Fig. 8. (a) The architecture of attentive pooling networks [64]. (b) The architecture of label-text matching model [65]. Kim et al. [66] proposed a semantic sentence matching approach using a densely-connected recurrent and co-atten”
A question-entailment approach to question answering (2019, background): “ogistic Regression in question-question similarity at SemEval 2017 (best system [11] and second best system [21]), and the high performance achieved by neural networks on larger datasets such as SNLI [8, 28, 12, 20]. We ﬁrst deﬁne the RQE task, then present the two approaches, and evaluate their per”
Exploiting Edge Features in Graph Neural Networks. (2018, unknown): “Attention mechanisms have been widely employed in many sequence-based tasks [3][33][16].”
A Lightweight Recurrent Network for Sequence Modeling (2019, background): “…natural language processing (NLP) tasks can be categorized as sequence modeling tasks, where recurrent networks (RNNs) are widely applied and contribute greatly to state-of-the-art neural systems (Yang et al., 2018; Peters et al., 2018; Zhang et al., 2018; Chen et al., 2018; Kim et al., 2019).”
Generating Persona Consistent Dialogues by Exploiting Natural Language Inference (2019, methodology): “2015), deep neural network methods have made promising progress (Chen et al. 2017; Gong, Luo, and Zhang 2018; Kim et al. 2019).”
Multi-Level Matching and Aggregation Network for Few-Shot Relation Classification (2019, methodology): “Some other methods are based on joint models (Chen et al., 2017; Gong et al., 2017; Kim et al., 2018), which use cross-features to represent the local (i.e., word-level and phrase-level) alignments for better performance.”
Commonsense Reasoning for Natural Language Understanding: A Survey of Benchmarks, Resources, and Approaches (2019, methodology): “, 2015) reports that the best performing system (Kim et al., 2019), partly inspired by DENSENET (Huang, Liu, van der Maaten, & Weinberger, 2016), uses a densely connected RNN while concatenating features from an attention mechanism to recurrent features in the network.”
ChemTEB: Chemical Text Embedding Benchmark, an Overview of Embedding Models Performance & Efficiency on a Specific Domain (2024, unknown): “Additionally, specialized NLP tasks like natural language inference and paraphrase recognition are tackled by datasets such as MultiNLI [26] and PAWS [27], which offer challenging benchmarks for testing the depth of understanding in sentence embeddings.”