Convolutional Neural Network for Paraphrase Identification

By Wenpeng Yin, Hinrich Schütze. 2015

236 citations recorded by the source corpus.

We present a new deep learning architecture Bi-CNN-MI for paraphrase identification (PI). Based on the insight that PI requires comparing two sentences on multiple levels of granularity, we learn multigranular sentence representations using convolutional neural network (CNN) and model interaction features at each level. These features are then the input to a logistic classifier for PI. All parameters of the model (for embeddings, convolution and classification) are directly optimized for PI. To address the lack of training data, we pretrain the network in a novel way using a language modeling task. Results on the MSRP corpus surpass that of previous NN competitors.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

A Structured Self-attentive Sentence Embedding (2017, background): “…networks (Kalchbren-ner et al., 2014; dos Santos & Gatti, 2014; Kim, 2014) as an intermediate step in creating sentence representations to solve a wide variety of tasks including classiﬁcation and ranking (Yin & Sch¨utze, 2015; Palangi et al., 2016; Tan et al., 2016; Feng et al., 2015).”
Key-Value Memory Networks for Directly Reading Documents (2016, methodology): “…al. , 2015), the best published results on T REC QA and W IKI QA have been obtained by either convolutional neural networks (Santos et al. , 2016; Yin and Schütze, 2015; Wang et al. , 2016) or recurrent neural networks (Miao et al. , 2015) – both usually with attention mechanisms inspired by…”
Towards Universal Paraphrastic Sentence Embeddings (2015, methodology): “…hierarchical structure (Zhao et al., 2015; Chen et al., 2015b), convolutional neural networks (Kalchbrenner et al., 2014; Kim, 2014; Hu et al., 2014; Yin & Sch¨utze, 2015; He et al., 2015), and recurrent neural networks using long short-term memory (Tai et al., 2015; Ling et al., 2015; Liu et al.,…”
Reasoning about Entailment with Neural Attention (2015, background): “Despite the success of neural networks for paraphrase detection (e.g. Socher et al., 2011; Hu et al., 2014; Yin and Schütze, 2015), end-to-end differentiable neural architectures failed to get close to acceptable performance for RTE due to the lack of large high-quality datasets.”
Natural Language Inference by Tree-Based Convolution and Heuristic Matching (2015, background): “As shown, inherent structural information in sentences is oftentimes important to natural language understanding.”
Continual Learning for Sentence Representations Using Conceptors (2019, background): “, (Yin and Schütze, 2015; Arora et al., 2017; Cer et al., 2018; Pagliardini et al., 2018)), most, if not all, of them are trained on a priori fixed corpora.”
Representation learning for very short texts using weighted word embedding aggregation (2016, methodology): “…embeddings through a mean, max, min... function is still one of the most easy and widely used techniques to derive sentence embeddings, often in combination with an MLP or convolutional network (Weston et al. (2014); dos Santos and Gatti (2014); Yin and Sch¨utze (2015); Collobert et al. (2011)).”
ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs (2015, methodology): “, 2011), recursive autoencoder; (iv) Bi-CNNMI (Yin and Schütze, 2015a), a bi-CNN architecture; and (v) MPSSM-CNN (He et al.”
Multi-Perspective Sentence Similarity Modeling with Convolutional Neural Networks (2015, methodology): “Recent successes in sentence similarity have been obtained by using neural networks (Tai et al., 2015; Yin and Schütze, 2015).”
A Primer on Neural Network Models for Natural Language Processing (2015, neutral): “…2014; dos Santos, Xiang, & Zhou, 2015), event detection (Chen, Xu, Liu, Zeng, & Zhao, 2015; Nguyen & Grishman, 2015), paraphrase identification (Yin & Schütze, 2015) semantic role labeling (Collobert, Weston, Bottou, Karlen, Kavukcuoglu, & Kuksa, 2011), question answering (Dong, Wei, Zhou, &…”
Pairwise Word Interaction Modeling with Deep Neural Networks for Semantic Similarity Measurement (2016, methodology): “Our approach only uses pretrained word embeddings, and unlike several previous neural network models (Yin and Schütze, 2015; Tai et al., 2015), we do not use sparse features, unsupervised model pretraining, syntactic parsers, or external resources like WordNet.”
Learning Generic Sentence Representations Using Convolutional Neural Networks (2016, neutral): “4 − Bi-CNN-MI (Yin and Schütze, 2015) − − − − − 78.”
Siamese CBOW: Optimizing Word Embeddings for Sentence Representations (2016, neutral): “In the first setting, word vectors are typically used as features or network initialisations (Kenter and de Rijke, 2015; Hu et al., 2014; Severyn and Moschitti, 2015; Yin and Schütze, 2015).”
Modelling Interaction of Sentence Pair with Coupled-LSTMs (2016, methodology): “Semi-interaction Models Some improved methods focus on utilizing multi-granularity representation (word, phrase and sentence level), such as MultiGranCNN (Yin and Schütze, 2015) and MultiPerspective CNN (He et al.”
UMD-TTIC-UW at SemEval-2016 Task 1: Attention-Based Multi-Perspective Convolutional Neural Networks for Textual Similarity Measurement (2016, background): “Competitive systems in recent years are mostly based on neural networks (He et al., 2015; Tai et al., 2015; Yin and Schütze, 2015; He and Lin, 2016), which can alleviate data sparseness with pretraining and distributed representations.”
Applying deep learning to answer selection: A study and an open task (2015, methodology): “[23] use a CNN for answer selection; Yin and Schütze [24] use a bi-CNN for paraphrase identification.”
A Deep Architecture for Semantic Matching with Multiple Positional Sentence Representations (2015, methodology): “2011), DeepMatch (Lu and Li 2013), Bi-CNN-MI (Yin and Schütze 2015a) and MultiGranCNN (Yin and Schütze 2015b).”
Do Language Models Plagiarize? (2022, background): “mance is robust, and it focuses on the longest plagiarized substrings unlike existing plagiarism detectors trained and evaluated on labeled sentence pairs (Shahmohammadi et al., 2021; Socher et al., 2011; Yin and Schütze, 2015).”
A Survey of the Usages of Deep Learning for Natural Language Processing (2020, methodology): “Schütze [135] proposed a Bi-CNN-MI (MI for multigranular interaction features), consisting of a pretrained CNN sentence”
Large-Scale, Diverse, Paraphrastic Bitexts via Sampling and Clustering (2019, methodology): “Convolutional neural networks have been introduced by Yin and Schütze (2015) and Chen et al. (2018), and further augmented with LSTMs (Kubal and Nimkar, 2018) and attention mechanisms (Fan et al., 2018).”