Discriminative Improvements to Distributional Sentence Similarity

By Yangfeng Ji, Jacob Eisenstein. 2013

162 citations recorded by the source corpus.

Matrix and tensor factorization have been applied to a number of semantic relatedness tasks, including paraphrase identification. The key idea is that similarity in the latent space implies semantic relatedness. We describe three ways in which labeled data can improve the accuracy of these approaches on paraphrase classification. First, we design a new discriminative term-weighting metric called TF-KLD, which outperforms TF-IDF. Next, we show that using the latent representation from matrix factorization as features in a classification algorithm substantially improves accuracy. Finally, we combine latent features with fine-grained n-gram overlap features, yielding performance that is 3% more accurate than the prior state-of-the-art.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohit Bansal
2. Luke S. Zettlemoyer
3. Graham Neubig
4. Jianfeng Gao
5. Kevin Gimpel
6. Hinrich Schütze
7. Douwe Kiela
8. Kyunghyun Cho

In Natural Language Processing

1. Mohit Bansal
2. Kevin Gimpel
3. John Wieting
4. Douwe Kiela
5. Hinrich Schütze
6. Graham Neubig
7. Jimmy J. Lin
8. Luke S. Zettlemoyer

Citations in the corpus

• An efficient framework for learning sentence representations (2018, methodology): “The models are AdaSent (Zhao et al., 2015), CNN (Kim, 2014), TF-KLD (Ji & Eisenstein, 2013) and Dependency-Tree LSTM (Tai et al., 2015).”
• Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, methodology): “ates methods that we trained, other transfer models have been extracted from (Hill et al., 2016). For best published supervised methods (no transfer), we consider AdaSent (Zhao et al., 2015), TF-KLD (Ji and Eisenstein, 2013), Tree-LSTM (Tai et al., 2015) and Illinois-LH system (Lai and Hockenmaier, ”
• Skip-Thought Vectors (2015, neutral): “1 TF-KLD [29] 80.”
• SentEval: An Evaluation Toolkit for Universal Sentence Representations (2018, neutral): “Results 1 correspond to AdaSent (Zhao et al., 2015), 2 to BLSTM-2DCNN (Zhou et al., 2016), 3 to TF-KLD (Ji and Eisenstein, 2013) and 4 to Illinois-LH system (Lai and Hockenmaier, 2014).”
• Convolutional Neural Network for Paraphrase Identification (2015, methodology): “Second, (Socher et al., 2011) is partially based on parsing as is some other work on paraphrase iden-tification (e.g., Wan et al. (2006), Ji and Eisenstein (2013)).”
• Multi-Perspective Sentence Similarity Modeling with Convolutional Neural Networks (2015, methodology): “Ji and Eisenstein (2013) used matrix factorization techniques to obtain sentence representations, and combined them with fine-tuned sparse features us-ing an SVM classifier for similarity prediction.”
• ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations (2017, neutral): “, 2017)y 81.1 86.3 92.4 90.2 84.6 88.2 76.2/83.1 0.884 86.3 Supervised (Direct) Naive Bayes - SVM 79.4 81.8 93.2 86.3 83.1 - - - - AdaSent (Zhao et al., 2015) 83.1 86.3 95.5 93.3 - 92.4 - - - TF-KLD (Ji and Eisenstein, 2013) - - - - - - 80.4/85.9 - - Illinois-LH (Lai and Hockenmaier, 2014) - - - - -”
• PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification (2015, background): “, 2014), measuring the semantic similarity of texts (Han et al., 2013; Ji and Eisenstein, 2013; Sultan et al., 2014b), monolingual alignment (Yao et al.”
• Modelling Sentence Pairs with Tree-structured Attentive Encoder (2016, methodology): “This dataset is derived from the AI2 Elementary School Science Questions released by Allen Institute.”
• Short Text Similarity with Word Embeddings (2015, methodology): “Another strong method is presented in [22] where features from dependency parser are used to train a supervised method.”
• A Continuously Growing Dataset of Sentential Paraphrases (2017, methodology): “This is an open-sourced adaptation (Xu et al., 2014) of LEXDIS-CRIM (Ji and Eisenstein, 2013)”
• Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations (2017, unknown): “, 2017)y 81.1 86.3 92.4 90.2 84.6 88.2 76.2/83.1 0.884 86.3 Supervised (Direct) Naive Bayes - SVM 79.4 81.8 93.2 86.3 83.1 - - - - AdaSent (Zhao et al., 2015) 83.1 86.3 95.5 93.3 - 92.4 - - - TF-KLD (Ji and Eisenstein, 2013) - - - - - - 80.4/85.9 - - Illinois-LH (Lai and Hockenmaier, 2014) - - - - -”
• MultiGranCNN: An Architecture for General Matching of Text Chunks on Multiple Levels of Granularity (2015, background): “Paraphrase identification (PI) is a typical task of sentence matching and it has been frequently studied (Qiu et al., 2006; Blacoe and Lapata, 2012; Madnani et al., 2012; Ji and Eisenstein, 2013).”
• SemEval-2015 Task 1: Paraphrase and Semantic Similarity in Twitter (PIT) (2015, methodology): “These can be categorized into supervised methods (Qiu et al., 2006; Wan et al., 2006; Das and Smith, 2009; Socher et al., 2011; Blacoe and Lapata, 2012; Madnani et al., 2012; Ji and Eisenstein, 2013) and unsupervised methods (Mihalcea et al.”