pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference
By Mandar Joshi, Eunsol Choi, Omer Levy, Daniel S. Weld, Luke S. Zettlemoyer. 2018
49 citations recorded by the source corpus.
Reasoning about implied relationships (e.g. paraphrastic, common sense, encyclopedic) between pairs of words is crucial for many cross-sentence inference problems. This paper proposes new methods for learning and using embeddings of word pairs that implicitly represent background knowledge about such relationships. Our pairwise embeddings are computed as a compositional function of each word’s representation, which is learned by maximizing the pointwise mutual information (PMI) with the contexts in which the the two words co-occur. We add these representations to the cross-sentence attention layer of existing inference models (e.g. BiDAF for QA, ESIM for NLI), instead of extending or replacing existing word embeddings. Experiments show a gain of 2.7% on the recently released SQuAD 2.0 and 1.3% on MultiNLI. Our representations also aid in better generalization with gains of around 6-7% on adversarial SQuAD datasets, and 8.8% on the adversarial entailment test set by Glockner et al. (2018).
Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Luke S. Zettlemoyer
- Mohit Bansal
- Percy Liang
- Danqi Chen
- Hannaneh Hajishirzi
- Yejin Choi
- Omer Levy
- Graham Neubig
In Natural Language Processing
- Luke S. Zettlemoyer
- Yejin Choi
- Mohit Bansal
- Omer Levy
- Percy Liang
- Danqi Chen
- Hinrich Schütze
- Hannaneh Hajishirzi
In Representation & Self-Supervised Learning
- Omer Levy
- Danqi Chen
- Luke S. Zettlemoyer
- Mandar Joshi
- José Camacho-Collados
- Daniel S. Weld
- Yinhan Liu
- Zied Bouraoui
Citations in the corpus
- SpanBERT: Improving Pre-training by Representing and Predicting Spans (2019, methodology): “Related to our SBO objective, pair2vec (Joshi et al., 2019) encodes word-pair relations using a negative sampling-based multivariate objective during pre-training.”
- The Effect of Natural Distribution Shift on Question Answering Models (2020, neutral): “BiDAF++ (Single) (Joshi et al., 2019) https://worksheets.”
- Inducing Relational Knowledge from BERT (2019, background): “vectors encoding the relationship between two words (Washio and Kato 2018a; Jameel, Bouraoui, and Schockaert 2018; Espinosa Anke and Schockaert 2018; Joshi et al. 2019; Washio and Kato 2018b; Camacho-Collados et al. 2019).”