Rethinking Skip-thought: A Neighborhood based Approach

By Shuai Tang, Hailin Jin, Chen Fang, Zhaowen Wang, Virginia de Sa. 2017

17 citations recorded by the source corpus.

We study the skip-thought model with neighborhood information as weak supervision. More specifically, we propose a skip-thought neighbor model to consider the adjacent sentences as a neighborhood. We train our skip-thought neighbor model on a large corpus with continuous sentences, and then evaluate the trained model on 7 tasks, which include semantic relatedness, paraphrase detection, and classification benchmarks. Both quantitative comparison and qualitative investigation are conducted. We empirically show that, our skip-thought neighbor model performs as well as the skip-thought model on evaluation tasks. In addition, we found that, incorporating an autoencoder path in our model didn’t aid our model to perform better, while it hurts the performance of the skip-thought model.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Samuel R. Bowman
2. Yonatan Belinkov
3. Mohit Bansal
4. Ellie Pavlick
5. Tal Linzen
6. Yejin Choi
7. Luke S. Zettlemoyer
8. Yoav Goldberg

In Natural Language Processing

1. Samuel R. Bowman
2. Yonatan Belinkov
3. Ellie Pavlick
4. Tal Linzen
5. Mohit Bansal
6. Yejin Choi
7. Yoav Goldberg
8. Graham Neubig

In Representation & Self-Supervised Learning

1. Samuel R. Bowman
2. C. Pal
3. Alex Wang
4. Adam Trischler
5. Yoshua Bengio
6. Ellie Pavlick
7. Edouard Grave
8. Ian Tenney

Citations in the corpus

• Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (2018, methodology): “We use the original splits. sentence from WT and predict the following sentence (Kiros et al., 2015; Tang et al., 2017).”
• Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (2018, background): “The first 6 rows are taken from Wang et al. (2017), the next 4 are from Tomar et al.”
• Probing What Different NLP Tasks Teach Machines about Function Word Comprehension (2019, methodology): “Skip-Thought Drawing from Kiros et al. (2015) and Tang et al. (2017), we train a sequence-to-sequence model on skip-thought, which is a task of generating the next sentence in the discourse given the previous sentence.”
• Looking for ELMo's friends: Sentence-Level Pretraining Beyond Language Modeling (2018, methodology): “SkipThought Our SkipThought model (Kiros et al., 2015; Tang et al., 2017) is a sequence-tosequence model that reads a sentence from WikiText-103 running text and attempts to decode the following sentence from that text.”