A Closer Look at Skip-gram Modelling

By David Guthrie, Ben Allison, Wei Liu, Louise Guthrie, Yorick Wilks. International Conference on Language Resources and Evaluation, 2006

375 citations recorded by the source corpus.

Data sparsity is a large problem in natural language processing that refers to the fact that language is a system of rare events, so varied and complex, that even using an extremely large corpus, we can never accurately model all possible strings of words. This paper examines the use of skip-grams (a technique where by n-grams are still stored to model language, but they allow for tokens to be skipped) to overcome the data sparsity problem. We analyze this by computing all possible skip-grams in a training corpus and measure how many adjacent (standard) n-grams these cover in test documents. We examine skip-gram modelling using one to four skips with various amount of training data and test against similar documents as well as documents generated from a machine translation system. In this paper we also determine the amount of extra training data required to achieve skip-gram coverage using standard adjacent tri-grams.

Subfields: Artificial Intelligence, Computer Networks and Communications, Information Systems

View Paper: https://www.semanticscholar.org/paper/a0968b29aa9d4d5aae6456d7c1dbbe62fbfa9b0d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Information Systems

Citations in the corpus

Online Learning with Gated Linear Networks (2017, methodology): “ach pixel; a row-major ordering was used to linearize the two dimensional pixel locations. Base Layer The base layer for each GLN was determined by taking the outputs of a number of skip-gram models (Guthrie et al., 2006). The context functions used by these skip-gram models included a number of spe”
A multidimensional approach for detecting irony in Twitter (2013, background): “But instead of looking for sequences of adjacent words (simple n-grams), we look for word sequences that contain (or skip over) arbitrary gaps; hence the name skip-grams (cf. Guthrie et al. 2006; Chin-Yew and Och 2004).”
SciTaiL: A Textual Entailment Dataset from Science Question Answering (2018, methodology): “We compute the proportion of unigrams, 1skip bigrams, and 1-skip trigrams (Guthrie et al. 2006) in the hypothesis that are also present in the premise as three features13.”
Skip N-grams and Ranking Functions for Predicting Script Events (2012, methodology): “This approach to collecting n-gram statistics is sometimes called skip-gram modeling, and it can reduce data sparsity by extracting more event pairs per chain (Guthrie et al., 2006).”