Charagram: Embedding Words and Sentences via Character n-grams

By John Wieting, Mohit Bansal, Kevin Gimpel, Karen Livescu. 2016

197 citations recorded by the source corpus.

We present Charagram embeddings, a simple approach for learning character-based compositional models to embed textual sequences. A word or sentence is represented using a character n-gram count vector, followed by a single nonlinear transformation to yield a low-dimensional embedding. We use three tasks for evaluation: word similarity, sentence similarity, and part-of-speech tagging. We demonstrate that Charagram embeddings outperform more complex architectures based on character-level recurrent and convolutional neural networks, achieving new state-of-the-art performance on several similarity tasks.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Deep Contextualized Word Representations (2018, methodology): “Previously proposed methods overcome some of the shortcomings of traditional word vectors by either enriching them with subword information (e.g., Wieting et al., 2016; Bojanowski et al., 2017) or learning separate vectors for each word sense (e.g., Neelakantan et al., 2014).”
Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, background): “Recent work on generating sentence embeddings range from models that compose word embeddings (Le and Mikolov, 2014; Arora et al., 2017; Wieting et al., 2016) to more complex neural network architectures.”
SentEval: An Evaluation Toolkit for Universal Sentence Representations (2018, neutral): “Charagram-phrase numbers were taken from (Wieting et al., 2016).”
Revisiting Recurrent Networks for Paraphrastic Sentence Embeddings (2017, positive): “We consider the problem of learning general-purpose, paraphrastic sentence embeddings, revisiting the setting of Wieting et al. (2016b). While they”
Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translation (2018, background): “There has also been work in enriching these word representations using sub-word information (Wieting et al. 2016; Bojanowski et al. 2017).”
Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features (2017, background): “…embeddings very signiﬁcantly exceeds the state of the art, while keeping the model simplicity as well as training and inference complexity exactly as low as in averaging methods (Wieting et al., 2016a; Arora et al., 2017), thereby also putting the title of (Arora et al., 2017) in perspective.”
ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations (2017, methodology): “sfer, and sentence simpliﬁcation to core NLP problems like machine translation. We show the utility of PARANMT-50M by using it to train paraphrastic sentence embeddings using the learning framework ofWieting et al. (2016b). We primarily evaluate our sentence embeddings on the SemEval semantic textua”
Rare Words: A Major Problem for Contextualized Embeddings And How to Fix it by Attentive Mimicking (2019, methodology): “The former use subword information to infer a word’s meaning; this includes n -grams (Wieting et al. 2016; Bojanowski et al. 2017; Salle and Villavicencio 2018), morphemes (Lazari-dou et al. 2013; Luong, Socher, and Manning 2013) and characters (Pinter, Guthrie, and Eisenstein 2017).”
Dynamic Language Models for Continuously Evolving Content (2021, background): “In a separate line of studies, researchers break-down a word to more fine-grained units, including characters [2, 20, 28, 32, 49], character-based ngrams [7, 41, 45], and subwords (eg, wordpiece [46] and byte-pairencodings [22, 37]).”
A Bilingual Generative Transformer for Semantic Sentence Embedding (2019, methodology): “For instance, some of the top performing techniques use word embedding averaging (Wieting et al., 2016b), character n-grams (Wieting et al., 2016a), and subword embedding averaging (Wieting et al., 2019b) to create representations.”
Neural Paraphrase Identification of Questions with Noisy Pretraining (2017, methodology): “Our pretraining procedure is reminiscent of several recent papers (Wieting et al., 2016, inter alia) who aim for general purpose character n-gram embeddings.”
Learning Semantic Representations for Novel Words: Leveraging Both Form and Context (2018, background): “For this reason, another direction chosen by several authors is to resort to n -grams instead of morphemes (Wieting et al. 2016; Ataman and Federico 2018).”
Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints (2017, methodology): “State-of-the-Art Wieting et al. (2016) note that the hyperparameters of the widely used Paragram-SL999 vectors (Wieting et al., 2015) are tuned on SimLex-999, and as such are not comparable to methods which holdout the dataset.”