Hash Embeddings for Efficient Word Representations

By Dan Svenstrup, Jonas Meinertz Hansen, Ole Winther. 2017

99 citations recorded by the source corpus.

We present hash embeddings, an efficient method for representing words in a continuous vector form. A hash embedding may be seen as an interpolation between a standard word embedding and a word embedding created using a random hash function (the hashing trick). In hash embeddings each token is represented by $k$ $d$-dimensional embeddings vectors and one $k$ dimensional weight vector. The final $d$ dimensional representation of the token is the product of the two. Rather than fitting the embedding vectors for each token these are selected by the hashing trick from a shared pool of $B$ embedding vectors. Our experiments show that hash embeddings can easily deal with huge vocabularies consisting of millions of tokens. When using a hash embedding there is no need to create a dictionary before training nor to perform any kind of vocabulary pruning after training. We show that models trained using hash embeddings exhibit at least the same level of performance as models trained using regular embeddings across a wide range of tasks. Furthermore, the number of parameters needed by such an embedding is only a fraction of what is required by a regular embedding. Since standard embeddings and embeddings constructed using the hashing trick are actually just special cases of a hash embedding, hash embeddings can be considered an extension and improvement over the existing regular embedding types.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Efficiency & Compression

Citations in the corpus

Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, methodology): “While [14] still rely on a relatively sophisticated pipeline that maps Unicode codepoints to hash embeddings [77], we embed raw UTF-8 bytes directly.”
Survey on categorical data for neural networks (2020, methodology): “We find research that challenges Chollet’s assumption in “Hash embeddings for efficient word representations” [59].”
Learning to Embed Categorical Features without Embedding Tables for Recommendation (2020, methodology): “A common aggregation approach is ‘add’ [30, 34, 38], which can be simply expressed as e = F (b) = W b = W [b(1) ; b(2) ; .”
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP (2021, methodology): “One possible solution is used in Clark et al. (2021)’s CANINE, where instead of giving each character its own embedding, they hash all possible codepoints with multiple hash functions into smaller sets to share parameters across codepoints (similar to Svenstrup et al. (2017)).”
Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation (2021, methodology): “Character hash embeddings: CANINE embeds these codepoint integers using multiple hash functions, a vocabulary-free generalization of the word hash embedding trick (Svenstrup et al., 2017).”