Encoding word order in complex embeddings

By Benyou Wang, Donghao Zhao, Christina Lioma, Qiuchi Li, Peng Zhang, Jakob Grue Simonsen. 2019

137 citations recorded by the source corpus.

Sequential word order is important when processing text. Currently, neural networks (NNs) address this by modeling word position using position embeddings. The problem is that position embeddings capture the position of individual words, but not the ordered relationship (e.g., adjacency or precedence) between individual word positions. We present a novel and principled solution for modeling both the global absolute positions of words and their order relationships. Our solution generalizes word embeddings, previously defined as independent vectors, to continuous word functions over a variable (position). The benefit of continuous functions over variable positions is that word representations shift smoothly with increasing positions. Hence, word representations in different positions can correlate with each other in a continuous function. The general solution of these functions can be extended to complex-valued variants. We extend CNN, RNN and Transformer NNs to complex-valued versions to incorporate our complex embedding (we make all code available). Experiments on text classification, machine translation and language modeling show gains over both classical word embeddings and position-enriched word embeddings. To our knowledge, this is the first work in NLP to link imaginary numbers in complex-valued representations to concrete meanings (i.e., word order).

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

RoFormer: Enhanced Transformer with Rotary Position Embedding (2021, methodology): “In addition to these approaches, the authors of Liu et al. [2020] have proposed to model the dependency of position encoding from the perspective of Neural ODE Chen et al. [2018a], and the authors of Wang et al. [2020] have proposed to model the position information in complex space.”
Rethinking Positional Encoding in Language Pre-training (2020, background): “For example, Shiv & Quirk (2019) extended the sequence positional encoding to tree-based positional encoding in Transformer; Wang et al. (2019) extended positional encoding to complex-valued domain; Liu et al. (2020) modeled the positional encoding by dynamical systems.”
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, background): “Recent attempts [133, 173, 174, 175, 176] propose new encoding schemes of numbers to improve the LMs’ numerical capabilities, probably helping LIFT in the regression tasks.”
Conditional Positional Encodings for Vision Transformers (2021, background): “Complex-value embeddings (Wang et al., 2019) are an extension to model global absolute encodings and show improvement.”
A Survey of Transformers (2021, background): “[139] propose to encode positional information in word embeddings, by generalizing embedding to continuous (complex-valued) functions over positions.”
Investigating the Limitations of the Transformers with Simple Arithmetic Tasks (2021, unknown): “Novel ways of encoding positions of tokens in the transformer architecture were proposed, but they were mostly evaluated on natural language processing tasks, showing small performance gains (Ke et al., 2020; He et al., 2020; Wang et al., 2019; Huang et al., 2020).”
What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding (2020, methodology): “Furthermore, Wang et al. (2019) extended the embedding space from real numbers to complex values , and also proposed a new learnable positional encoding function instead of a simple position embedding mapping.”