Representation learning for very short texts using weighted word embedding aggregation

By Cedric De Boom, Steven Van Canneyt, Thomas Demeester, Bart Dhoedt. 2016

197 citations recorded by the source corpus.

We create text representations by weighing word embeddings using idf information.A novel median-based loss is designed to mitigate the negative effect of outliers.A dataset of semantically related textual pairs from Wikipedia and Twitter is made.Our method outperforms all word embedding baselines in a semantic similarity task.Our method is out-of-the-box and thus requires no retraining in different contexts. Short text messages such as tweets are very noisy and sparse in their use of vocabulary. Traditional textual representations, such as tf-idf, have difficulty grasping the semantic meaning of such texts, which is important in applications such as event detection, opinion mining, news recommendation, etc. We constructed a method based on semantic word embeddings and frequency information to arrive at low-dimensional representations for short texts designed to capture semantic similarity. For this purpose we designed a weight-based model and a learning procedure based on a novel median-based loss function. This paper discusses the details of our model and the optimization methods, together with the experimental results on both Wikipedia and Twitter data. We find that our method outperforms the baseline approaches in the experiments, and that it generalizes well on different word embeddings without retraining. Our method is therefore capable of retaining most of the semantic information in the text, and is applicable out-of-the-box.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Pretrained Transformers for Text Ranking: BERT and Beyond (2020, background): “In fact, ranking with dense representations predates BERT by many years [6, 11, 15, 29, 38, 41].”
NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding (2019, background): “In these works, each word is mapped into a continuous embedding space, and two semantically relevant words will have large cosine similarity in this space [96], [97], [98].”
Enhancing Crowd Wisdom Using Explainable Diversity Inferred from Social Media (2018, methodology): “Moreover, they used a traditional word vector-based user representation and computed diversity as the cosine distance between these users [20].”