Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations

By Andreas Ruckl'e, Steffen Eger, Maxime Peyrard, Iryna Gurevych. 2018

102 citations recorded by the source corpus.

Average word embeddings are a common baseline for more sophisticated sentence embedding techniques. However, they typically fall short of the performances of more complex models such as InferSent. Here, we generalize the concept of average word embeddings to power mean word embeddings. We show that the concatenation of different types of power mean word embeddings considerably closes the gap to state-of-the-art methods monolingually and substantially outperforms these more complex techniques cross-lingually. In addition, our proposed method outperforms different recently proposed baselines such as SIF and Sent2Vec by a solid margin, thus constituting a much harder-to-beat monolingual baseline.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Steffen Eger
2. Mohit Bansal
3. Yang Gao
4. Wei Zhao
5. Iryna Gurevych
6. Percy Liang
7. Yejin Choi
8. Graham Neubig

In Natural Language Processing

1. Iryna Gurevych
2. Wei Zhao
3. Steffen Eger
4. Yang Gao
5. Mohit Bansal
6. Yejin Choi
7. Pengfei Liu
8. Dan Roth

Citations in the corpus

• Better Rewards Yield Better Summaries: Learning to Summarise Without References (2019, methodology): “PMeans is a simple yet powerful sentence encoding method (R¨uckl´e et al., 2018).”
• BERTScore: Evaluating Text Generation with BERT (2019, background): “[PMEANS]Apply power means (Rckl et al., 2018) to aggregate the information of different layers.”
• MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance (2019, methodology): “We study two alternatives for φ : (i) the concatenation of power means (R¨uckl´e et al., 2018) as a generalized pooling mechanism, and (ii) a routing mechanism for aggregation (Zhao et al., 2018, 2019).”
• Survey on reinforcement learning for language processing (2021, methodology): “For the representations of utterances, Word2vec has been extended to Doc2vec [64], and other simple schemes are based on a weighted combination of contained word vectors [2, 96].”