Evaluating the Stability of Embedding-based Word Similarities
By Maria Antoniak, David M. Mimno. 2018
190 citations recorded by the source corpus.
Word embeddings are increasingly being used as a tool to study word associations in specific corpora. However, it is unclear whether such embeddings reflect enduring properties of language or if they are sensitive to inconsequential variations in the source documents. We find that nearest-neighbor distances are highly sensitive to small changes in the training corpus for a variety of algorithms. For all methods, including specific documents in the training set can result in substantial variations. We show that these effects are more prominent for smaller training corpora. We recommend that users never rely on single embedding models for distance calculations, but rather average over multiple bootstrap samples, especially for small corpora.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Katherine A. Keith
- David D. Jensen
- Jose Miguel Such
- Brendan T. O'Connor
- Natalia Criado
- Michael D. Conover
- Tom van Nuenen
- Xavier Ferrer Aran
Citations in the corpus
- Text and Causal Inference: A Review of Using Text to Remove Confounding from Causal Estimates (2020, background): “, 2015) and the construction of the training corpus (Antoniak and Mimno, 2018); and fine-tuned language model performance is sensitive to random restarts (Phang et al.”
- Discovering and Categorising Language Biases in Reddit (2020, methodology): “To test the stability of our approach we created 10 bootstrapped models of /r/TheRedPill in a similar way as done by (Antoniak and Mimno 2018), including randomly sampling 50% of the original dataset and averaging the results over the multiple bootstrapped samples.”
- Considerations for the Interpretation of Bias Measures of Word Embeddings (2019, background): “In this light, it becomes critical to understand the sensitivity of the proposed metric to the particular term set sampled, and ideally, characterize the distribution of the metric under many such samples as partially explored in (Antoniak and Mimno 2018).”