Exploring Numeracy in Word Embeddings

By Aakanksha Naik, Abhilasha Ravichander, Carolyn Penstein Rosé, Eduard H. Hovy. 2019

65 citations recorded by the source corpus.

Word embeddings are now pervasive across NLP subfields as the de-facto method of forming text representataions. In this work, we show that existing embedding models are inadequate at constructing representations that capture salient aspects of mathematical meaning for numbers, which is important for language understanding. Numbers are ubiquitous and frequently appear in text. Inspired by cognitive studies on how humans perceive numbers, we develop an analysis framework to test how well word embeddings capture two essential properties of numbers: magnitude (e.g. 3<4) and numeration (e.g. 3=three). Our experiments reveal that most models capture an approximate notion of magnitude, but are inadequate at capturing numeration. We hope that our observations provide a starting point for the development of methods which better capture numeracy in NLP systems.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2021, background): “Within the context of pre-trained LMs, researchers have found that pre-trained embeddings and LMs can perform simple operations such as addition and subtraction when the number of digits is small but fail when the numbers are larger [9, 88, 139].”
CausaLM: Causal Model Explanation Through Counterfactual Language Models (2020, methodology): “Behavioral methods evaluate models on specific examples that reflect an hypothesis regarding linguistic phenomena they capture (Sennrich 2017; Isabelle, Cherry, and Foster 2017; Naik et al. 2019).”
Do NLP Models Know Numbers? Probing Numeracy in Embeddings (2019, background): “In concurrent work, Naik et al. (2019) also explore numeracy in word vectors.”
Investigating the Limitations of Transformers with Simple Arithmetic Tasks (2021, methodology): “Recent studies have explored the numerical capabilities learned by neural networks trained on large amounts of texts (Talmor et al., 2019; Jiang et al., 2019; Naik et al., 2019; Wallace et al., 2019; Lin et al., 2020; Johnson et al., 2020; Mishra et al., 2020).”
UnNatural Language Inference (2020, background): “We also ﬁnd that for both experts, accuracy on per-mutations from D c was higher than on D f , which veriﬁes ﬁndings that showed high word overlap can give hints about the ground truth label (Dasgupta et al., 2018; Poliak et al., 2018; Gururangan et al., 2018; Naik et al., 2019).”
Representing Numbers in NLP: a Survey and a Vision (2021, methodology): “…task in NLP from previously published experiments over a subset of the described number representations: Abstract Probes Word Embeddings vastly out-perform random embedding baselines on abstract probes such as numeration, magnitude comparison, and sorting (Wallace et al., 2019; Naik et al., 2019).”
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, background): “[171] Aakanksha Naik, Abhilasha Ravichander, Carolyn Rose, and Eduard Hovy.”
Math Word Problem Solving with Explicit Numerical Values (2021, background): “Numerical Value Representations: Some recent studies have explored the numerical value representations in language models (Naik et al., 2019; Chen et al., 2019; Wallace et al., 2019).”
Mapping Language Models to Grounded Conceptual Spaces (2022, unknown): “We describe each body of work below.”
Investigating the Limitations of the Transformers with Simple Arithmetic Tasks (2021, unknown): “A common finding is that the learned embeddings capture magnitude (e.g., 2 < 3), but many models fail to capture numeracy (e.g., two=2) (Naik et al., 2019; Wallace et al., 2019; Ren & Du, 2020; Zhang et al., 2020).”