Do Language Embeddings capture Scales?

By Xikun Zhang, Deepak Ramachandran, Ian Tenney, Yanai Elazar, Dan Roth. 2020

100 citations recorded by the source corpus.

Pretrained Language Models (LMs) have been shown to possess significant linguistic, common sense and factual knowledge. One form of knowledge that has not been studied yet in this context is information about the scalar magnitudes of objects. We show that pretrained language models capture a significant amount of this information but are short of the capability required for general common-sense reasoning. We identify contextual information in pre-training and numeracy as two key factors affecting their performance, and show that a simple method of canonicalizing numbers can have a significant effect on the results.

Subfields: Artificial Intelligence, Literature and Literary Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Investigating the Limitations of Transformers with Simple Arithmetic Tasks (2021, methodology): “…in this representation: (1) the T5 model was pretrained on large amounts of textual data, so it likely knows that “hundred” is larger than “ten” (Zhang et al., 2020); (2) Digits are surrounded by tokens that describe their signiﬁcance (“hundred”, “thousand”, etc.), thus making it easier to ﬁnd…”
Investigating the Limitations of the Transformers with Simple Arithmetic Tasks (2021, unknown): “We can anticipate two advantages in this representation: (1) the T5 model was pretrained on large amounts of textual data, so it likely knows that “hundred” is larger than “ten” (Zhang et al., 2020); (2) Digits are surrounded by tokens that describe their significance (“hundred”, “thousand”, etc.”
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, background): “[170] Xikun Zhang, Deepak Ramachandran, Ian Tenney, Yanai Elazar, and Dan Roth.”
Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks (2023, background): “…being trained on only textual data, LMs have been shown to be able to structure their representations of perceptual concepts such as size and color (Abdou et al., 2021; Patel and Pavlick, 2022; Zhang et al., 2020; Ilharco et al., 2021; i.a. ) in a way that credibly mirrors the physical world.”
Representing Numbers in NLP: a Survey and a Vision (2021, methodology): “MCC or multi-class classiﬁcation is another number decoder which outputs a distribution, but a discrete one: over log-scaled bins of numbers, e.g., 1-10, 10-100, and so on (Zhang et al., 2020).”
A Survey of Deep Learning for Mathematical Reasoning (2022, unknown): “Previous literature has been limited to the discussion of speciﬁc aspects, such as solving math word problems (Bhattacharya, 2017; Zhang et al., 2019; Ughade and Kumbhar, 2019), representing numbers representation (Thawani et al., 2021), or solving informal problems (Meadows and Freitas, 2022).”