Reasoning with Transformer-based Models: Deep Learning, but Shallow Reasoning

61 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Reasoning Like Program Executors (2022, background): “Meanwhile, neural models are notorious for their reliance on correlations among concrete tokens of a representation system and are usually assumed to be hard to grasp abstract rules of a symbolic reasoner (Helwe et al., 2021; Sinha et al., 2021).”
Faith and Fate: Limits of Transformers on Compositionality (2023, methodology): “We observe that all analyzed models match the Relative Information Gain prediction that o 1 (whether the first element goes into the output sequence or not) should be the easiest value to predict (see Figures 30, 31, and 32).”
Understanding Natural Language Understanding Systems. A Critical Analysis (2023, background): “Mahowald et al. (2023) review evidence revealing that in all these areas even the largest models like GPT-3 show performances still very far from human ones (Helwe, Clavel and Suchanek 2021, Kauf et al. 2022, Schuster and Linzen 2022, among several others).”
A Review on Language Models as Knowledge Bases (2022, background): “Furthermore, previous work demonstrates that there are theoretical limitations to transformers that prevents them from performing certain types of reasoning tasks (Hahn, 2020; Bhattamishra et al., 2020; Helwe et al., 2021).”