When a sentence does not introduce a discourse entity, Transformer-based models still sometimes refer to it
By Sebastian Schuster, Tal Linzen. 2022
30 citations recorded by the source corpus.
Understanding longer narratives or participating in conversations requires tracking of discourse entities that have been mentioned. Indefinite noun phrases (NPs), such as ‘a dog’, frequently introduce discourse entities but this behavior is modulated by sentential operators such as negation. For example, ‘a dog’ in ‘Arthur doesn’t own a dog’ does not introduce a discourse entity due to the presence of negation. In this work, we adapt the psycholinguistic assessment of language models paradigm to higher-level linguistic phenomena and introduce an English evaluation suite that targets the knowledge of the interactions between sentential operators and indefinite NPs. We use this evaluation suite for a fine-grained investigation of the entity tracking abilities of the Transformer-based models GPT-2 and GPT-3. We find that while the models are to a certain extent sensitive to the interactions we investigate, they are all challenged by the presence of multiple NPs and their behavior is not systematic, which suggests that even models at the scale of GPT-3 do not fully acquire basic entity tracking abilities.
Subfields: Natural Language Processing, Computational Linguistics & Semantics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Natural Language Processing
- Yejin Choi
- Ronan Le Bras
- Maarten Sap
- Daniel Fried
- Hannaneh Hajishirzi
- Alessandro Lenci
- Noah A. Smith
- Mohit Bansal
In Computational Linguistics & Semantics
Citations in the corpus
- Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (2022, background): “, 2021) or examining how LLMs handle entities and people (Onoe et al., 2022; Schuster and Linzen, 2022), which could shed light onto better data choices and inductive biases towards neural Theory of Mind and social intelligence.”
- Understanding Natural Language Understanding Systems. A Critical Analysis (2023, background): “Mahowald et al. (2023) review evidence revealing that in all these areas even the largest models like GPT-3 show performances still very far from human ones (Helwe, Clavel and Suchanek 2021, Kauf et al. 2022, Schuster and Linzen 2022, among several others).”