Exploring Generalization Ability of Pretrained Language Models on Arithmetic and Logical Reasoning

By Cunxiang Wang, Boyuan Zheng, Y. Niu, Yue Zhang. 2021

24 citations recorded by the source corpus.

To quantitatively and intuitively explore the generalization ability of pre-trained language models (PLMs), we have designed several tasks of arithmetic and logical reasoning. We both analyse how well PLMs generalize when the test data is in the same distribution as the train data and when it is different, for the latter analysis, we have also designed a cross-distribution test set other than the in-distribution test set. We conduct experiments on one of the most advanced and publicly released generative PLM - BART. Our research finds that the PLMs can easily generalize when the distribution is the same, however, it is still difficult for them to generalize out of the distribution.

Subfields: Artificial Intelligence, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Carrie J. Cai
2. Michael Terry
3. Tongshuang Sherry Wu
4. Chitta Baral
5. Swaroop Mishra
6. Mihir Parmar
7. Percy Liang
8. Man Luo

In Prompting & Reasoning

1. Carrie J. Cai
2. Swaroop Mishra
3. Chitta Baral
4. Mihir Parmar
5. Man Luo
6. Percy Liang

Citations in the corpus

• AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts (2021, background): “Because LLMs are designed to grasp the form of language, rather than the meaning [7], they can struggle on tasks like sequential arithmetic problems, multi-hop question answering, recognizing and comparing sentences, or those that require branching logic [9, 11, 26, 66, 67].”