Pre-training Text-to-Text Transformers for Concept-centric Common Sense

By Wangchunshu Zhou, Dong-Ho Lee, Ravi Kiran Selvam, Seyeon Lee, Bill Yuchen Lin, Xiang Ren. 2020

73 citations recorded by the source corpus.

Pre-trained language models (PTLM) have achieved impressive results in a range of natural language understanding (NLU) and generation (NLG) tasks. However, current pre-training objectives such as masked token prediction (for BERT-style PTLMs) and masked span infilling (for T5-style PTLMs) do not explicitly model the relational commonsense knowledge about everyday concepts, which is crucial to many downstream tasks that need common sense to understand or generate. To augment PTLMs with concept-centric commonsense knowledge, in this paper, we propose both generative and contrastive objectives for learning common sense from the text, and use them as intermediate self-supervised learning tasks for incrementally pre-training PTLMs (before task-specific fine-tuning on downstream datasets). Furthermore, we develop a joint pre-training framework to unify generative and contrastive objectives so that they can mutually reinforce each other. Extensive experimental results show that our method, concept-aware language model (CALM), can pack more commonsense knowledge into the parameters of a pre-trained text-to-text transformer without relying on external knowledge graphs, yielding better performance on both NLU and NLG tasks. We show that while only incrementally pre-trained on a relatively small corpus for a few steps, CALM outperforms baseline methods by a consistent margin and even comparable with some larger PTLMs, which suggests that CALM can serve as a general, plug-and-play method for improving the commonsense reasoning ability of a PTLM.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Xiang Ren
3. Ji-rong Wen
4. Chenguang Zhu
5. Wayne Xin Zhao
6. Yang Liu
7. Mohit Bansal
8. Yejin Choi

In Computer Vision and Pattern Recognition

1. Jianfeng Gao
2. Xiang Ren
3. Chenguang Zhu
4. Yonatan Bisk
5. Ji-rong Wen
6. Yang Liu
7. Wayne Xin Zhao
8. Michael Zeng

In Large Language Models

1. Xiang Ren
2. Chenguang Zhu
3. Yang Liu
4. Ji-rong Wen
5. Wayne Xin Zhao
6. Michael Zeng
7. Jian-Yun Nie
8. Kun Zhou

In Natural Language Processing

1. Jianfeng Gao
2. Xiang Ren
3. Chenguang Zhu
4. Yang Liu
5. Michael Zeng
6. Yejin Choi
7. Ji-rong Wen
8. Wayne Xin Zhao

In Model Architectures

1. Xiang Ren
2. Jian-Yun Nie
3. Kun Zhou
4. Ji-rong Wen
5. Jianfeng Gao
6. Daniel J. McDuff
7. Shijin Wang
8. Yichong Xu

Citations in the corpus

• Continual Pre-training of Language Models for Math Problem Understanding with Syntax-Aware Memory Network (2022, methodology): “…the model parameters with BERT-like tasks on domain or task related corpus ( e.g., scientific (Beltagy et al., 2019) and bio-media (Lee et al., 2020)), or design new pre-training objectives for task adaption ( e.g., commonsense reasoning (Zhou et al., 2021) and dialogue adaption (Li et al., 2020)).”
• A review of ensemble learning and data augmentation models for class imbalanced problems: combination, implementation and evaluation (2023, methodology): “Hence, it is important to use tree-based ensemble models (XGBoost and Random Forests) with data augmentation methods in such problem domains.”
• Leveraging Visual Knowledge in Language Tasks: An Empirical Study on Intermediate Pre-training for Cross-Modal Knowledge Transfer (2022, background): “Works that use such corpus present knowledge-related pre-training objectives such as concept order recovering (Zhou et al., 2021), entity category prediction (Yu et al.”
• Pre-Trained Language Models for Text Generation: A Survey (2022, methodology): “Recent research has found that integrating knowledge from external knowledge sources can enhance the text generation performance of PLMs [175, 225].”
• KB-VLP: Knowledge Based Vision and Language Pretraining (2021, background): “While there are works using knowledge during pretraining of language models (Yu et al., 2020; Xu et al., 2021; Ros-set et al., 2021; Zhou et al., 2020; He et al., 2020a; Xiong et al., 2019; He et al., 2020b; Agarwal et al., 2021), most have not been applied to multi-modal transformers (e.g. for…”
• Retrieval Enhanced Model for Commonsense Generation (2021, methodology): “Meanwhile, referring to (Zhou et al., 2021), we design a pre-training task for Common-Gen which continue to pre-train RE-T5 on pseudo concept sets extracted from external corpora.”