Story Ending Prediction by Transferable BERT

By Zhongyang Li, Xiao Ding, Ting-Wei Liu. 2019

54 citations recorded by the source corpus.

Recent advances, such as GPT and BERT, have shown success in incorporating a pre-trained transformer language model and fine-tuning operation to improve downstream NLP systems. However, this framework still has some fundamental problems in effectively incorporating supervised knowledge from other related tasks. In this study, we investigate a transferable BERT (TransBERT) training framework, which can transfer not only general language knowledge from large-scale unlabeled data but also specific kinds of knowledge from various semantically related supervised tasks, for a target task. Particularly, we propose utilizing three kinds of transfer tasks, including natural language inference, sentiment classification, and next action prediction, to further train BERT based on a pre-trained model. This enables the model to get a better initialization for the target task. We take story ending prediction as the target task to conduct experiments. The final result, an accuracy of 91.8%, dramatically outperforms previous state-of-the-art baseline methods. Several comparative experiments give some helpful suggestions on how to select transfer tasks to improve BERT.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xipeng Qiu
2. Xuanjing Huang
3. Minlie Huang
4. Xiaoyan Zhu
5. Yunfan Shao
6. Chenhui Chu
7. Tianxiang Sun
8. Yige Xu

In Computer Vision and Pattern Recognition

1. Minlie Huang
2. Xiaoyan Zhu
3. Xipeng Qiu
4. Ning Dai
5. Tianxiang Sun
6. Zekun Yang
7. Haruo Takemura
8. Yuta Nakashima

In Natural Language Processing

1. Xipeng Qiu
2. Xuanjing Huang
3. Chenhui Chu
4. Yunfan Shao
5. Minlie Huang
6. Yige Xu
7. Tianxiang Sun
8. Ning Dai

Citations in the corpus

• Pre-trained models for natural language processing: A survey (2020, methodology): “[103] also used a two-stage transfer for the story ending prediction.”
• BERT Representations for Video Question Answering (2020, background): “BERT is a powerful bidirectional network based on language Trans-formers [39] and it has been shown to outperform LSTMs in several natural language processing tasks [7, 9, 24].”
• SentiLARE: Linguistic Knowledge Enhanced Language Representation for Sentiment Analysis (2019, methodology): “Task-specific Pre-trained Models : We used BERT-PT (Xu et al., 2019), TransBERT (Li et al., 2019), and SentiBERT (Yin et al., 2020) as task-specific pre-trained baselines.”