Analyzing the Forgetting Problem in the Pretrain-Finetuning of Dialogue Response Models

16 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Continual Learning for Natural Language Generation in Task-oriented Dialog Systems (2020, background): “To deal with this issue, He et al. (2019) proposed to replay pre-train data during ﬁne-tuning heavily, and Chen et al. (2020) proposed an improved Adam optimizer to recall knowledge captured during pre-training.”
The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents (2019, neutral): “He et al. (2019) recently obtained strong results on the DailyDialog and Cornell Movie tasks in terms of perplexity by pre-training on 10% of CC-NEWS (Bakhtin et al., 2019), thus using 100 million sentences (2.7 billion words) and then ﬁne-tuning a transformer based model with a multi-task strategy.”
A Literature Survey of Recent Advances in Chatbots (2021, background): “Although the data appears to be clean and less noisy compared to other datasets, it is also smaller, therefore less suited for more complex dialogue models [48,51,54,57] (No information was provided by the authors regarding the number of tokens in the dataset.”