Total Recall: a Customized Continual Learning Method for Neural Semantic Parsers

By Zhuang Li, Lizhen Qu, Gholamreza Haffari. 2021

16 citations recorded by the source corpus.

This paper investigates continual learning for semantic parsing. In this setting, a neural semantic parser learns tasks sequentially without accessing full training data from previous tasks. Direct application of the SOTA continual learning algorithms to this problem fails to achieve comparable performance with re-training models with all seen tasks because they have not considered the special properties of structured outputs yielded by semantic parsers. Therefore, we propose TotalRecall, a continual learning method designed for neural semantic parsers from two aspects: i) a sampling method for memory replay that diversifies logical form templates and balances distributions of parse actions in a memory; ii) a two-stage training method that significantly improves generalization capability of the parsers across tasks. We conduct extensive experiments to study the research problems involved in continual semantic parsing and demonstrate that a neural semantic parser trained with TotalRecall achieves superior performance than the one trained directly with the SOTA continual learning algorithms and achieve a 3-6 times speedup compared to re-training from scratch.

Subfields: Artificial Intelligence, Information Extraction, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hang Su
2. Jun Zhu
3. Liyuan Wang
4. Xingxing Zhang
5. Zhuang Li
6. Yuan-Fang Li
7. Guilin Qi
8. Massimo Caccia

In Information Extraction

1. Jun Zhu
2. Gholamreza Haffari

In Meta & Continual Learning

1. Xingxing Zhang
2. Liyuan Wang
3. Hang Su
4. Jun Zhu
5. Guilin Qi
6. Massimo Caccia
7. Gholamreza Haffari
8. Simone Scardapane

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “MBPA++ [442], EMAR [465], DnR [396], ARPER [298], Total Recall [253]), generative replay (PAGeR [415], LAMOL [395], ACM [499], NER [434]), parameter allocation (TPEM [131]), modular network (ProgModel [377]), meta-learning (MetaMBPA++ [442], MeLL [422], CML [455]), etc.”
• Pretrained Language Model in Continual Learning: A Comparative Study (2022, background): “CL in NLP is still a nascent topic, as reflected by the relatively smaller number of proposed methods (Biesialska et al., 2020; Li et al., 2021).”