Low Resource Multi-Task Sequence Tagging - Revisiting Dynamic Conditional Random Fields

By Jonas Pfeiffer, Edwin Simpson, Iryna Gurevych. 2020

5 citations recorded by the source corpus.

We compare different models for low resource multi-task sequence tagging that leverage dependencies between label sequences for different tasks. Our analysis is aimed at datasets where each example has labels for multiple tasks. Current approaches use either a separate model for each task or standard multi-task learning to learn shared feature representations. However, these approaches ignore correlations between label sequences, which can provide important information in settings with small training datasets. To analyze which scenarios can profit from modeling dependencies between labels in different tasks, we revisit dynamic conditional random fields (CRFs) and combine them with deep neural networks. We compare single-task, multi-task and dynamic CRF setups for three diverse datasets at both sentence and document levels in English and German low resource scenarios. We show that including silver labels from pretrained part-of-speech taggers as auxiliary tasks can improve performance on downstream tasks. We find that especially in low-resource scenarios, the explicit modeling of inter-dependencies between task predictions outperforms single-task as well as standard multi-task models.

Subfields: Artificial Intelligence, Natural Language Processing, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Iryna Gurevych
2. Jonas Pfeiffer
3. Andreas Ruckl'e
4. Ivan Vulic
5. Sebastian Ruder
6. Aishwarya Kamath
7. Kyunghyun Cho
8. Percy Liang

In Natural Language Processing

1. Iryna Gurevych
2. Jonas Pfeiffer
3. Andreas Ruckl'e
4. Aishwarya Kamath
5. Kyunghyun Cho
6. Ivan Vulic
7. Sebastian Ruder
8. Mohit Bansal

In Optimization & Training

1. Jonas Pfeiffer
2. Iryna Gurevych
3. Kyunghyun Cho
4. Andreas Ruckl'e
5. Aishwarya Kamath
6. Yoshua Bengio
7. Tapani Raiko
8. Asja Fischer

Citations in the corpus

• AdapterFusion: Non-Destructive Task Composition for Transfer Learning (2020, methodology): “This makes it difficult to effectively transfer knowledge across tasks with all the tasks being solved equally well (Pfeiffer et al., 2020a), thus considerably limiting the applicability of multi-task learning in many scenarios.”