Universal Natural Language Processing with Limited Annotations: Try Few-shot Textual Entailment as a Start

By Wenpeng Yin, Nazneen Rajani, Dragomir R. Radev, Richard Socher, Caiming Xiong. 2020

70 citations recorded by the source corpus.

A standard way to address different NLP problems is by first constructing a problem-specific dataset, then building a model to fit this dataset. To build the ultimate artificial intelligence, we desire a single machine that can handle diverse new problems, for which task-specific annotations are limited. We bring up textual entailment as a unified solver for such NLP problems. However, current research of textual entailment has not spilled much ink on the following questions: (i) How well does a pretrained textual entailment system generalize across domains with only a handful of domain-specific examples? and (ii) When is it worth transforming an NLP task into textual entailment? We argue that the transforming is unnecessary if we can obtain rich annotations for this task. Textual entailment really matters particularly when the target NLP task has insufficient annotations. Universal NLP can be probably achieved through different routines. In this work, we introduce Universal Few-shot textual Entailment (UFO-Entail). We demonstrate that this framework enables a pretrained entailment model to work well on new entailment domains in a few-shot setting, and show its effectiveness as a unified solver for several downstream NLP tasks such as question answering and coreference resolution when the end-task annotations are limited. Code: this https URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Making Pre-trained Language Models Better Few-shot Learners (2021, background): “termediate training (Phang et al., 2018; Yin et al., 2020), where a related, intermediate task is given.”
Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, background): “The current mainstream of zero-shot learning is based on transfer learning: By converting a set of tasks with abundant annotations into instruction templates [42, 54, 70, 74], entailment pairs [79, 80] or question-answer formats [50, 86] and fine-tuning PLMs on them, the PLMs acquire the cross-task ”
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, methodology): “…2019; Liu et al., 2020; Hosseini-Asl et al., 2020; Liu et al., 2021), which has been augmented by positional encoding (e.g., row/column embedding) (Herzig et al., 2020; Yin et al., 2020a) and template-based linearization (Chen et al., 2020a,b; Oguz et al., 2021), and planning (Su et al., 2021).”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “UFO-Entail (Yin et al., 2020) formulates multiplechoice QA and co-reference resolution as textual entailment tasks and examines the performance in few-shot settings.”
Incremental Few-shot Text Classification with Multi-round New Classes: Formulation, Dataset and System (2021, critical): “All previous few-shot learning models (Snell et al., 2017; Gidaris and Komodakis, 2018; Yin et al., 2020; Xia et al., 2020b; Nguyen et al., 2020) fail to solve this problem since they relied on the large-scale labeled data of base classes to train a robust system.”
Continual Few-Shot Learning for Text Classification (2021, methodology): “Recent work has shown the generalization capability of large pre-trained models to handle multiple tasks with zero to few training examples (Schick and Schütze, 2021; Brown et al., 2020; Yin et al., 2020).”
Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (2021, methodology): “Following prior evidence of positive transfer from NLI to other tasks (Conneau and Kiela, 2018; Yin et al., 2020; Phang et al., 2018), we initialize the out-of-domain TREC from MNLI.”
Improving In-Context Few-Shot Learning via Self-Supervised Training (2022, background): “…on unifying task formats to maximize the benefits of human annotations, including question answering (Zhong et al., 2021), textual entailment (Yin et al., 2019, 2020; Wang et al., 2021a), and many other tasks (McCann et al., 2018; Keskar et al., 2019; Raffel et al., 2020; Bragg et al., 2021).”
Can NLI Models Verify QA Systems' Predictions? (2021, methodology): “…entailment can be used to enhance the accuracy of the open-domain QA systems; Trivedi et al. (2019) used a pretrained NLI model to select relevant sentences for multi-hop question answering; Yin et al. (2020) tested whether NLI models generalize to QA setting in a few-shot learning scenario.”
Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners (2021, background): “This issue can be addressed by intermediate training (Phang et al., 2018; Yin et al., 2020; Zhao et al., 2021), and will be analyzed in the future work.”
Zero-shot Learning by Generating Task-specific Adapters (2021, background): “, 2020), coreference resolution as textual entailment (Yin et al., 2020).”
Discovering Latent Knowledge in Language Models Without Supervision (2022, methodology): “…models (e.g. RoBERTa and DeBERTa) cannot be easily used to do zero-shot classification out of the box, so to evaluate them we follow the method of Yin et al. (2020): we finetune both models on an NLI dataset (MNLI, which we do not evaluate on) and treat the difference between the entailment and…”