MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification

By Jiaao Chen, Zichao Yang, Diyi Yang. 2020

415 citations recorded by the source corpus.

This paper presents MixText, a semi-supervised learning method for text classification, which uses our newly designed data augmentation method called TMix. TMix creates a large amount of augmented training samples by interpolating text in hidden space. Moreover, we leverage recent advances in data augmentation to guess low-entropy labels for unlabeled data, hence making them as easy to use as labeled data. By mixing labeled, unlabeled and augmented data, MixText significantly outperformed current pre-trained and fined-tuned models and other state-of-the-art semi-supervised learning methods on several text classification benchmarks. The improvement is especially prominent when supervision is extremely limited. We have publicly released our code at https://github.com/GT-SALT/MixText.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2020, methodology): “, 2020) and MixText (Chen et al., 2020), two state-of-the-art methods for semisupervised learning in NLP that rely on data augmentation.”
Improving and Simplifying Pattern Exploiting Training (2021, background): “However, performance can suffer when there is very limited labeled data available for a downstream task (Xie et al., 2020; Chen et al., 2020).”
Making Pre-trained Language Models Better Few-shot Learners (2021, background): “learning (Miyato et al., 2017; Xie et al., 2020; Chen et al., 2020), where a set of unlabeled examples are given; (2) meta-learning (Yu et al.”
Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, background): “More broadly, semi-supervised learning additionally leverages unlabeled task-specific data, where data augmentation [7, 73], regularization [43] and bootstrapping [56] methods are commonly used.”
Continual Sequence Generation with Adaptive Compositional Modules (2022, methodology): “Inspired by interpolation-based data augmentation (Chen et al., 2020, 2021) and neural architecture search (Liu et al., 2018), we utilize Hidden State Mixing for module selection.”
Continual Learning for Text Classification with Information Disentanglement Based Regularization (2021, methodology): “(Zhang et al., 2015; Chen et al., 2020) to evaluate our methods, including AG News (news clas-siﬁcation), Yelp (sentiment analysis), DBPedia (Wikipedia article classiﬁcation), Amazon (sentiment analysis), and Yahoo!”
Noisy Channel Language Model Prompting for Few-Shot Text Classification (2021, background): “Prior work in few-shot learning has used different approaches, including semi-supervised learning with data augmentation or consistency training (Miyato et al., 2017; Clark et al., 2018; Xie et al., 2020; Chen et al., 2020) and meta learning (Finn et al.”
A Survey on Data Augmentation for Text Classification (2021, positive): “The authors further demonstrate that their method is superior to other augmentation methods, such as EDA [2], round-trip-translation, and Tmix [97], and that the performance increases if larger models for the classification are used.”
CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding (2020, methodology): “…have successfully improved large-scale neural-network-based models, (Laine & Aila, 2017; Xie et al., 2019; Berthelot et al., 2019; Sohn et al., 2020; He et al., 2020; Khosla et al., 2020; Chen et al., 2020b), however, the majority of existing research is geared towards computer vision tasks.”
GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation (2021, methodology): “One of the notable methods is MixText (Chen et al., 2020), in which BERT is used to generate novel augmentation samples from interpolated embedding spaces.”
An Empirical Survey of Data Augmentation for Limited Data Learning in NLP (2021, methodology): “, 2020b), or between different training examples in general hidden space (Chen et al., 2020c).”
Deep Learning--based Text Classification (2020, methodology): “In general, data augmentation using VAE or other models [164, 165] is widely used for semi-supervised or weakly supervised text classification.”
SeqMix: Augmenting Active Sequence Labeling via Sequence Mixup (2020, methodology): “Guo et al. (2019a); Chen et al. (2020a) apply hidden-space Mixup for text classiﬁcation.”
True Few-Shot Learning with Prompts—A Real-World Perspective (2021, background): “…the considered settings are often not true few-shot settings (Perez et al., 2021; Logan IV et al., 2021) mainly for two reasons: For one, some approaches (e.g., Xie et al., 2019; Zhang et al., 2020; Chen et al., 2020; Tam et al., 2021) make use of large development sets to optimize hyperparameters.”
A Survey on Deep Semi-Supervised Learning (2021, background): “Text classification AG News [95], BPpedia [96], Yahoo! Answers [97], IMDB [98], Yelp review [95], Snippets [99], Ohsumed [100], TagMyNews [101], MR [102], Elec [103], Rotten Tomatoes [102], RCV1 [104] [105], [106], [107], [108], [109], [110]”
Data-centric Artificial Intelligence: A Survey (2023, neutral): “Beyond image data, basic manipulation often needs to be tailored for the other data types, such as permutation and jittering in time-series data [238], mixing data in the hidden space for text data to retain semantic meanings [40], and mixing graphon for graph data [88].”
Better Robustness by More Coverage: Adversarial Training with Mixup Augmentation for Robust Fine-tuning (2020, methodology): “Recently, several works have attempted to apply mixup on language representations to improve performance on downstream NLP tasks (Sun et al., 2020; Chen et al., 2020; Cheng et al., 2020), on knowledge distillation (Liang et al., 2020) and on compositional generalization benchmarks (Guo et al.,…”
Few-Shot Text Generation with Pattern-Exploiting Training (2020, methodology): “For optimizing hyperparameters, much previous work uses development sets that are larger than the training sets by multiple orders of magnitude (e.g., Xie et al., 2019; Zhang et al., 2020; Chen et al., 2020); however, assuming the existence of such large development sets is highly inconsistent with ”