Small and Practical BERT Models for Sequence Labeling

By Henry Tsai, Jason Riesa, Melvin Johnson, Naveen Arivazhagan, Xin Li, Amelia Archer. 2019

128 citations recorded by the source corpus.

We propose a practical scheme to train a single multilingual sequence labeling model that yields state of the art results and is small and fast enough to run on a single CPU. Starting from a public multilingual BERT checkpoint, our final model is 6x smaller and 27x faster, and has higher accuracy than a state-of-the-art multilingual baseline. We show that our model especially outperforms on low-resource languages, and works on codemixed input text without being explicitly trained on codemixed examples. We showcase the effectiveness of our method by reporting on part-of-speech tagging and morphological prediction on 70 treebanks and 48 languages.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (2019, background): “An application of multi-distillation is multi-linguality: Tsai et al. [2019] adopts a similar approach to us by pre-training a multilingual model from scratch solely through distillation.”
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (2020, methodology): “Different from task-speciﬁc distillation, task-agnostic LM distillation mimics the behavior of the original pre-trained LMs and the student model can be directly ﬁne-tuned on downstream tasks (Tsai et al., 2019; Sanh et al., 2019; Jiao et al., 2019; Sun et al., 2019b).”
A Primer in BERTology: What We Know About How BERT Works (2020, background): “53 BERT6 No WNLI, CoLA and STS-B MiniBERT(Tsai et al., 2019) ×6§ 98% ×27§ mBERT3 CoNLL-2018 POS and morphology BiLSTM soft (Tang et al.”
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices (2020, methodology): “There have been some efforts that taskspecifically distill BERT into compact models (Turc et al., 2019; Tang et al., 2019; Sun et al., 2019; Tsai et al., 2019).”
Pre-trained models for natural language processing: A survey (2020, methodology): “Distillation from soft target probabilities can also be used in task-specific models, such as information retrieval [121], and sequence labeling [181].”
Intriguing Properties of Compression on Multilingual Models (2022, background): “…by scale, efﬁciency techniques that reduce model size while maintaining comparable aggregate performance are widely used, such as quantization (Shen et al., 2020), compression (Michel et al., 2019; La-gunas et al., 2021) and distillation (Tsai et al., 2019; Sanh et al., 2019; Pu et al., 2021).”
Probing Pretrained Language Models for Lexical Semantics (2020, background): “‘recovered’ from the models); this also suggests why pretrained LMs have been successful in tasks where this knowledge is directly useful, such as NER and POS tagging (Tenney et al., 2019; Tsai et al., 2019).”
Frequency Effects on Syntactic Rule Learning in Transformers (2021, methodology): “To propose candidate nouns, we ﬁrst ran a POS tag-ger (Tsai et al., 2019) over the pre-training dataset, and retrieved all nouns occurring at least 100 times.”
Structure-Level Knowledge Distillation For Multilingual Sequence Labeling (2020, methodology): “(Tsai et al., 2019) distilled a large M-BERT model into a three layer M-BERT model for sequence labeling and achieved a competitively high accuracy with signiﬁcant speed improvements.”
Supplementary Material: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers (2020, unknown): “MiniBERT [37] uses the soft target distributions for masked language modeling predictions to guide the training of the multilingual student model and shows its effectiveness on sequence labeling tasks.”
GoEmotions: A Dataset of Fine-Grained Emotions (2020, methodology): “We mask proper names referring to people with a [NAME] token, using a BERT-based Named Entity Tagger (Tsai et al., 2019).”
Learning Helpful Inductive Biases from Self-Supervised Pretraining (2020, background): “The other configurations represent a scale roughly based on settings used in Sanh et al. (2019), Vaswani et al. (2017), Jiao et al. (2019), and Tsai et al. (2019).”
LRC-BERT: Latent-representation Contrastive Knowledge Distillation for Natural Language Understanding (2020, methodology): “MiniBERT (Tsai et al. 2019) uses soft target distribution for mask language modeling prediction, guides the training of multilingual student models.”
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers (2020, methodology): “In this work, we focus on task-agnostic compression of pretrained Transform-ers (Sanh et al., 2019; Tsai et al., 2019; Jiao et al., 2019; Sun et al., 2019b; Wang et al., 2020).”
A Practical Survey on Faster and Lighter Transformers (2021, unknown): “[40], and Jiao et al.”
XtremeDistil: Multi-stage Distillation for Massive Multilingual Models (2020, background): “There has been some limited exploration of such techniques for sequence tagging tasks like NER (Izsak et al., 2019; Shi et al., 2019) or multilingual tasks (Tsai et al., 2019).”
Structured Pruning of Large Language Models (2019, methodology): “, 2019), and compressing BERT models via distillation (Chia et al., 2019; Jiao et al., 2019; Sanh et al., 2019; Sun et al., 2019; Tsai et al., 2019; Turc et al., 2019).”