Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling

By Liyuan Liu, Xiang Ren, Jingbo Shang, Jian Peng, Jiawei Han. 2018

46 citations recorded by the source corpus.

Many efforts have been made to facilitate natural language processing tasks with pre-trained language models (LMs), and brought significant improvements to various applications. To fully leverage the nearly unlimited corpora and capture linguistic information of multifarious levels, large-size LMs are required; but for a specific task, only parts of these information are useful. Such large-sized LMs, even in the inference stage, may cause heavy computation workloads, making them too time-consuming for large-scale applications. Here we propose to compress bulky LMs while preserving useful information with regard to a specific task. As different layers of the model keep different information, we develop a layer selection method for model pruning using sparsity-inducing regularization. By introducing the dense connectivity, we can detach any layer without affecting others, and stretch shallow and wide LMs to be deep and narrow. In model training, LMs are learned with layer-wise dropouts for better robustness. Experiments on two benchmark datasets demonstrate the effectiveness of our method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Efficiency & Compression

Citations in the corpus

Reducing Transformer Depth on Demand with Structured Dropout (2019, methodology): “DistilBERT occasionally performs worse than BERT of the same size trained from scratch, which confirms the findings of Liu et al. (2018) about the performance of pruned models compared to 1The 48 layer model was trained for 300K updates. training small models from scratch.”
The Right Tool for the Job: Matching Model and Instance Complexities (2020, neutral): “Liu et al. (2018) and Fan et al. (2020) pruned deep models by applying dropout to different layers, which allows dynamic control of High conﬁdence by the model is sometimes explained by “easy” classes that are predicted with high F 1 (e.g., sports in AG).”
On the Variance of the Adaptive Learning Rate and Beyond (2019, methodology): “So we only need to show Using Gautschi’s inequality ( Γ( x +1) Γ( x + s ) ( x + 1) 1 − s ), we have Our implementation is based on the previous work (Liu et al., 2018).”
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance (2019, background): “…of multi-layer attentive neural architectures (Devlin et al., 2018; Peters et al., 2018), consolidating layer-wise information remains an open problem as different layers capture information at disparate scales and task-speciﬁc layer selection methods may be limited (Liu et al., 2018, 2019).”
Pre-trained models for natural language processing: A survey (2020, methodology): “[107] used layer-wise pruning and dense connection to speed up ELMo’s inference on NER.”
Structured Pruning of Large Language Models (2019, background): “Model weights are often removed via thresholding and l1 regularization during the pruning process (Narang et al., 2017b; Liu et al., 2018).”
A Survey on Deep Learning for Named Entity Recognition (2018, methodology): “Figure 10 shows the architecture of LM-LSTM-CRF model [124], [125].”