UER: An Open-Source Toolkit for Pre-training Models

By Zhe Zhao, Hui Chen, Jinbin Zhang, Xin Zhao, Tao Liu, Wei Lu, Xi Chen, Haotang Deng, Qi Ju, Xiaoyong Du. 2019

145 citations recorded by the source corpus.

Existing works, including ELMO and BERT, have revealed the importance of pre-training for NLP tasks. While there does not exist a single pre-training model that works best in all cases, it is of necessity to develop a framework that is able to deploy various pre-training models efficiently. For this purpose, we propose an assemble-on-demand pre-training toolkit, namely Universal Encoder Representations (UER). UER is loosely coupled, and encapsulated with rich modules. By assembling modules on demand, users can either reproduce a state-of-the-art pre-training model or develop a pre-training model that remains unexplored. With UER, we have built a model zoo, which contains pre-trained models based on different corpora, encoders, and targets (objectives). With proper pre-trained models, we could achieve new state-of-the-art results on a range of downstream datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Natural Language Processing, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohit Bansal
2. Sebastian Gehrmann
3. Yoav Goldberg
4. Noah A. Smith
5. Graham Neubig
6. Percy Liang
7. Yejin Choi
8. Luke S. Zettlemoyer

In Computer Vision and Pattern Recognition

1. Eduard H. Hovy
2. Sebastian Gehrmann
3. Luc Van Gool
4. Sebastian Ruder
5. Mohit Bansal
6. Kyle Richardson
7. Vinay Uday Prabhu
8. Thomas Scialom

In Large Language Models

1. Yiming Cui
2. Jascha Narain Sohl-Dickstein
3. Caroline De Brun
4. Eduard H. Hovy
5. Paul-Alexis Dray
6. Haotang Deng
7. Junyi Li
8. Niklas Muennighoff

In Natural Language Processing

1. Sebastian Gehrmann
2. Wanxiang Che
3. Kyle Richardson
4. Thomas Scialom
5. Sebastian Ruder
6. Dian Yu
7. Kai Sun
8. Yiming Cui

In ML Systems & Hardware

1. Kaizhao Liang
2. Jascha Narain Sohl-Dickstein
3. Peng Zhou
4. Sebastian Ruder
5. Zhe Zhao

Citations in the corpus

• CLUE: A Chinese Language Understanding Evaluation Benchmark (2020, methodology): “Different from existing pre-training model toolkits (Wolf et al., 2019; Zhao et al., 2019), PyCLUE is designed with a goal of quick model performance validations on the CLUE benchmark.”
• FastBERT: a Self-distilling BERT with Adaptive Inference Time (2020, background): “UER (Zhao et al., 2019) pre-trains BERT in different Chinese corpora.”
• FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark (2021, methodology): “A smaller beam width (30) and Chinese T5 models provided by UER 7 (Zhao et al., 2019) are used to shorten the time for template geneartion and evaluation due to resource limit.”
• ET-BERT: A Contextualized Datagram Representation with Pre-training Transformers for Encrypted Traffic Classification (2022, methodology): “All the experiments are implemented with Pytorch 1.8.0 and UER [44], conducted with NVIDIA Tesla V100S GPUs.”
• NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation (2021, methodology): “CLUENER (Xu et al., 2020; Zhao et al., 2019) is used for tagging named entities in Chinese.”