Pre-Training with Whole Word Masking for Chinese BERT

By Yiming Cui, Wanxiang Che, Ting-Wei Liu, Bing Qin, Ziqing Yang, Shijin Wang, Guoping Hu. 2019

1,316 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

Citations in the corpus

Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction (2022, methodology): “We next adopt a pre-trained language model such as BERT (Devlin et al., 2019) or Roberta (Cui et al., 2019; Liu et al., 2019) to obtain the quantity representation q for each quantity q .”
CLUE: A Chinese Language Understanding Evaluation Benchmark (2020, methodology): “• RoBERTa-large uses a 24 layer Transformer (Vaswani et al., 2017) • RoBERTa-wwm-ext-large (Cui et al., 2019a) has a network structure of RoBERTa-large and training procedure of RoBERTa-wwm-ext-base.”
A Primer in BERTology: What We Know About How BERT Works (2020, background): “Masks can be applied to full words instead of word-pieces (Devlin et al., 2019; Cui et al., 2019).”
Pre-Trained Models: Past, Present and Future (2021, neutral): “Similar ideas have also been explored in ERNIE (Sun et al., 2019c,d) (where a whole entity is masked), NEZHA (Wei et al., 2019), and Whole Word Masking (Cui et al., 2019).”
FastBERT: a Self-distilling BERT with Adaptive Inference Time (2020, methodology): “Any pre-training method used for BERT-like models (e.g., BERT-WWM (Cui et al., 2019), RoBERTa (Liu et al., 2019), and ERNIE (Sun et al., 2019b)) can be directly applied.”
Pre-trained models for natural language processing: A survey (2020, background): “Language-Specific ERNIE(Baidu) [170], BERT-wwm-Chinese [29], NEZHA [198], ZEN [37] BERTje [33], CamemBERT [125], FlauBERT [95], RobBERT [35]”
K-BERT: Enabling Language Representation with Knowledge Graph (2019, methodology): “In optimizing pre-training process, Baidu-ERNIE (Sun et al. 2019) and BERT-WWM (Cui et al. 2019) adopt whole-word masking rather than single character masking for pre-training BERT in Chinese corpora.”
CPM: A Large-scale Generative Chinese Pre-trained Language Model (2020, methodology): “Although there are some previous works providing powerful Chinese pre-trained language models (Cui et al., 2020; Xu et al., 2020; Wei et al., 2019; Sun et al., 2019; Cui et al., 2019a), their capabilities are limited due to the model size.”
CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Model (2020, methodology): “Our model RoBERTalarge-clue trained with 100g achieve same performance as RoBERTa-wwm-large’ (Cui et al., 2019), slightly better performance in two tasks, IFLYTEK and CMNLI.”
MPNet: Masked and Permuted Pre-training for Language Understanding (2020, methodology): “Additionally, we also apply whole word mask [12] and relative positional embedding [13]6 in our model pre-training since these tricks have been successfully validated in previous works [5, 14].”
ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation (2021, neutral): “0[33], ERNIE-Gram[79], SKEP[80], RoBERTa-wwm-ext-large[81] (marked as RoBERTa*), ALBERT[82], MacBERT[83], Zen 2.”
OCNLI: Original Chinese Natural Language Inference (2020, methodology): “Additionally, we establish baseline results based on a standard set of NLI models (Chen et al., 2017) tailored to Chinese, as well as new pre-trained Chinese transformer models (Cui et al., 2019).”
Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca (2023, methodology): “For the other Chinese LLaMA models (basic version), we utilize a 20GB general Chinese corpus for pre-training, which is consistent with the corpora used by Chinese BERT-wwm (Cui et al., 2021), MacBERT (Cui et al.”
g2pM: A Neural Grapheme-to-Phoneme Conversion Package for MandarinChinese Based on a New Open Benchmark Dataset (2020, methodology): “Finally, we test the pretrained Chinese BERT model [17].”
IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding (2020, neutral): “Cui et al. (2019)”
PhoBERT: Pre-trained language models for Vietnamese (2020, methodology): “For other languages, one could retrain a language-specific model using the BERT architecture (Cui et al., 2019; de Vries et al., 2019; Vu et al., 2019; Martin et al., 2020) or employ existing pre-trained multilingual BERT-based models (Devlin et al.”
FLAT: Chinese NER Using Flat-Lattice Transformer (2020, methodology): “The BERT in the experiment is ‘BERT-wwm’ released by Cui et al. (2019).”
What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams (2020, background): “L ARGE - WWM - EXT ) (Cui et al. 2019).”
Generating Adversarial Examples in Chinese Texts Using Sentence-Pieces (2020, background): “In Chinese texts generation, a whole-word-mask strategy is introduced to predict the entire word properly (Cui et al., 2019), while the tokenization is still character-level.”