CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters

By Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, Junichi Tsujii. 2020

178 citations recorded by the source corpus.

Due to the compelling improvements brought by BERT, many recent representation models adopted the Transformer architecture as their main building block, consequently inheriting the wordpiece tokenization system despite it not being intrinsically linked to the notion of Transformers. While this system is thought to achieve a good balance between the flexibility of characters and the efficiency of full words, using predefined wordpiece vocabularies from the general domain is not always suitable, especially when building models for specialized domains (e.g., the medical domain). Moreover, adopting a wordpiece tokenization shifts the focus from the word level to the subword level, making the models conceptually more complex and arguably less convenient in practice. For these reasons, we propose CharacterBERT, a new variant of BERT that drops the wordpiece system altogether and uses a Character-CNN module instead to represent entire words by consulting their characters. We show that this new model improves the performance of BERT on a variety of medical domain tasks while at the same time producing robust, word-level, and open-vocabulary representations.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yi Tay
2. Tao Qin
3. Luke S. Zettlemoyer
4. Xu Tan
5. Donald Metzler
6. Tie-Yan Liu
7. Sebastian Ruder
8. Mohit Bansal

In Natural Language Processing

1. Yi Tay
2. Sebastian Ruder
3. Donald Metzler
4. Colin Raffel
5. Graham Neubig
6. Hinrich Schütze
7. Noah Constant
8. Dara Bahri

Citations in the corpus

• ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2021, methodology): “These approaches include ELMo (Peters et al., 2018), CharacterBERT (El Boukkouri et al., 2020) and many others (Ling et al., 2015; Chung et al., 2016; Kim et al., 2016; Józefowicz et al., 2016; Wang et al., 2020; Wei et al., 2021).”
• Charformer: Fast Character Transformers via Gradient-based Subword Tokenization (2021, methodology): “Such character-aware representations have lately been applied to deep Transformer models (El Boukkouri et al., 2020; Ma et al., 2020).”
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “For example in CharacterBERT [66], finegrained word representation is generated from character embeddings using a character encoder based on CharCNN and highway layer [122].”
• Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation (2021, methodology): “Boukkouri et al. (2020) showed that B ERT ’s Wikipedia+BooksCorpus WordPiece vocabulary results in excessive segmentation when fine-tuning on medical data, diminishing the benefit of pre-training as a strategy.”
• Mixed-Phoneme BERT: Improving BERT with Mixed Phoneme and Sup-Phoneme Representations for Text to Speech (2022, background): “, character [20], subword [18,21]), which enables the TTS system to generate speech with better pronunciation and expressiveness.”