German’s Next Language Model
By Branden Chan, Stefan Schweter, Timo Möller. International Conference on Computational Linguistics, 2020
346 citations recorded by the source corpus.
In this work we present the experiments which lead to the creation of our BERT and ELECTRA based German language models, GBERT and GELECTRA. By varying the input training data, model size, and the presence of Whole Word Masking (WWM) we were able to attain SoTA performance across a set of document classification and named entity recognition (NER) tasks for both models of base and large size. We adopt an evaluation driven approach in training these models and our results indicate that both adding more data and utilizing WWM improve model performance. By benchmarking against existing German models, we show that these models are the best German models to date. All trained models will be made publicly available to the research community.
Subfields: Artificial Intelligence
View Paper: https://www.semanticscholar.org/paper/61f42619eca0afaf2cff9a899bbdea88182885f8
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Julia Kreutzer
- Orhan Firat
- Ankur Bapna
- Artem Sokolov
- Isaac Caswell
- Sneha Kudugunta
- Toan Q. Nguyen
- Graham Neubig
Citations in the corpus
- Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2021, methodology): “OSCAR has also been used in independent studies to train monolingual or multilingual language models (ar, as, bn, de, el, fr, gu, he, hi, kn, ml, mr, nl, or, pa, ro, ta, te) and subsequently evaluate them on various downstream tasks (Antoun et al., 2021; Kakwani et al., 2020; Wilie et al., 2020; Cha”
- Contrastive Learning Using Graph Embeddings for Domain Adaptation of Language Models in the Process Industry (2025, unknown): “Model selection For LM fine-tuning, we selected several pre-trained and domain-adapted mono-lingual or multilingual pre-trained LMs. Specifically, we selected state-of-the-art LMs with BERT architectures that support the German language: GBERT-base (Chan et al., 2020), multilingual mBERT-base (Devli”