A Robustly Optimized BERT Pre-training Approach with Post-training

By Zhuang Liu, Wayne Lin, Yafei Shi, Jun Zhao. China National Conference on Chinese Computational Linguistics, 2021

729 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Emergent Structures and Training Dynamics in Large Language Models (2022, background): “RoBERTa (Zhuang et al., 2021)) and LSTMs (Hochreiter and Schmidhuber, 1997) agree that models acquire linguistic knowledge early in the learning process.”
A Comparative Analysis of Pretrained Language Models for Text-to-Speech (2023, methodology): “Regarding transformer encoders for NLU, since the emergence of BERT, many BERT-like models have been released with more pretraining data [4, 5], novel pretraining methods [6], more efficient architectures [7], or smaller architectures with faster inference [8, 9, 10, 11].”
Contextual Slate GLM Bandits with Limited Adaptivity (2026, unknown): “Experimental Design : All experiments are conducted using RoBERTa-large [LLSZ21] as the base model and Nomic-Embed-Text-v1.”
Efficient Algorithms for Logistic Contextual Slate Bandits with Bandit Feedback (2025, unknown): “After selecting the 4 items (slate), the resulting prompt (also containing the input query q ) is passed through the RoBERTa [Zhuang et al., 2021] model and a possible answer for q is generated.”
VIBE: Vector Index Benchmark for Embeddings (2025, unknown): “Our text embedding models cover both the relatively older but widely applied models DistilRoBERTa (Sanh et al., 2019; Zhuang et al., 2021) and MiniLM (Wang et al., 2020), as well as the recent top-performing industry models nomic-embed-text-v1.”