ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction

By Seyone Chithrananda, Gabriel Grand, Bharath Ramsundar. arXiv.org, 2020

752 citations recorded by the source corpus.

GNNs and chemical fingerprints are the predominant approaches to representing molecules for property prediction. However, in NLP, transformers have become the de-facto standard for representation learning thanks to their strong downstream task transfer. In parallel, the software ecosystem around transformers is maturing rapidly, with libraries like HuggingFace and BertViz enabling streamlined training and introspection. In this work, we make one of the first attempts to systematically evaluate transformers on molecular property prediction tasks via our ChemBERTa model. ChemBERTa scales well with pretraining dataset size, offering competitive downstream performance on MoleculeNet and useful attention-based visualization modalities. Our results suggest that transformers offer a promising avenue of future work for molecular representation learning and property prediction. To facilitate these efforts, we release a curated dataset of 77M SMILES from PubChem suitable for large-scale self-supervised pretraining.

Subfields: Computational Theory and Mathematics, Materials Chemistry, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/95ce6f77e26b496ffb705a0a3b54f2fb7a6d2452

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Materials Chemistry

In Molecular Biology

Citations in the corpus

MolGenSurvey: A Systematic Survey in Machine Learning Models for Molecule Design (2022, methodology): “For molecular string representation, existing work considers advanced representation methods from other domains, including convolutional neural network (CNN) [19], recurrent neural network (RNN) [20], [21], [22], and Transformer [23], [24], [25].”
Chemformer: a pre-trained transformer for computational chemistry (2021, positive): “Other approaches make use of the encoder stack of the Transformer only, along with a fully-visible attention mask [9, 10, 12].”
Transformer-based artificial neural networks for the conversion between chemical notations (2021, methodology): “Transformer has been successfully applied to chemical-related problem: prediction of outcomes of organic reactions 14 , QSAR modelling 15 and the creation of molecular representations 16 .”
ChemBERTa-2: Towards Chemical Foundation Models (2022, methodology): “ChemBERTa [2] introduced a BERT-like transformer model that learns molecular fingerprints through semi-supervised pretraining and pretrained it on a dataset of 10M compounds.”
Translation between Molecules and Natural Language (2022, methodology): “Other advances such as BERT (Devlin et al., 2019) have also been applied to the domain, such as MolBERT (Fabian et al., 2020) and ChemBERTa (Chithrananda et al., 2020), which use SMILES strings as inputs to pre-train a BERT-esque model.”
Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generation (2025, unknown): “The evaluated domain-pretrained models include SciBERT [35], PubMedBERT [40], Biomed-BERT [60], ChemicalBERT [42], ChemBERTa [43], and MatSciBERT [39].”
ChEmbed: Enhancing Chemical Literature Search Through Domain-Specific Text Embeddings (2025, unknown): “Other methods represent chemical structures directly, using SMILES strings [21, 22, 23, 24] or molecular graphs [25, 26].”