SMILES-BERT: Large Scale Unsupervised Pre-Training for Molecular Property Prediction

By Sheng Wang, Yuzhi Guo, Yuhong Wang, Hongmao Sun, Junzhou Huang. ACM International Conference on Bioinformatics, Computational Biology and Biomedicine, 2019

540 citations recorded by the source corpus.

With the rapid progress of AI in both academia and industry, Deep Learning has been widely introduced into various areas in drug discovery to accelerate its pace and cut R&D costs. Among all the problems in drug discovery, molecular property prediction has been one of the most important problems. Unlike general Deep Learning applications, the scale of labeled data is limited in molecular property prediction. To better solve this problem, Deep Learning methods have started focusing on how to utilize tremendous unlabeled data to improve the prediction performance on small-scale labeled data. In this paper, we propose a semi-supervised model named SMILES-BERT, which consists of attention mechanism based Transformer Layer. A large-scale unlabeled data has been used to pre-train the model through a Masked SMILES Recovery task. Then the pre-trained model could easily be generalized into different molecular property prediction tasks via fine-tuning. In the experiments, the proposed SMILES-BERT outperforms the state-of-the-art methods on all three datasets, showing the effectiveness of our unsupervised pre-training and great generalization capability of the pre-trained model.

Subfields: Computational Theory and Mathematics, Materials Chemistry, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/3d99747cc3e13d22f21e02c35e82b57d2e351e2a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Materials Chemistry

In Molecular Biology

Citations in the corpus

Large Language Models on Graphs: A Comprehensive Survey (2023, unknown): “Numerous studies [152]–[154] have shown that training on different string-based views of the same molecule can improve the sequential model’s performance, as these data augmentation approaches manage to retain the permutation-invariance nature of graphs.”
MolGenSurvey: A Systematic Survey in Machine Learning Models for Molecule Design (2022, methodology): “For molecular string representation, existing work considers advanced representation methods from other domains, including convolutional neural network (CNN) [19], recurrent neural network (RNN) [20], [21], [22], and Transformer [23], [24], [25].”
Chemformer: a pre-trained transformer for computational chemistry (2021, unknown): “Other approaches make use of the encoder stack of the Transformer only, along with a fully-visible attention mask [9, 10, 12].”
ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction (2020, background): “In parallel, we aim to scale up pretraining, ﬁrst to the full PubChem 77M dataset, then to even larger sets like ZINC-15 (with 270 million compounds).”
Self-Supervised Graph Transformer on Large-Scale Molecular Data (2020, methodology): “By contending that molecule can be transformed into sequential representation— SMILES [59], the work by [58] tries to adopt the BERT-style method to pretrain the model, and Liu et.”
Uni-Mol: A Universal 3D Molecular Representation Learning Framework (2023, methodology): “In most current MRL methods, one starts with representing molecules as 1D sequential strings, such as SMILES [16, 17, 18] and InChI [19, 20, 21], or 2D graphs [22, 11, 23, 12, 24].”
ChEmbed: Enhancing Chemical Literature Search Through Domain-Specific Text Embeddings (2025, unknown): “Other methods represent chemical structures directly, using SMILES strings [21, 22, 23, 24] or molecular graphs [25, 26].”