Chemformer: a pre-trained transformer for computational chemistry

By Ross Irwin, Spyridon Dimitriadis, Jiazhen He, Esben Jannik Bjerrum. 2021

468 citations recorded by the source corpus.

Transformer models coupled with a simplified molecular line entry system (SMILES) have recently proven to be a powerful combination for solving challenges in cheminformatics. These models, however, are often developed specifically for a single application and can be very resource-intensive to train. In this work we present the Chemformer model—a Transformer-based model which can be quickly applied to both sequence-to-sequence and discriminative cheminformatics tasks. Additionally, we show that self-supervised pre-training can improve performance and significantly speed up convergence on downstream tasks. On direct synthesis and retrosynthesis prediction benchmark datasets we publish state-of-the-art results for top-1 accuracy. We also improve on existing approaches for a molecular optimisation task and show that Chemformer can optimise on multiple discriminative tasks simultaneously. Models, datasets and code will be made available after publication.

Subfields: Computational Theory and Mathematics, Materials Chemistry, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Materials Chemistry

In Molecular Biology

Citations in the corpus

Translation between Molecules and Natural Language (2022, methodology): “For the molecular modality, we directly utilize the 100 million SMILES strings used in Chemformer (Irwin et al., 2021).”
The rise and potential of large language model based agents: a survey (2025, unknown): “Currently, numerous eﬀorts in various specialized domains aim to overcome this challenge [308–310].”
Autonomous chemical research with large language models (2023, unknown): “These models have been successfully applied to various domains, including natural language 1–5 , biological 6,7 and chemical research 8–10 as well as code generation 11,12 .”
Large Language Models on Graphs: A Comprehensive Survey (2023, methodology): “The model was pre-trained on two monolingual corpora: the Colossal Clean Crawled Corpus (C4) [29] for the natural language modality and one million SMILES [156] for the molecule modality.”
Augmenting large language models with chemistry tools (2023, background): “…field, has been impacted through expert-designed artificial intelligence (AI) systems that tackle specific problems, such as reaction prediction 16–20 , retrosynthesis planning 21–27 , molecular property prediction 28–32 , de-novo molecular generation 33,34 , materials design 35,36 and, more…”
Artificial Intelligence For Natural Product Drug Discovery (2023, background): “In many cases, pre-trained models will also yield higher prediction accuracies 140 .”