Optimization of Molecules via Deep Reinforcement Learning

By Zhenpeng Zhou, Steven M. Kearnes, Li Li, Richard N. Zare, Patrick F. Riley. Scientific Reports, 2018

665 citations recorded by the source corpus.

We present a framework, which we call Molecule Deep Q-Networks (MolDQN), for molecule optimization by combining domain knowledge of chemistry and state-of-the-art reinforcement learning techniques (double Q-learning and randomized value functions). We directly define modifications on molecules, thereby ensuring 100% chemical validity. Further, we operate without pre-training on any dataset to avoid possible bias from the choice of that set. MolDQN achieves comparable or better performance against several other recently published algorithms for benchmark molecular optimization tasks. However, we also argue that many of these tasks are not representative of real optimization problems in drug discovery. Inspired by problems faced during medicinal chemistry lead optimization, we extend our model with multi-objective reinforcement learning, which maximizes drug-likeness while maintaining similarity to the original molecule. We further show the path through chemical space to achieve optimization for a molecule to understand how the model works.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Materials Chemistry

View Paper: https://www.semanticscholar.org/paper/2ba00083a72558d45e4baecff0425ef6272f7a40

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Materials Chemistry

Citations in the corpus

Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development (2021, methodology): “We compare domain SOTA methods including Screening (Lyu et al. 2019) (simulated as random sampling), Graph-GA (graph-based genetic algorithm) (Jensen 2019), and ML SOTA methods including stringbased LSTM (Segler et al. 2018), GCPN (Graph Convolutional Policy Network) (You et al. 2018), MolDQN (Deep ”
MolGenSurvey: A Systematic Survey in Machine Learning Models for Molecule Design (2022, methodology): “In contrast, combinatorial optimization methods (COMs) have less requirement on the amount of training data, while the trade-off is that algorithm designers need to call the oracles during the exploration to the chemical space [103], [104], [105], [106], [107].”
3D Equivariant Diffusion for Target-Aware Molecule Generation and Affinity Prediction (2023, methodology): “Thus, many graph-based methods are proposed to operate directly on graphs (Liu et al., 2018; Shi et al., 2020; Jin et al., 2018; 2020; You et al., 2018; Zhou et al., 2019).”
Deep Learning for Deep Chemistry: Optimizing the Prediction of Chemical Patterns (2019, background): “…of target molecules and guided exploration of chemical space (Kowalik et al., 2012; Houben and Lapkin, 2015; Houben et al., 2015; Zielinski et al., 2017; Häse et al., 2018; Min et al., 2018; Zhou et al., 2018; Ahn et al., 2019; Choi et al., 2019; Gromski et al., 2019; Matsuzaka and Uesawa, 2019).”
Differentiable Scaffolding Tree for Molecular Optimization (2021, neutral): “Combinatorial optimization methods mainly include deep reinforcement learning (DRL) [49, 52, 23, 15] and evolutionary learning methods [36, 21, 47, 11].”
Model-based reinforcement learning for biological sequence design (2020, background): “, 2016), small molecules (Zhou et al., 2019), and RNA sequences that fold into a particular structure (Runge et al.”
All SMILES Variational Autoencoder (2019, methodology): “Following prior work, we optimize QED and logP penalized by the synthetic accessibility score and the number of large rings [9, 27, 35, 56, 70, 72].”
Machine Learning for Synthetic Data Generation: a Review (2023, neutral): “…molecule distribution, including variational autoencoder (VAE) [21], [76], [77], generative adversarial network (GAN) [78], energy-based model (EBM) [79], [80], diffusion model [81], reinforcement learning (RL) [22], [24], [82], genetic algorithm [83], sampling-based methods [84], [85], etc.”
BERT Learns (and Teaches) Chemistry (2020, background): “One slightly related work creates molecule Deep Q-Networks (MolDQN), for molecule optimization by combining domain knowledge of chemistry, and reinforcement learning techniques [36].”
Geometric deep learning on molecular representations (2021, methodology): “GNNs have also been combined with variational autoencoders [48, 88–90] and reinforcement learning [47, 91, 92].”
Sample-Efficient Optimization in the Latent Space of Deep Generative Models via Weighted Retraining (2020, methodology): “This is identical to the scoring function from references [29, 5, 21, 65, 64], except that we bound the score below by − 6 to prevent points with highly-negative scores from substantially impacting the optimization procedure.”