Modern Hopfield Networks and Attention for Immune Repertoire Classification
By Michael Widrich, Bernhard Schäfl, Milena Pavlović, Hubert Ramsauer, Lukas Gruber, Markus Holzleitner, Johannes Brandstetter, Geir Kjetil Ferkingstad Sandve, Victor Greiff, Sepp Hochreiter and 1 more. 2020
148 citations recorded by the source corpus.
A central mechanism in machine learning is to identify, store, and recognize patterns. How to learn, access, and retrieve such patterns is crucial in Hopfield networks and the more recent transformer architectures. We show that the attention mechanism of transformer architectures is actually the update rule of modern Hop-field networks that can store exponentially many patterns. We exploit this high storage capacity of modern Hopfield networks to solve a challenging multiple instance learning (MIL) problem in computational biology: immune repertoire classification. Accurate and interpretable machine learning methods solving this problem could pave the way towards new vaccines and therapies, which is currently a very relevant research topic intensified by the COVID-19 crisis. Immune repertoire classification based on the vast number of immunosequences of an individual is a MIL problem with an unprecedentedly massive number of instances, two orders of magnitude larger than currently considered problems, and with an extremely low witness rate. In this work, we present our novel method DeepRC that integrates transformer-like attention, or equivalently modern Hopfield networks, into deep learning architectures for massive MIL such as immune repertoire classification. We demonstrate that DeepRC outperforms all other methods with respect to predictive performance on large-scale experiments, including simulated and real-world virus infection data, and enables the extraction of sequence motifs that are connected to a given disease class. Source code and datasets: https://github.com/ml-jku/DeepRC
Subfields: Biomedical Engineering, Molecular Biology, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Biomedical Engineering
In Molecular Biology
- Sergey Ovchinnikov
- David Baker
- J. Dauparas
- Roshan Rao
- Sepp Hochreiter
- Peter K. Koo
- Günter Klambauer
- Michael Widrich
In Model Architectures
- Günter Klambauer
- Michael Widrich
- Angela Bitto-Nemling
- Lukas Gruber
- Bernhard Schäfl
- Sepp Hochreiter
- Johannes Brandstetter
- Hubert Ramsauer
Citations in the corpus
- CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP (2021, methodology): “The state vectors retrieve following vectors from modern Hopfield networks (Ramsauer et al., 2021; 2020; Widrich et al., 2020): Uxi = U softmax(β U xi) , Uyi = U softmax(β U yi) , (A98) Vxi = V softmax(β V xi) , Vyi = V softmax(β V yi) .”
- Single Layers of Attention Suffice to Predict Protein Contacts (2020, background): “Widrich et al. (2020) and Ramsauer et al. (2020) connect Hopfield networks to attention, then apply them to immune repertoire classification.”
- Hopular: Modern Hopfield Networks for Tabular Data (2022, background): “Hopular is a Deep Learning architecture, where each layer is equipped with continuous modern Hopfield networks (Ramsauer et al., 2021; Widrich et al., 2020).”
- Hopfield Networks is All You Need (2020, methodology): “With over 30 billion sequences in total, this represents one of the largest multiple instance learning experiments ever conducted [15] (for details see [70], appendix, Section A2).”
- History Compression via Language Models in Reinforcement Learning (2022, methodology): “MHNs have already been successfully applied to immune repertoire classification (Widrich et al., 2020), and chemical reaction prediction (Seidl et al., 2022).”