How to Represent Part-Whole Hierarchies in a Neural Network

By Geoffrey E. Hinton. 2021

231 citations recorded by the source corpus.

Abstract This article does not describe a working system. Instead, it presents a single idea about representation that allows advances made by several different groups to be combined into an imaginary system called GLOM.1 The advances include transformers, neural fields, contrastive representation learning, distillation, and capsules. GLOM answers the question: How can a neural network with a fixed architecture parse an image into a part-whole hierarchy that has a different structure for each image? The idea is simply to use islands of identical vectors to represent the nodes in the parse tree. If GLOM can be made to work, it should significantly improve the interpretability of the representations produced by transformer-like systems when applied to vision or language.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Knowledge Graphs & Representation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Josh Tenenbaum
2. Antonio Torralba
3. Chuang Gan
4. Yoshua Bengio
5. Quoc V. Le
6. Sergey Levine
7. Mohit Bansal
8. Abhinav Kumar Gupta

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Luc Van Gool
3. Jitendra Malik
4. Antonio Torralba
5. Andrea Vedaldi
6. Trevor Darrell
7. Xiaogang Wang
8. Andrew Zisserman

In Model Architectures

1. Josh Tenenbaum
2. Antonio Torralba
3. Chuang Gan
4. X. Zhang
5. Guiguang Ding
6. Jian‐Yuan Sun
7. Yoshua Bengio
8. Jungong Han

In Knowledge Graphs & Representation

1. Antonio Torralba
2. Chuang Gan
3. Geoffrey E. Hinton
4. Josh Tenenbaum

Citations in the corpus

• PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning (2021, background): “Also, results show that current models fail to represent part-whole hierarchies delicately, encouraging the design of novel neural networks that specializes in this aspect ( e.g., Hinton [19] proposes some feasible ideas.)”
• The Forward-Forward Algorithm: Some Preliminary Investigations (2022, background): “The key to overcoming this apparent limitation of FF is to treat a static image as a rather boring video that is processed by a multi-layer recurrent neural network (Hinton, 2021).”
• Scaling Up Your Kernels to 31×31: Revisiting Large Kernel Design in CNNs (2022, background): “More works [21,40,44,100,121] explained the superiority of ViTs from different point of views.”
• Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning (2022, background): “Broader Impacts: Many problems in biology and medicine have hierarchical-like relationships [34, 48, 55].”