Coordination Among Neural Modules Through a Shared Global Workspace

By Anirudh Goyal, Aniket Rajiv Didolkar, Alex Lamb, Kartikeya Badola, Nan Rosemary Ke, Nasim Rahaman, Jonathan Binas, Charles Blundell, Michael C. Mozer, Yoshua Bengio. 2021

117 citations recorded by the source corpus.

Deep learning has seen a movement away from representing examples with a monolithic hidden state towards a richly structured state. For example, Transformers segment by position, and object-centric architectures decompose images into entities. In all these architectures, interactions between different elements are modeled via pairwise interactions: Transformers make use of self-attention to incorporate information from other positions; object-centric architectures make use of graph neural networks to model interactions among entities. However, pairwise interactions may not achieve global coordination or a coherent, integrated representation that can be used for downstream tasks. In cognitive science, a global workspace architecture has been proposed in which functionally specialized components share information through a common, bandwidth-limited communication channel. We explore the use of such a communication channel in the context of deep learning for modeling the structure of complex environments. The proposed method includes a shared workspace through which communication among different specialist modules takes place but due to limits on the communication bandwidth, specialist modules must compete for access. We show that capacity limitations have a rational basis in that (1) they encourage specialization and compositionality and (2) they facilitate the synchronization of otherwise independent specialists.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Model Architectures

Citations in the corpus

Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, methodology): “[27]) use this procedure to efficiently map set inputs to a simple, homogeneous output spaces on small-scale problems.”
Perceiver: General Perception with Iterative Attention (2021, background): “…architectures with attention to the longer-horizon past (Rae et al., 2020; Dai et al., 2019) and to produce architectures that write to and/or read from ﬁxed-size arrays or memories (Santoro et al., 2018; Goyal et al., 2021), all while keeping the cost of each operation linear in the input size.”
Scalable Adaptive Computation for Iterative Generation (2022, neutral): “…et al., 2015), Neural Turing Machines (Graves et al., 2014), Stack-RNN (Joulin & Mikolov, 2015), Set Transformer (Lee et al., 2019), Memory Transformers (Burtsev et al., 2020), Slot Attention (Locatello et al., 2020), BigBird (Zaheer et al., 2020), and Workspace models (Goyal et al., 2021).”
Continual Learning via Local Module Composition (2021, background): “[28] Anirudh Goyal, Aniket Didolkar, Alex Lamb, Kartikeya Badola, Nan Rosemary Ke, Nasim Rahaman, Jonathan Binas, Charles Blundell, Michael Mozer, and Yoshua Bengio.”
XCiT: Cross-Covariance Image Transformers (2021, background): “Similarly, Goyal et al. [25] use a global workspace though which items interact, albeit one that is shared across layers.”
Luna: Linear Unified Nested Attention (2021, methodology): “Finally, we discuss the differences between Luna and three closely related models: Linformer (Wang et al., 2019), Set Transformer (Lee et al., 2019) (§3.4) and Shared Workspace (Goyal et al., 2021).”
Fast and Slow Learning of Recurrent Independent Mechanisms (2021, background): “…of the two attention mechanisms, Input Attention and Communication Attention, are learnt slowly in the outer loop as meta parameters. et al., 2020; Goyal et al., 2021) to learn a modular architecture consisting of a set of independent modules which compete with each other to attend to an input…”
Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation (2022, background): “Latent-space Transformers that use fixed-size latents instead of the full context, are one such approach [1, 103].”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, methodology): “In this line, one recent technique learns recurrent independent mechanisms by encouraging modules to become independent via a competition procedure, and combines the modules in general graph structures (Mittal et al., 2020; Goyal et al., 2021; 2022).”
On the link between conscious function and general intelligence in humans and machines (2022, background): “The broad concept in GWT of having information that is kept both independent and shared as necessary has already inspired several novel neural network architectures (Goyal et al., 2019; 2021; Juliani et al., 2022).”