Object Files and Schemata: Factorizing Declarative and Procedural Knowledge in Dynamical Systems

By Anirudh Goyal, Alex Lamb, Phanideep Gampa, Philippe Beaudoin, Sergey Levine, Charles Blundell, Yoshua Bengio, Michael C. Mozer. 2020

75 citations recorded by the source corpus.

Modeling a structured, dynamic environment like a video game requires keeping track of the objects and their states (\emph{declarative} knowledge) as well as predicting how objects behave (\emph{procedural} knowledge). Black-box models with a monolithic hidden state often lack \emph{systematicity}: they fail to apply procedural knowledge consistently and uniformly. For example, in a video game, correct prediction of one enemy's trajectory does not ensure correct prediction of another's. We address this issue via an architecture that factorizes declarative and procedural knowledge and that imposes modularity within each form of knowledge. The architecture consists of active modules called \emph{object files} that maintain the state of a single object and invoke passive external knowledge sources called \emph{schemata} that prescribe state updates. To use a video game as an illustration, two enemies of the same type will share schemata but will each have their own object file to encode their distinct state (e.g., health, position). We propose to use attention to control the determination of which object files to update, the selection of schemata, and the propagation of information between object files. The resulting architecture is a drop-in replacement conforming to the same input-output interface as normal recurrent networks (e.g., LSTM, GRU) yet achieves substantially better generalization on environments that have factorized declarative and procedural knowledge, including a challenging intuitive physics benchmark.

Subfields: Hardware and Architecture, Information Systems and Management, Knowledge Graphs & Representation, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Hardware and Architecture

1. Bernhard Schölkopf
2. Yoshua Bengio

In Information Systems and Management

1. Bernhard Schölkopf

In Knowledge Graphs & Representation

1. Bernhard Schölkopf
2. Yoshua Bengio
3. Anirudh Goyal
4. Nicolas Manfred Otto Heess

In Signal Processing

1. Stefan Bauer
2. Anirudh Goyal
3. Yoshua Bengio
4. Nal Kalchbrenner
5. Bernhard Schölkopf
6. Nan Rosemary Ke
7. Francesco Locatello
8. Michael C. Mozer

Citations in the corpus

• Toward Causal Representation Learning (2021, background): “Note that the problem of object-centric representation learning [11], [40], [84], [87], [88], [138], [155], [160], [255], [262] can also be considered a special case of disentangled factorization as discussed here.”
• Towards Causal Representation Learning (2021, background): “learning [10, 39, 83, 86, 87, 138, 155, 160, 262, 255] can also be considered a special case of disentangled factorization as discussed here.”
• Inductive biases for deep learning of higher-level cognition (2020, background): “In terms of model-induced parallelism, the SCOFF approach (Goyal et al., 2020) shows some promise in this direction, where most of the computation is decentralized in each of the experts, and the conscious processing is just the tip of the iceberg in terms of computational cost.”
• Neural Production Systems (2021, methodology): “NPS is an interaction network, therefore we use other widely used interaction networks such as multihead attention and graph neural networks (Goyal et al. (2019), Goyal et al. (2020), Veerapaneni et al. (2019), Kipf et al. (2019)) for comparison.”