Shaking the foundations: delusions in sequence models for interaction and control

By Pedro A. Ortega, Markus Kunesch, Gr'egoire Del'etang, Tim Genewein, Jordi Grau-Moya, Joel Veness, Jonas Buchli, Jonas Degrave, Bilal Piot, Julien Pérolat and 9 more. 2021

75 citations recorded by the source corpus.

The recent phenomenal success of language models has reinvigorated machine learning research, and large sequence models such as transformers are being applied to a variety of domains. One important problem class that has remained relatively elusive however is purposeful adaptive behavior. Currently there is a common perception that sequence models"lack the understanding of the cause and effect of their actions"leading them to draw incorrect inferences due to auto-suggestive delusions. In this report we explain where this mismatch originates, and show that it can be resolved by treating actions as causal interventions. Finally, we show that in supervised learning, one can teach a system to condition or intervene on data by training with factual and counterfactual error signals respectively.

Subfields: Reinforcement Learning, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Safety, Robustness & Interpretability

Citations in the corpus

GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, background): “However, other failure modes can be introduced by the LM training procedure, such as “delusions” or “hallucinations” (Ortega et al., 2021; Gao, 2021; Maynez 112 et al., 2020).”
Language Model Cascades (2022, methodology): “Ortega et al. (2021) explains a formalism for LM ﬁnetun-ing with causal graphical models in order to extend the pre-2 dictive capabilities of AI agents towards more adaptive behaviour.”
A Generalist Agent (2022, methodology): “Generating actions using an autoregressive model can lead to causal “self-delusion” biases when there are confounding variables (Ortega et al., 2021).”
You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environments (2022, unknown): “Ortega et al. [30] give a high level view of how sequence modeling can be affected by delusions in various problem settings when not treating actions as interventions.”
Is Conditional Generative Modeling all you need for Decision-Making? (2022, background): “Naive extensions to partially observed MDPs (POMDPs) may cause self-delusions (Ortega et al., 2021) in Decision Diffuser.”