Choose a Transformer: Fourier or Galerkin

By Shuhao Cao. Neural Information Processing Systems, 2021

450 citations recorded by the source corpus.

In this paper, we apply the self-attention from the state-of-the-art Transformer in Attention Is All You Need for the first time to a data-driven operator learning problem related to partial differential equations. An effort is put together to explain the heuristics of, and to improve the efficacy of the attention mechanism. By employing the operator approximation theory in Hilbert spaces, it is demonstrated for the first time that the softmax normalization in the scaled dot-product attention is sufficient but not necessary. Without softmax, the approximation capacity of a linearized Transformer variant can be proved to be comparable to a Petrov-Galerkin projection layer-wise, and the estimate is independent with respect to the sequence length. A new layer normalization scheme mimicking the Petrov-Galerkin projection is proposed to allow a scaling to propagate through attention layers, which helps the model achieve remarkable accuracy in operator learning tasks with unnormalized data. Finally, we present three operator learning experiments, including the viscid Burgers' equation, an interface Darcy flow, and an inverse interface coefficient identification problem. The newly proposed simple attention-based operator learner, Galerkin Transformer, shows significant improvements in both training cost and evaluation accuracy over its softmax-normalized counterparts.

Subfields: Computational Mechanics, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/72f207c777e4a17180cc54ccc6a743d5f43227af

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Statistical and Nonlinear Physics

Citations in the corpus

Transformer for Partial Differential Equations' Operator Learning (2022, methodology): “For Burgers’ equation and Darcy ﬂow, the dataset splitting is same as in Cao (2021).”
GNOT: A General Neural Operator Transformer for Operator Learning (2023, methodology): “Transformer models (Cao, 2021; Li et al., 2022b), based on attention mechanism, are proposed since they have a larger model capacity.”
Transolver: A Fast Transformer Solver for PDEs on General Geometries (2024, methodology): “…the PDE-solving task as an iterative updated process (Li et al., 2020b) and prove that canonical attention is a Monte-Carlo approximation of the integral operator on the input domain Ω (Cao, 2021; Kovachki et al., 2023), which can be used to approximate the solving process of each iteration step.”
Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers (2021, background): “, sparse transformer Child et al. (2019), image transformer Parmar et al.”