Transformer for Partial Differential Equations' Operator Learning

By Zijie Li, Kazem Meidani, Amir Barati Farimani. Trans. Mach. Learn. Res., 2022

328 citations recorded by the source corpus.

Data-driven learning of partial differential equations' solution operators has recently emerged as a promising paradigm for approximating the underlying solutions. The solution operators are usually parameterized by deep learning models that are built upon problem-specific inductive biases. An example is a convolutional or a graph neural network that exploits the local grid structure where functions' values are sampled. The attention mechanism, on the other hand, provides a flexible way to implicitly exploit the patterns within inputs, and furthermore, relationship between arbitrary query locations and inputs. In this work, we present an attention-based framework for data-driven operator learning, which we term Operator Transformer (OFormer). Our framework is built upon self-attention, cross-attention, and a set of point-wise multilayer perceptrons (MLPs), and thus it makes few assumptions on the sampling pattern of the input function or query locations. We show that the proposed framework is competitive on standard benchmark problems and can flexibly be adapted to randomly sampled input.

Subfields: Electrical and Electronic Engineering, Numerical Analysis, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/179070d3d43e97d1ce4d12127a3dc63581328809

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Electrical and Electronic Engineering

In Numerical Analysis

In Statistical and Nonlinear Physics

Citations in the corpus

Geometry-Informed Neural Operator for Large-Scale 3D PDEs (2023, methodology): “This includes the Operator Transformer (OFormer) [39], Mesh-Independent Neural Operator (MINO) [40], and the General Neural Operator Transformer (GNOT) [41].”
Transolver: A Fast Transformer Solver for PDEs on General Geometries (2024, methodology): “Besides, to address the quadratic complexity of attention, OFormer (Li et al., 2023c), GNOT (Hao et al., 2023) and ONO (Xiao et al., 2024) utilize the well-established linear Transformers, such as Reformer (Ki-taev et al., 2020), Performer (Choromanski et al., 2021b) and Galerkin Transformer (Cao,…”
GNOT: A General Neural Operator Transformer for Operator Learning (2023, methodology): “Transformer models (Cao, 2021; Li et al., 2022b), based on attention mechanism, are proposed since they have a larger model capacity.”