Masked Label Prediction: Unified Massage Passing Model for Semi-Supervised Classification

By Yunsheng Shi, Zhengjie Huang, Wenjin Wang, Hui Zhong, Shikun Feng, Yu Sun. 2020

1,148 citations recorded by the source corpus.

Graph neural network (GNN) and label propagation algorithm (LPA) are both message passing algorithms, which have achieved superior performance in semi-supervised classification. GNN performs feature propagation by a neural network to make predictions, while LPA uses label propagation across graph adjacency matrix to get results. However, there is still no effective way to directly combine these two kinds of algorithms. To address this issue, we propose a novel Unified Message Passaging Model (UniMP) that can incorporate feature and label propagation at both training and inference time. First, UniMP adopts a Graph Transformer network, taking feature embedding and label embedding as input information for propagation. Second, to train the network without overfitting in self-loop input label information, UniMP introduces a masked label prediction strategy, in which some percentage of input label information are masked at random, and then predicted. UniMP conceptually unifies feature propagation and label propagation and is empirically powerful. It obtains new state-of-the-art semi-supervised classification results in Open Graph Benchmark (OGB).

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Representation & Self-Supervised Learning

Citations in the corpus

Training Graph Neural Networks with 1000 Layers (2021, methodology): “RevGAT-Wide uses 5 attention layers with 3 heads and 356 channels for each head and outperforms the current top performer UniMP v2 (Shi et al., 2020) (74.”
Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs (2022, neutral): “, 2019; Kim & Oh, 2021) or Transformer-like dot product attention (Zhang et al., 2018a; Gao & Ji, 2019; Shi et al., 2020; Dwivedi & Bresson, 2020; Kim & Oh, 2021; Kreuzer et al., 2021).”
OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs (2021, methodology): “Regarding the GNN architectures, the BD-PGL adopted the expressive Transformer-based UniMP architecture (Shi et al., 2020), while the Academic adopted the standard MPNN (Gilmer et al.”
Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation (2021, methodology): “Constructing π(τ) ≈ R(τ)/Z, e.g. as per Buesing et al. (2019), MaxEnt RL (Haarnoja et al., 2017), or via an autoregressive method (Nash and Durkan, 2019; Shi et al., 2021) has a particular problem as shown below: if multiple action sequences ~a (i.e. multiple trajectories τ ) lead to a final state…”
A critical look at the evaluation of GNNs under heterophily: are we really making progress? (2023, methodology): “To overcome this limitation, we also use a model with a more powerful attention mechanism — Graph Transformer ( GT ) (Shi et al., 2020), which is an adaptation of the popular Transformer architecture (Vaswani et al., 2017) to graphs.”
G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering (2024, neutral): “In this work, we introduce a new GraphQA benchmark for real-world graph question answering and present G-Retriever , an architecture adept at complex and creative queries.”
How Attentive are Graph Attention Networks? (2021, methodology): “…GNNs with Transformer-style attention (dotproduct) (Zhang et al., 2018; Thekumparampil et al., 2018; Gao and Ji, 2019; Lukovnikov and Fischer, 2021; Shi et al., 2020; Dwivedi and Bresson, 2020; Busbridge et al., 2019; Rong et al., 2020a; Veličković et al., 2020), and Lee et al. (2018) conducted…”
Do Transformers Really Perform Badly for Graph Representation? (2021, background): “In this section, we highlight the most recent works which attempt to develop standard Transformer architecture-based GNN or graph structural encoding, but spend less effort on elaborating the works by adapting attention mechanism to GNNs [33, 54, 7, 23, 1, 46, 47, 55, 44].”
Do Transformers Really Perform Bad for Graph Representation? (2021, background): “In this section, we highlight the most recent works which attempt to develop standard Transformer architecture-based GNN or graph structural encoding, but spend less effort on elaborating the works by adapting attention mechanism to GNNs [32, 55, 7, 22, 1, 47, 48, 56, 45].”
Fast, accurate antibody structure prediction from deep learning on massive set of natural antibodies (2022, methodology): “Residue node embeddings are updated by graph transformer (GT) 370 layers, which extend the powerful transformer architecture to include edge information (34).”
Structure-Aware Transformer for Graph Representation Learning (2022, background): “…by encoding some carefully selected graph theoretic properties such as centrality measures and shortest path distances as positional representations (Ying et al., 2021) or by using GNNs to integrate the graph structure (Rong et al., 2020; Jain et al., 2021; Mialon et al., 2021; Shi et al., 2021).”
Language is All a Graph Needs (2023, background): “…the OGB Leaderboard 1 , including DRGAT, RevGAT, etc., as the baselines (Zhang et al., 2022a; Hamilton et al., 2017; Kipf and Welling, 2016; Li et al., 2020; Han et al., 2023a; Shi et al., 2020; Yu et al., 2022a; Veliˇckovi´c et al., 2017; Sun et al., 2020; Li et al., 2021; Zhang et al., 2023a).”
Physics-informed graph neural Galerkin networks: A unified framework for solving PDE-governed forward and inverse problems (2021, background): “The proposed framework is generic and can be coupled with various state-of-the-art GCNs [48–52] (available in torch geometric https://pytorch-geomet ric.readthedocs.io/en/latest/, which has lots of off-the-shelf examples).”
Combining Label Propagation and Simple Models Out-performs Graph Neural Networks (2020, methodology): “Recent research connects GNNs to label propagation (Wang & Leskovec, 2020; Jia & Benson, 2020) as well as Markov Random ﬁelds (Qu et al., 2019; Gao et al., 2019), and some techniques use ad hoc incorporation of label information in the features (Shi et al., 2020).”
RAS: Retrieval-And-Structuring for Knowledge-Intensive LLM Generation (2025, unknown): “For graph encoding, we implement Graph Transformer (Shi et al., 2020), selected for its robust capability in handling non-fully-connected graphs—a common characteristic of text-extracted triples.”
Multi-View Encoders for Performance Prediction in LLM-Based Agentic Workflows (2025, unknown): “To test architecture sensitivity, we conduct an ablation over five diverse GNN backbones—GCN, GAT, GCN-II, Graph Transformer, and Dir-GNN—while keeping the prompt and code views fixed.”