HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation

By Yuval Nirkin, Lior Wolf, Tal Hassner. 2020

252 citations recorded by the source corpus.

We present a novel, real-time, semantic segmentation network in which the encoder both encodes and generates the parameters (weights) of the decoder. Furthermore, to allow maximal adaptivity, the weights at each decoder block vary spatially. For this purpose, we design a new type of hypernetwork, composed of a nested U-Net for drawing higher level context features, a multi-headed weight generating module which generates the weights of each block in the decoder immediately before they are consumed, for efficient memory utilization, and a primary network that is composed of novel dynamic patch-wise convolutions. Despite the usage of less-conventional blocks, our architecture obtains real-time performance. In terms of the runtime vs. accuracy trade-off, we surpass state of the art (SotA) results on popular semantic segmentation benchmarks: PASCAL VOC 2012 (val. set) and real-time semantic segmentation on Cityscapes, and CamVid. The code is available: https://nirkin.com/hyperseg.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Lior Wolf
2. Eliya Nachmani
3. Anh Tuan Tran
4. Jiacong Xu
5. Binh-Son Hua
6. Tomer Amit
7. Zixiang Xiong
8. Tan M. Dinh

Citations in the corpus

• Representation Compensation Networks for Continual Semantic Segmentation (2022, neutral): “Data-driven deep neural networks [65,73,98,109] have made many milestones in semantic segmentation.”
• SegDiff: Image Segmentation with Diffusion Probabilistic Models (2021, background): “These include fully convolutional networks [31], encoder-decoder architectures with skip-connections, such as U-Net [38], transformer-based architectures, such as the segformer [50], and even architectures that combine hypernetworks, such as [36].”
• PIDNet: A Real-time Semantic Segmentation Network Inspired by PID Controllers (2022, methodology): “Before fine-tuning our models, we pre-train them by ImageNet [41] as most of previous works doing [20,34,35].”
• HyperInverter: Improving StyleGAN Inversion via Hypernetwork (2021, background): “They were first proposed by Ha et al. [15] and have been used in a wide range of applications from semantic segmentation [34], 3D scene representation [29, 42], neural architecture search (NAS) [54] to continual learning [45].”