DIANet: Dense-and-Implicit Attention Network

By Zhongzhan Huang, Senwei Liang, Mingfu Liang, Haizhao Yang. 2019

66 citations recorded by the source corpus.

Attention networks have successfully boosted the performance in various vision problems. Previous works lay emphasis on designing a new attention module and individually plug them into the networks. Our paper proposes a novel-and-simple framework that shares an attention module throughout different network layers to encourage the integration of layer-wise information and this parameter-sharing module is referred to as Dense-and-Implicit-Attention (DIA) unit. Many choices of modules can be used in the DIA unit. Since Long Short Term Memory (LSTM) has a capacity of capturing long-distance dependency, we focus on the case when the DIA unit is the modified LSTM (called DIA-LSTM). Experiments on benchmark datasets show that the DIA-LSTM unit is capable of emphasizing layer-wise feature interrelation and leads to significant improvement of image classification accuracy. We further empirically show that the DIA-LSTM has a strong regularization ability on stabilizing the training of deep networks by the experiments with the removal of skip connections (He et al. 2016a) or Batch Normalization (Ioffe and Szegedy 2015) in the whole residual network.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhongzhan Huang
2. Senwei Liang
3. Wei He
4. Haizhao Yang
5. Mingfu Liang
6. Ming Liang

In Computer Vision and Pattern Recognition

1. Zhongzhan Huang
2. Senwei Liang
3. Wei He
4. Haizhao Yang
5. Mingfu Liang
6. Ming Liang
7. Xiaogang Wang
8. Hongsheng Li

In Media Technology

1. Zhongzhan Huang
2. Senwei Liang
3. Haizhao Yang
4. Mingfu Liang
5. Ming Liang
6. Wei He

Citations in the corpus

• AlterSGD: Finding Flat Minima for Continual Learning by Alternative Training (2021, background): “Recently, deep neural networks (DNNs) became the dominant paradigm in various real-world applications, for example, image classification [12, 18, 17, 16, 19], semantic segmentation [43, 32, 27], object detection [39, 15, 38, 46, 35].”
• Instance Enhancement Batch Normalization: an Adaptive Regulator of Batch Noise (2019, methodology): “First, the added-in module extracts internal information of a networks which can be squeezed channel-wise information (Hu, Shen, and Sun 2018; Li et al. 2019; Huang et al. 2019) or spatial information (Wang et al. 2018; Li, Hu, and Yang 2019).”
• Efficient Attention Network: Accelerate Attention by Searching Where to Plug (2020, methodology): “Previous works mainly focus on improving the capacity of self-attention module [31, 15] or reducing the parameters [21, 18, 15].”
• Convolution-Weight-Distribution Assumption: Rethinking the Criteria of Channel Pruning (2020, unknown): “, 2009) DIANet(Huang et al., 2019) Kaiming-uniform(He et al.”