Dynamic Recursive Neural Network

By Qiushan Guo, Zhipeng Yu, Yichao Wu, Ding Liang, Haoyu Qin, Junjie Yan. 2019

62 citations recorded by the source corpus.

This paper proposes the dynamic recursive neural network (DRNN), which simplifies the duplicated building blocks in deep neural network. Different from forwarding through different blocks sequentially in previous networks, we demonstrate that the DRNN can achieve better performance with fewer blocks by employing block recursively. We further add a gate structure to each block, which can adaptively decide the loop times of recursive blocks to reduce the computational cost. Since the recursive networks are hard to train, we propose the Loopy Variable Batch Normalization (LVBN) to stabilize the volatile gradient. Further, we improve the LVBN to correct statistical bias caused by the gate structure. Experiments show that the DRNN reduces the parameters and computational cost and while outperforms the original model in term of the accuracy consistently on CIFAR-10 and ImageNet-1k. Lastly we visualize and discuss the relation between image saliency and the number of loop time.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Jan Kautz
3. Dacheng Tao
4. Arash Vahdat
5. Ming-Hsuan Yang
6. Arun Mallya
7. José Manuel Álvarez
8. Trevor Darrell

In Electrical and Electronic Engineering

1. Shiji Song
2. Yulin Wang
3. Gao Huang
4. Jan Kautz
5. José Manuel Álvarez
6. Arash Vahdat
7. Arun Mallya

In Model Architectures

1. Arun Mallya
2. Hongxu Yin
3. Shiji Song
4. Arash Vahdat
5. Gao Huang
6. Yulin Wang
7. Honghui Wang
8. Le Yang

Citations in the corpus

• Dynamic Neural Networks: A Survey (2021, methodology): “Although the weightsharing scheme seems similar to the aforementioned IamNN [66], the skipping policy of [67] differs significantly: gating modules are exploited to decide the recursion depth.”
• A-ViT: Adaptive Tokens for Efficient Vision Transformer (2021, background): “One way is to change the architecture of the model to support adaptive computations [4,14,15, 18, 25, 27, 30, 37, 42, 51, 54].”