Fast dynamic routing based on weighted kernel density estimation

By Suofei Zhang, Wei Zhao, Xiaofu Wu, Quan Zhou. 2018

67 citations recorded by the source corpus.

Capsules as well as dynamic routing between them are most recently proposed structures for deep neural networks. A capsule groups data into vectors or matrices as poses rather than conventional scalars to represent specific properties of target instance. Based on pose, a capsule should be attached to a probability (often denoted as activation) for its presence. The dynamic routing helps capsule network achieve more generalization capacity with fewer model parameters. However, the bottleneck, which prevents widespread applications of capsule, is the expense of computation during routing. To address this problem, we generalize existing routing methods within the framework of weighted kernel density estimation, proposing two fast routing methods with different optimization strategies. Our methods prompt the time efficiency of routing by nearly 40% with negligible performance degradation. By stacking a hybrid of convolutional layers and capsule layers, we construct a network architecture to handle inputs at a resolution of 64 × 64 pixels. The proposed models achieve a parallel performance with other leading methods in multiple benchmarks.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Erik Cambria
2. Wei Zhao
3. Steffen Eger
4. Yang Gao
5. Percy Liang
6. Zhou Zhao
7. Jianfeng Gao
8. Yee Whye Teh

In Computational Mechanics

1. Geoffrey E. Hinton
2. Wei Zhao
3. Tolga Birdal
4. Haowen Deng
5. Sara Sabour
6. Fei Liu
7. Yee Whye Teh
8. Yongheng Zhao

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Geoffrey E. Hinton
3. Dacheng Tao
4. Xiaogang Wang
5. Yi Yang
6. Jitendra Malik
7. Andrew Zisserman
8. Cordelia Schmid

Citations in the corpus

• Stacked Capsule Autoencoders (2019, background): “Iterative routing is inefficient and has prompted further research (Wang and Liu, 2018; Zhang et al., 2018; Li et al., 2018).”
• MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance (2019, methodology): “Specifically, we introduce a nonparametric clustering with Kernel Density Estimation (KDE) for routing since KDE bridges a family of kernel functions with underlying empirical distributions, which often leads to computational efficiency (Zhang et al., 2018), defined as: where d ( .”
• Towards Scalable and Reliable Capsule Networks for Challenging NLP Applications (2019, methodology): “Despite some recent success in NLP tasks (Wang et al., 2018; Xia et al., 2018; Xiao et al., 2018; Zhang et al., 2018a; Zhao et al., 2018), a few important obstacles still hinder the development of capsule networks for mature NLP applications.”
• 3D Point Capsule Networks (2018, methodology): “[47] unified the existing routing methods under one umbrella and proposed weighted kernel density estimation based routing methods.”
• Investigating Capsule Networks with Dynamic Routing for Text Classification (2018, methodology): “(Zhang et al., 2018) generalized existing routing methods within the framework of weighted kernel density estimation.”