Towards Scalable and Reliable Capsule Networks for Challenging NLP Applications
By Wei Zhao, Haiyun Peng, Steffen Eger, Erik Cambria, Min Yang. 2019
130 citations recorded by the source corpus.
Obstacles hindering the development of capsule networks for challenging NLP applications include poor scalability to large output spaces and less reliable routing processes. In this paper, we introduce: (i) an agreement score to evaluate the performance of routing processes at instance-level; (ii) an adaptive optimizer to enhance the reliability of routing; (iii) capsule compression and partial routing to improve the scalability of capsule networks. We validate our approach on two NLP tasks, namely: multi-label text classification and question answering. Experimental results show that our approach considerably improves over strong competitors on both tasks. In addition, we gain the best results in low-resource settings with few training instances.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Tasks (2021, methodology): “Recently, capsule networks (Hinton et al., 2011) have been used in sentiment classification and text classification (Chen and Qian, 2019; Zhao et al., 2019).”
- MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance (2019, methodology): “We study two alternatives for φ : (i) the concatenation of power means (R¨uckl´e et al., 2018) as a generalized pooling mechanism, and (ii) a routing mechanism for aggregation (Zhao et al., 2018, 2019).”
- A Survey on Data Augmentation for Text Classification (2021, unknown): “Large pre-trained models are currently state of the art, nevertheless, we advise taking further advancements and findings in the research landscape into account, as for example deep belief networks [151], capsule networks [152], or task-specialized networks e.”
- Canonical Capsules: Unsupervised Capsules in Canonical Pose (2020, background): “Techniques such as Dynamic Routing [37, 47] and EM-algorithms [24] have been proposed as potential architectures, and have found applications ranging from medical imaging [1] to language understanding [57].”
- Deep Learning--based Text Classification (2020, unknown): “[71–73] propose a text classification model based on a variant of CapsNets.”
- SenticNet 6: Ensemble Application of Symbolic and Subsymbolic AI for Sentiment Analysis (2020, background): “More recent approaches have been focusing on the development of sentiment-specific word embeddings [44], which are able to encode more affective clues than regular word vectors, and on the use of context-aware subsymbolic approaches such as attention modeling [32, 33] and capsule networks [13, 66].”
- Automatic Text Evaluation through the Lens of Wasserstein Barycenters (2021, background): “Precisely, let’s assume that a CE is composed of L layers, the Moverscore’s context representation is given for each word ωj by Φ(ωj) = T (φ1(ωj), . . . , φL(ωj)), where the transformation T is either power means (Rücklé et al., 2018a) or aggregation routines depicted in (Zhao et al., 2018, 2019a).”
- Sentiment analysis using deep learning architectures: a review (2019, background): “Moreover, capsule networks are getting famous in natural language processing for various text classification tasks (Du et al. 2019a; Zhao et al. 2019; Kim et al. 2019).”
- Aspect-based sentiment analysis via affective knowledge enhanced graph convolutional networks (2021, unknown): “Besides, for capsule network-based models [21], Du et al. [22] construct vector-based feature representations and cluster features for aspect-related sentiment extraction based on an EM routing algorithm, Chen and Qian [23] transfer document-level knowledge to aspect sentiment classification with…”
- A survey on empathetic dialogue systems (2020, background): “, capsule networks [148], but also by deconstructing the problem into all the relevant subtasks involved in conversation understanding, e.”