Learning word dependencies in text by means of a deep recurrent belief network

By Iti Chaturvedi, Yew-Soon Ong, Ivor Wai-Hung Tsang, Roy E. Welsch, Erik Cambria. 2016

117 citations recorded by the source corpus.

We propose a deep recurrent belief network with distributed time delays for learning multivariate Gaussians. Learning long time delays in deep belief networks is difficult due to the problem of vanishing or exploding gradients with increase in delay. To mitigate this problem and improve the transparency of learning time-delays, we introduce the use of Gaussian networks with time-delays to initialize the weights of each hidden neuron. From our knowledge of time delays, it is possible to learn the long delays from short delays in a hierarchical manner. In contrast to previous works, here dynamic Gaussian Bayesian networks over training samples are evolved using Markov Chain Monte Carlo to determine the initial weights of each hidden layer of neurons. In this way, the time-delayed network motifs of increasing Markov order across layers can be modeled hierarchically using a deep model. To validate the proposed Variable-order Belief Network (VBN) framework, it is applied for modeling word dependencies in text. To explore the generality of VBN, it is further considered for a real-world scenario where the dynamic movements of basketball players are modeled. Experimental results obtained showed that the proposed VBN could achieve over 30% improvement in accuracy on real-world scenarios compared to the state-of-the-art baselines.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Ensemble application of convolutional and recurrent neural networks for multi-label text categorization (2017, background): “Recently, several deep learning works have applied neural networks, such as CNN and RNN, to various NLP tasks, including multi-task learning [7], sentence classification [8], multimodal analysis [22], semantic relevance detection [10], [23], polarity detection [24], machine translation [25], sarcasm”
Convolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis (2016, background): “Deep CNNs have good accuracy on topic classification of videos, however they can get stuck in local minima on fine-grained problems such as sentiment and emotion detection [28], [29].”
Natural language based financial forecasting: a survey (2017, background): “With the recent advances in deep learning, this vector representation, or word embedding [78,26], is better formed.”
MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare (2021, background): “Selfsupervised large-scale pretraining facilitates the learning of semantic and contextual information and benefits various downstream applications such as text classification (Chaturvedi et al., 2016), sentiment analysis (Cambria et al.”
A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks (2016, background): “, 2016a), recurrent belief networks (Chaturvedi et al., 2016), statistical learning theory (Oneto et al.”