Initialization matters: Orthogonal Predictive State Recurrent Neural Networks

By Krzysztof Marcin Choromanski, Carlton Downey, Byron Boots. 2018

22 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Rethinking Attention with Performers (2020, methodology): “It turns out that: The proof of that fact can be found in the supplement of (Choromanski et al., 2018b), yet we provide it below for completeness and the convenience of the Reader: Lemma 3.”
Recurrent Neural Filters: Learning Independent Bayesian Filtering Steps for Time Series Prediction (2019, methodology): “Predictive state RNNs (PSRNN) (Choromanski et al., 2018; Downey et al., 2017; Venkatraman et al., 2017) use an alternative formulation of the Bayes ﬁlter, utilising a state representation that corresponds to the statistics of a distribution for future observations.”
Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers (2020, methodology): “ORFs were introduced to reduce the variance of Monte Carlo estimators [56, 13, 12, 10, 41, 9, 11] and we show in Secs.”
Learning with Hyperspherical Uniformity (2021, background): “There are quite a number of relational regularizations that have been used in neural networks, such as orthogonality regularization [4, 52, 67, 32, 15], unitary constraint [34, 81, 3], decorrelation [67, 16, 85], spectral regularization [86], low-rank regularization [76], angular constraint [84, 44]”
Demystifying Orthogonal Monte Carlo and Beyond (2020, methodology): “Monte Carlo (MC) methods are widely applied in machine learning in such domains as: dimensionality reduction [1, 3, 2], scalable kernel methods with random feature maps [34], generative modeling and variational autoencoders via sliced Wasserstein distances [36], approximating Gaussian smoothings in ”