Trellis Networks for Sequence Modeling

By Sequence Modeling, Shaojie Bai, J. Kolter, Vladlen Koltun. 2018

165 citations recorded by the source corpus.

We present trellis networks, a new architecture for sequence modeling. On the one hand, a trellis network is a temporal convolutional network with special structure, characterized by weight tying across depth and direct injection of the input into deep layers. On the other hand, we show that truncated recurrent networks are equivalent to trellis networks with special sparsity structure in their weight matrices. Thus trellis networks with general weight matrices generalize truncated recurrent networks. We leverage these connections to design high-performing trellis networks that absorb structural and algorithmic elements from both recurrent and convolutional models. Experiments demonstrate that trellis networks outperform the current state of the art methods on a variety of challenging benchmarks, including word-level language modeling and character-level language modeling tasks, and stress tests designed to evaluate long-term memory retention. The code is available at this https URL .

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Compressive Transformers for Long-Range Sequence Modelling (2019, background): “This dataset has been a popular recent LM benchmark due to the potential to exploit longer-range dependencies (Grave et al., 2016; Rae et al., 2018; Bai et al., 2018b).”
Deep Equilibrium Models (2019, methodology): “…(with passthrough connections from the input to each layer), which consist of the update We note that this model encapsulates classes such as the trellis network [8] and the universal transformer [18] (which is typically not written with passthrough connections, but this is a trivial modiﬁcation).”
Efficiently Modeling Long Sequences with Structured State Spaces (2021, methodology): “[4] 99.20 98.13 73.42 TCN [3] 99.0 97.2 - LSTM [17, 21] 98.9 95.11 63.01 r-LSTM [42] 98.4 95.2 72.2 Dilated GRU [5] 99.0 94.6 - Dilated RNN [5] 98.0 96.1 - IndRNN [25] 99.0 96.0 - expRNN [24] 98 Baselines.”
HiPPO: Recurrent Memory with Optimal Polynomial Projections (2020, background): “This has spurred various approaches to extend this fixed context window subjected to compute and storage constraints [6, 15, 18, 42, 59, 60, 64, 74].”
RAFT: Recurrent All-Pairs Field Transforms for Optical Flow (2020, methodology): “Our method also has ties to TrellisNet [5] and Deep Equilibrium Models (DEQ) [6], which both use depth tied weights over a large number of layers.”
Training Graph Neural Networks with 1000 Layers (2021, methodology): “Inspired by efficient architectures from computer vision and natural language processing (Gomez et al., 2017; Xie et al., 2017; Bai et al., 2018; 2019), here we investigate several methods to obtain more efficient GNNs that use less memory than conventional architectures while achieving state-of-the”
Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers (2021, result): “In contrast to previous baselines [4, 31, 44], we did not use hyperparameters for improving stability and regularization such as weight decay, gradient clipping, weight norm, input dropout, etc.”
Multiplicative Interactions and Where to Find Them (2020, unknown): “6 Trellis Networks Bai et al. (2019) - 30.”
SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction (2021, methodology): “Specifically, we apply intermediate supervision [5] on the output of each SCINet using the ground-truth values, to ease the learning of the intermediate temporal features.”
A Survey of Sound Source Localization with Deep Learning Methods (2021, unknown): “An indirect use of an MLP is explored in [111], where the authors use a 3-layer MLP to enhance the interaural phase difference (IPD) (see Section V) of the input signal, which is then used for DoA estimation.”
Multiscale Deep Equilibrium Models (2020, background): “Fourth, whereas past work on DEQs could leverage state-of-the-art weight-tied architectures for sequence modeling as the basis for the transformation fθ [4, 15], no such counterparts exist in state-of-the-art computer vision modeling.”
Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks (2021, methodology): “Also, the basic neural architectures we use are not new and build upon prior studies of weight sharing and recurrence [Pinheiro and Collobert, 2014, Liang and Hu, 2015, Alom et al., 2018, Bai et al., 2018, 2019, Lan et al., 2020, Jaegle et al., 2021].”
Parameter Efficient Multimodal Transformers for Video Representation Learning (2020, background): “Trellis network (Bai et al., 2019b) is a temporal convolutional architecture with weight-tying across time and depth.”