Tensor Switching Networks
By Chuan-Yung Tsai, Andrew M. Saxe, David D. Cox. 2016
10 citations recorded by the source corpus.
We present a novel neural network algorithm, the Tensor Switching (TS) network, which generalizes the Rectified Linear Unit (ReLU) nonlinearity to tensor-valued hidden units. The TS network copies its entire input vector to different locations in an expanded representation, with the location determined by its hidden unit activity. In this way, even a simple linear readout from the TS representation can implement a highly expressive deep-network-like function. The TS network hence avoids the vanishing gradient problem by construction, at the cost of larger representation size. We develop several methods to train the TS network, including equivalent kernels for infinitely wide and deep TS networks, a one-pass linear learning algorithm, and two backpropagation-inspired representation learning algorithms. Our experimental results demonstrate that the TS network is indeed more expressive and consistently learns faster than standard ReLU networks.
Subfields: Computational Mathematics, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Mathematics
In Computer Vision and Pattern Recognition
- Ameet Talwalkar
- Razvan Pascanu
- Virginia Smith
- Tom Schaul
- Maziar Sanjabi
- Diana Borsa
- Joseph Modayil
- Luc Van Gool
In Statistical and Nonlinear Physics
- Ameet Talwalkar
- Tom Schaul
- Razvan Pascanu
- Diana Borsa
- Joseph Modayil
- Yoshua Bengio
- Kyunghyun Cho
- Li Deng
Citations in the corpus
- Federated Multi-Task Learning (2017, background): “While MOCHA does not apply to non-convex deep learning models in its current form, we note that there may be natural connections between this approach and “convexified” deep learning models [6, 34, 51, 56] in the context of kernelized federated multi-task learning.”
- Ray Interference: a Source of Plateaus in Deep Reinforcement Learning (2019, background): “Assuming knowledge of the components of the objective, the multi-task literature offers a plethora of approaches to avoid negative interference, from modular or multi-head architectures to gating or attention mechanisms [3, 7, 32, 37, 41, 44].”