Norm-preserving Orthogonal Permutation Linear Unit Activation Functions (OPLU)

By Artem Chernodub, Dimitri Nowicki. 2016

30 citations recorded by the source corpus.

We propose a novel activation function that implements piece-wise orthogonal non-linear mappings based on permutations. It is straightforward to implement, and very computationally efficient, also it has little memory requirements. We tested it on two toy problems for feedforward and recurrent networks, it shows similar performance to tanh and ReLU. OPLU activation function ensures norm preservance of the backpropagated gradients, therefore it is potentially good for the training of deep, extra deep, and recurrent neural networks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dustin Tran
2. Balaji Lakshminarayanan
3. C. Pal
4. Roger Baker Grosse
5. Shreyas Padhy
6. James Lucas
7. Tania Bedrax-Weiss
8. Chenglong Bao

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Trevor Darrell
4. Jitendra Malik
5. Xiaogang Wang
6. Ming-Hsuan Yang
7. Bernt Schiele
8. Balaji Lakshminarayanan

In Model Architectures

1. James Lucas
2. Samuel Kadoury
3. Roger Baker Grosse
4. Zuoqiang Shi
5. C. Pal
6. Chenglong Bao
7. Kaisheng Ma
8. Yukang Shi

Citations in the corpus

• Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (2020, methodology): “, gradient penalty or norm-preserving activation (An et al., 2015; Anil et al., 2019; Chernodub and Nowicki, 2017; Gulrajani et al., 2017)), and we chose spectral normalization in this work due to its simplicity and its minimal impact on a DNN’s architecture and the optimization dynamics (Bartlett e”
• Sorting out Lipschitz function approximation (2018, methodology): “We make use of an alternative activation function called GroupSort — a variant of which was proposed by Chernodub & Nowicki [11] — which sorts groups of activations.”
• On orthogonality and learning recurrent networks with long term dependencies (2017, methodology): “The OPLU preserves norm, making a fully norm-preserving RNN possible.”
• Task-Oriented Feature Distillation (2020, background): “propose the orthogonal permutation layer as a novel activation function to perform non-linear mappling [9].”