Minimal gated unit for recurrent neural networks

By Guoxiang Zhou, Jianxin Wu, Chen-Lin Zhang, Zhi-Hua Zhou. 2016

370 citations recorded by the source corpus.

Recurrent neural networks (RNN) have been very successful in handling sequence data. However, understanding RNN and finding the best practices for RNN learning is a difficult task, partly because there are many competing and complex hidden units, such as the long short-term memory (LSTM) and the gated recurrent unit (GRU). We propose a gated unit for RNN, named as minimal gated unit (MGU), since it only contains one gate, which is a minimal design among all gated hidden units. The design of MGU benefits from evaluation results on LSTM and GRU in the literature. Experiments on various sequence data show that MGU has comparable accuracy with GRU, but has a simpler structure, fewer parameters, and faster training. Hence, MGU is suitable in RNN's applications. Its simple architecture also means that it is easier to evaluate and tune, and in principle it is easier to study MGU's properties theoretically and empirically.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Light Gated Recurrent Units for Speech Recognition (2018, background): “When avoiding it, the norm of the gradient tends to increase (see for instance the recurrent weights U h of M-GRU model).”
Capacity and Trainability in Recurrent Neural Networks (2016, methodology): “ determine which pieces of the LSTM were more important than the others. Our UGRNN architecture is directly inspired by the large impact of removing the forget gate from the LSTM (Gers et al., 1999). Zhou et al. (2016) introduced an architecture with minimal gating that is similar to the UGRNN, but ”
A Survey on Evolutionary Neural Architecture Search (2020, methodology): “For the time-series data processing, RNNs are more suitable, so the encoding space may contain cells including -RNN cell, LSTM [204], gated recurrent unit (GRU) [227], minimally gated unit (MGU) [228], and update-gated RNN (UGRNN) [229].”
Improving Speech Recognition by Revising Gated Recurrent Units (2017, methodology): “In [31], for instance, removing rt from the GRU model has led to a single-gate architecture called Minimal Gated Recurrent Unit (M-GRU), which achieves a performance comparable to that obtained by standard GRUs in handwritten digit recognition as well as in a sentiment classification task.”
A survey on deep learning-based fine-grained object classification and semantic segmentation (2017, neutral): “Other RNN models include GNU([4]), MGU([5]).”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, background): “Minimal gated unit (MGU) [40] couples the reset and update gates of GRU.”
GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series (2019, methodology): “Similarly, we derive the minimal GRU-ODE , a variant based on the minimal GRU (Zhou et al., 2016), described in appendix G.”
Twin Regularization for online speech recognition (2018, methodology): “Beyond standard LSTM [36] and GRU [37], we also considered recently proposed architectural variations [38]: MGRU [39] is the minimal GRU architecture based on replacing reset gate with updated gate activations, while light GRU (LiGRU) [40] directly avoids the reset gate and exploits ReLU activations”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, background): “The minimal gated unit (MGU) integrates the reset gate and update gate of the GRU [87].”
Investigating recurrent neural network memory structures using neuro-evolution (2019, methodology): “The authors hope that these results will guide future memory cell development, as the addition of simple neurons dramatically improved MGU performance, but also decreased GRU performance.”
Gate-variants of Gated Recurrent Unit (GRU) neural networks (2017, background): “Gated Recurrent Neural Network (RNN) have shown success in several applications involving sequential or temporal data [1-13].”