A gate is a function that takes such a tuple and produces a vector of values between zero and one. Note that any function in $\mathcal{S}_{sig}$ and in $\mathcal{S}_{tanh}$ is actually a gate because of the properties of the $tanh$ and $sig$ functions. We add gates into a recurrent neuron in order to control how much information will flow over the “recurrent connection”. That is, suppose that $\mathbf{h}^{\left(t\right)}$ is the layer’s current state that should be used at the next time step, a gated layer will use the vector $\hat{\mathbf{h}}^{\left(t\right)}=\mathbf{h}^{\left(t\right)}\circ\mathbf{s}\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\right),$ instead of using $\mathbf{h}^{\left(t\right)}$ (when the symbol $\circ$ denotes element-wise multiplication). Note that any coordinate in $\hat{\mathbf{h}}^{\left(t\right)}$ is a “moderated” version of the corresponding coordinate in $\mathbf{h}^{\left(t\right)}$ because each entry in the gate’s output is in $\left(0,1\right).$ We will see a concrete example in the next section.

The main addition to the recurrent layer structure is the use of gates and a memory vector for each time step, denoted by $\mathbf{c}^{\left(t\right)}$ . The LSTM layer gets as inputs the tuple $\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\right)$ and the previous memory vector $\mathbf{c}^{\left(t-1\right)},$ then outputs $\mathbf{h}^{\left(t\right)}$ and an updated memory vector $\mathbf{c}^{\left(t\right)}.$

Here is how an LSTM layer computes its outputs: it have the following gates $\mathbf{f},\mathbf{i},\mathbf{o}\in\mathcal{S}_{sig}$ named the forget, input and output gate respectively, and a state-transition function $\mathbf{s}\in\mathcal{S}_{tanh}$ . It first updates the memory vector $$\mathbf{c}^{\left(t\right)}=\mathbf{f}\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\right)\circ\mathbf{c}^{\left(t-1\right)}+\mathbf{i}\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\right)\circ\mathbf{s}\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\right)$$ and then computes $$\mathbf{h}^{\left(t\right)}=\mathbf{o}\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\right)\circ tanh\left(\mathbf{c}^{\left(t\right)}\right)$$ Those equations can be explained as follows: the first equation is an element-wise summation of two terms. The first term is the previous memory vector $\mathbf{c}^{\left(t-1\right)}$ moderated by the forget gate. That is, the layer uses the current input $\mathbf{x}^{\left(t\right)}$ and previous output $\mathbf{h}^{\left(t-1\right)}$ to determine how much to shrink each coordinate of the previous memory vector. The second term is the candidate for the new state, i.e. $\mathbf{s}\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\right),$ moderated by the input gate. Note that all the gates operate on the same input tuple $\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\right).$

The input and forget gates control the long-short term dependencies (i.e. the recurrent connection), and allow the LSTM layer to adaptively control the balance between new information that come from the state-transition function and the history information that comes from the memory vector, hence the names for the gates: input and forget.

Another difference is that LSTM layer controls how much of its inner-memory to expose by using the output gate. That is being formulated in the second equation.

The addition of gates is what preventing the exploding and vanishing gradients problem. It makes the LSTM layer able to learn long and short term dependencies at the cost of increasing the number of parameters that are needed to be trained, and that makes the network harder to optimize.

GRU has two gate functions $\mathbf{z},\mathbf{r}\in\mathcal{S}_{sig}$ named the update and reset gate respectively, and a state transition $\mathbf{s}\in\mathcal{S}_{tanh}.$ The input to GRU layer is only the tuple $\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\right)$ and the output is $\mathbf{h}^{\left(t\right)}$ computed as follow: first the layer computes its gates for the current time step, denoted by $$\mathbf{z}^{\left(t\right)}:=\mathbf{z}\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\right)$$ for the update gate and by $$\mathbf{r}^{\left(t\right)}:=\mathbf{r}\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\right)$$ for the reset gate. Then the output of the layer is $$\mathbf{h}^{\left(t\right)}=\left(1-\mathbf{z}^{\left(t\right)}\right)\circ\mathbf{h}^{\left(t-1\right)}+\mathbf{z}^{\left(t\right)}\circ\mathbf{s}\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\circ\mathbf{r}^{\left(t\right)}\right)$$ with all the arithmetic operations being done element wise.

The term $\mathbf{s}\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\circ\mathbf{r}^{\left(t\right)}\right)$ is a candidate for the next state. Note that the state-transition function accepts the previous state moderated by the reset gate, allowing it to forget past states (therefore the name of the gate: reset). Then the output of the layer is a linear interpolation between the previous state and the candidate state, controlled by the update gate.

As oppose to the LSTM cell, the GRU doesn’t have an output gate to control how much of its inner state to expose therefore the entire state is being exposed at each time step.

Both LSTM and GRU are very common in many recurrent network architectures and achieve great results in many tasks. LSTMs and GRUs can learn dependencies of various lengths which make the network very expressive. However, a too expressive network can leads sometimes to overfitting, to prevent that it is common to use some type of regularization, such as Dropout.

Next post I will discuss the dropout variations that are specific for RNNs.