This is the third post in the series about Recurrent Neural Networks. The first post defined the basic notations and the formulation of a Recurrent Cell, the second post discussed its extensions such as LSTM and GRU. Recall that LSTM layer receives vector $\mathbf{x}^{\left(t\right)}$ - the input for the current time step $t$ , and $\mathbf{c}^{\left(t-1\right)},\mathbf{h}^{\left(t-1\right)}$ the memory-vector and the output-vector from previous time-step respectively, then the layer computes a new memory vector $$\mathbf{c}^{\left(t\right)}=\mathbf{f}\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\right)\circ\mathbf{c}^{\left(t-1\right)}+\mathbf{i}\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\right)\circ\mathbf{s}\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\right)$$ where $\circ$ denotes element wise multiplication. Then the layer outputs $$\mathbf{h}^{\left(t\right)}=\mathbf{o}\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\right)\circ tanh\left(\mathbf{c}^{\left(t\right)}\right)$$ where $\mathbf{f},\mathbf{i},\mathbf{o}\in\mathcal{S}_{sigmoid}$ are the forget, input and output gates respectively, and $\mathbf{s}\in\mathcal{S}_{tanh}$ is the state transition function. $\mathcal{S}_{\sigma}$ denotes a family of all affine transformations $\mathbb{R}^{n}\times\mathbb{R}^{n}\to\mathbb{R}^{m}$ following by an element-wise activation function $\sigma$ , for any input-dimension $n$ and output-dimension $m$ .

Dropout is a very popular regularization mechanism that is being attached to layers of a neural network in order to reduce overfitting and improve generalization. Applying dropout to a layer starts by fixing some $p\in\left(0,1\right)$ . Then, any time that layer produces an output during training, each one of its neurons is being zeroed-out independently with probability $p$ , and with probability $1-p$ it is being scaled by $\frac{1}{1-p}$ . Scaling the output is important because it keeps the expected output-value of the neuron unchanged. During test time, the dropout mechanism is being turned-off, that is, the output of each neuron is being passed as is.

Wojciech Zaremba, Ilya Sutskever and Oriol Vinyals published in 2014 a paper that describe a successful dropout variation for LSTM layers. Their idea was that dropout should be applied only to the inputs of the layer and not to the recurrent connections. Moreover, a new mask should be generated for each time step.

On 2015, Taesup Moon, Heeyoul Choi, Hoshik Lee and Inchul Song published a different dropout variation: rnnDrop.

A relatively recent work of Yarin Gal and Zoubin Ghahramani from 2016 also use a make that is shared across time, however it is being applied to the inputs as well on the recurrent connections. This is one of the first successful dropout variants that actually apply the mask to the recurrent connection.

Another paper from 2016, of Stanislau Semeniuta, Aliaksei Severyn and Erhardt Barth demonstrate the mask being applied to some of the gates rather than the input\hidden vectors. For any time-step, generate $\mathbf{z}^{\left(t\right)}$ and then use it to mask the input gate $$\mathbf{c}^{\left(t\right)}=\mathbf{f}\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\right)\circ\mathbf{c}^{\left(t-1\right)}+\mathbf{z}^{\left(t\right)}\circ\mathbf{i}\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\right)\circ\mathbf{s}\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\right)$$ and the second LSTM equation left unchanged.

The very last dropout variation (to the day I wrote this post) is by David Krueger et al, from 2017. They suggested to treat the memory vector $\mathbf{c}^{\left(t\right)}$ and the output vector (a.k.a hidden vector) $\mathbf{h}^{\left(t\right)}$ as follows: each coordinate in each of the vectors either being updated as usual or preserved its value from previous time-step. As opposed to regular dropout, where “dropping a coordinate” meaning to make it zero, in zoneout it just keeps its previous value - acting as a random identity map that allows better gradients propagation through more time steps. Note that preserving a coordinate in the memory vector should not affect the computation of the hidden vector. Therefore we have to rewrite the formulas for the LSTM layer: given $\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}$ and $\mathbf{c}^{\left(t-1\right)}$ . Generate two masks $\mathbf{z}_{1}^{\left(t\right)},\mathbf{z}_{2}^{\left(t\right)}$ for the current time-step $t$ . Start by computing a candidate memory-vector with the regular formula, $$\mathbf{\hat{c}}^{\left(t\right)}=\mathbf{f}\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\right)\circ\mathbf{c}^{\left(t-1\right)}+\mathbf{i}\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\right)\circ\mathbf{s}\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\right)$$ and then use only some of its coordinates to update the memory-vector, that is, $$\mathbf{c}^{\left(t\right)}=\mathbf{z}_{1}^{\left(t\right)}\circ\mathbf{c}^{\left(t-1\right)}+\left(1-\mathbf{z}_{1}^{\left(t\right)}\right)\circ\mathbf{\hat{c}}^{\left(t\right)}$$ Similarly, for the hidden-vector, compute a candidate $$\mathbf{\hat{h}}^{\left(t\right)}=\mathbf{o}\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\right)\circ tanh\left(\mathbf{\hat{c}}^{\left(t\right)}\right)$$ and selectively update $$\mathbf{h}^{\left(t\right)}=\mathbf{z}_{2}^{\left(t\right)}\circ\mathbf{h}^{\left(t-1\right)}+\left(1-\mathbf{z}_{2}^{\left(t\right)}\right)\circ\mathbf{\hat{h}}^{\left(t\right)}$$ Observe again that the computation of the candidate hidden-vector is based on $\mathbf{\hat{c}}^{\left(t\right)}$ therefore unaffected by the mask $\mathbf{z}_{1}^{\left(t\right)}$ .

In the last year, dropout for LSTM networks gained more attention, and the list of different variants is growing quickly. This post tried to cover only some fraction of all the variants available out there. It is left for the curious reader to search the literature for more about this topic.