We start by reviewing some of the notations that we used during the series.

We defined $\mathcal{S}_{\sigma}$ to be the family of all affine transformations $\mathbb{R}^{n}\times\mathbb{R}^{n}\to\mathbb{R}^{m}$ for any dimensions $n$ and $m$ , followed by an element-wise activation function $\sigma$ , that is,

$$\mathcal{S}_{\sigma}=\left\{ \mathbf{x},\mathbf{h}\mapsto\sigma\left(W\mathbf{x}+U\mathbf{h}+\mathbf{b}\right)\mid\forall W,U,\mathbf{b}\right\}$$ and noted that with the $sigmoid\left(x\right)=\left(1+e^{-x}\right)^{-1}$ as the activation function, all the functions in $\mathcal{S}_{sig}$ are gate functions, that is, their output belongs to $\left[0,1\right]^{m}$ .

Usually we do not specify $n$ and $m$ , we assume that whenever there is a matrix multiplication of element-wise operation the dimensions always match, as they are not important for the discussion.

A new memory-vector is then being computed by $$\mathbf{c}^{\left(t\right)}=f^{\left(t\right)}\circ\mathbf{c}^{\left(t-1\right)}+i^{\left(t\right)}\circ\mathbf{s}\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\right)$$ where $\circ$ is element-wise multiplication and $\mathbf{s}\in\mathcal{S}_{tanh}$ is the state-transition function. The layer then outputs $\mathbf{h}^{\left(t\right)}=o^{\left(t\right)}\circ tanh\left(\mathbf{c}^{\left(t\right)}\right)$ .

The gate functions are of great importance for LSTM layer, they are what allow the layer to learn short and long term dependencies, and they also help avoiding the exploding and vanishing gradients problem.

We end up with a sum of two components: one that depends on the current input and one that depends on the past (i.e. $\mathbf{h}^{\left(t-1\right)}$ which encodes information about previous time steps), and the component with larger magnitude will dominate the transition. If $W\mathbf{x}^{\left(t\right)}$ is larger, the layer will not be sensitive enough to the past, and if $U\mathbf{h}^{\left(t-1\right)}$ is larger, then the layer will not be sensitive to subtle changes in the input.

The author noted that since in most cases the input $\mathbf{x}^{\left(t\right)}$ is an 1-hot vector, then multiply by $W$ is just selecting a specific column. So we may think of any gate function $\mathbf{s}\in\mathcal{S}_{\sigma}$ as a fixed base affine transformation $\mathbf{h}\mapsto U\mathbf{h}$ combined with input-dependent bias vector $\mathbf{b}_{\mathbf{x}^{\left(t\right)}}=W\mathbf{x}^{\left(t\right)}$ . That is $$\mathbf{s}\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\right)=U\mathbf{h}^{\left(t-1\right)}+\mathbf{b}_{\mathbf{x}^{\left(t\right)}}$$ that formula emphasis the additive effect of the current input-vector on the transition of $\mathbf{h}^{\left(t-1\right)}$ .

The goal in multiplicative LSTM (mLSTM), is to have an unique affine transformation (in our case, unique matrix $U$ ) for each possible input. Obviously, if the number of possible inputs is very large, the number of the parameters will explode and it won’t be feasible to train the network. To overcome that, the authors of the paper suggested to learn shared intermediate matrices that will be used to construct an unique $U$ for each input, and because the factorization is shared, there are less parameters to learn.

The factorization is defined as follow: the unique matrix $U_{\mathbf{x}^{\left(t\right)}}$ is constructed by $V_{1}diag\left(T\mathbf{x}^{\left(t\right)}\right)V_{2}$ where $V_{1},V_{2}$ and $T$ are intermediate matrices which are shared across all the possible inputs, and $diag\left(\mathbf{v}\right)$ is an operation that maps any vector $\mathbf{v}$ to a square diagonal matrix with the elements of $\mathbf{v}$ on its diagonal.

Note that the target dimension of $T$ may be arbitrarily large (in the paper they chose it to be the same as $\mathbf{h}^{\left(t\right)}$ ).

The difference between LSTM and mLSTM is only in the definition of the gate functions family, all the equations for updating the memory cell and the output are the same. In mLSTM the family $\mathcal{S}_{\sigma}$ is being replaced with $$\begin{aligned} \mathcal{S}'_{\sigma} & =\left\{ \mathbf{x},\mathbf{h}\mapsto\sigma\left(W\mathbf{x}+U_{\mathbf{x}}\mathbf{h}\right)\right\} \\ & =\left\{ \mathbf{x},\mathbf{h}\mapsto\sigma\left(\mathbf{b}_{\mathbf{x}}+V_{1}diag\left(T\mathbf{x}\right)V_{2}\mathbf{h}\right)\mid\forall W,V_{1},V_{2},T\right\} \end{aligned}$$ Note that we can reduce the number of parameters even further by forcing all the gate functions to use the same $V_{2}$ and $T$ matrices. That is, each gate will be parametrized only by $W$ and $V_{1}$ .

Formally, define the following transformation $\tau:\mathbb{R}^{n}\times\mathbb{R}^{n}\to\mathbb{R}^{m}$ such that $\tau\left(\mathbf{x},\mathbf{h}\right)=T\mathbf{x}\circ V_{2}\mathbf{h}$ and then we can define $\mathcal{S}'_{\sigma}$ for some fixed learned transformation $\tau$ as $$\begin{aligned} \mathcal{S}'_{\sigma,\tau} & =\left\{ \mathbf{x},\mathbf{h}\mapsto\sigma\left(W\mathbf{x}+V_{1}\tau\left(\mathbf{x},\mathbf{h}\right)\right)\mid\forall W,V_{1}\right\} \\ & =\left\{ \mathbf{x},\mathbf{h}\mapsto\mathbf{s}\left(\mathbf{x},\tau\left(\mathbf{x},\mathbf{h}\right)\right)\mid\mathbf{s}\in\mathcal{S}_{\sigma}\right\} \end{aligned}$$ in other words, mLSTM is an LSTM that apply its gates to the tuple $\left(\mathbf{x}^{\left(t\right)},\tau\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\right)\right)$ rather than $\left(\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\right)$ , and $\tau$ is another learned transformation. If you look again at the exact formula of $\tau$ you will see the multiplicative effect of the input vector on the transformation of $\mathbf{h}$ . That way, the authors said, $\mathcal{S}'_{\sigma,\tau}$ can yield much richer family of input-dependent transformations for $\mathbf{h}^{\left(t\right)}$ , which can be sensitive to the past as well as to subtle changes in the current input.

That paper from February 2017, applied batch normalization also to the recurrent connections in LSTM layers, and they showed empirically that it helped the network to converge faster. The usage is simple, each gate function $\mathbf{i},\mathbf{f},\mathbf{o}\in\mathcal{S}_{sig}$ and the transition function $\mathbf{s}\in\mathcal{S}_{tanh}$ computes $$\mathbf{x}^{\left(t\right)},\mathbf{h}^{\left(t-1\right)}\mapsto\sigma\left(BN\left(W\mathbf{h}^{\left(t-1\right)};\gamma_{h},\beta_{h}\right)+BN\left(W\mathbf{x}^{\left(t\right)};\gamma_{x},\beta_{x}\right)+\mathbf{b}\right)$$ when the hyper-parameters $\gamma$ and $\beta$ are shared across different gates. Then the output of the layer is $\mathbf{h}^{\left(t\right)}=o^{\left(t\right)}\circ tanh\left(BN\left(\mathbf{c}^{\left(t\right)};\gamma_{c},\beta_{c}\right)\right)$ . The authors suggested to set $\beta_{x}=\beta_{h}=\mathbf{0}$ because there is already a bias parameter $\mathbf{b}$ and to prevent redundancy. In addition, they said that sharing the internal $BN$ statistics across time degrades performance severely. Therefore, one should use “fresh” $BN$ operators for each time-step with their own internal statistics (but share the $\beta$ and $\gamma$ parameters).