建设网站装配式建筑楼房,wordpress网站数据备份,门户网站开发需求分析报告,网站建设用什么软件好深度学习笔记之循环神经网络——GRU的反向传播过程 引言回顾#xff1a; GRU \text{GRU} GRU的前馈计算过程场景设计 反向传播过程 T \mathcal T T时刻的反向传播过程 T − 1 \mathcal T - 1 T−1时刻的反向传播路径 T − 2 \mathcal T - 2 T−2时刻的反向传播路径 总结 引言 … 深度学习笔记之循环神经网络——GRU的反向传播过程 引言回顾 GRU \text{GRU} GRU的前馈计算过程场景设计 反向传播过程 T \mathcal T T时刻的反向传播过程 T − 1 \mathcal T - 1 T−1时刻的反向传播路径 T − 2 \mathcal T - 2 T−2时刻的反向传播路径 总结 引言
上一节介绍了门控循环单元 ( Gate Recurrent Unit,GRU ) (\text{Gate Recurrent Unit,GRU}) (Gate Recurrent Unit,GRU)本节我们参照 LSTM \text{LSTM} LSTM反向传播的格式观察 GRU \text{GRU} GRU的反向传播过程。
回顾 GRU \text{GRU} GRU的前馈计算过程 GRU \text{GRU} GRU的前馈计算过程表示如下 为了后续的反向传播过程将过程分解的细致一些。其中 Z ~ ( t ) , r ~ ( t ) \widetilde{\mathcal Z}^{(t)},\widetilde{r}^{(t)} Z (t),r (t)分别表示更新门、重置门的线性计算过程。 { Z ~ ( t ) W H ⇒ Z ⋅ h ( t − 1 ) W X ⇒ Z ⋅ x ( t ) b Z Z ( t ) σ ( Z ~ ( t ) ) r ~ ( t ) W H ⇒ r ⋅ h ( t − 1 ) W X ⇒ r ⋅ x ( t ) b r r ( t ) σ ( r ~ ( t ) ) h ~ ( t ) Tanh [ W H ⇒ H ~ ⋅ ( r ( t ) ∗ h ( t − 1 ) ) W X ⇒ H ~ ⋅ x ( t ) b H ~ ] h ( t ) ( 1 − Z ( t ) ) ∗ h ( t − 1 ) Z ( t ) ∗ h ~ ( t ) \begin{cases} \begin{aligned} \widetilde{\mathcal Z}^{(t)} \mathcal W_{\mathcal H \Rightarrow \mathcal Z} \cdot h^{(t-1)} \mathcal W_{\mathcal X \Rightarrow \mathcal Z} \cdot x^{(t)} b_{\mathcal Z} \\ \mathcal Z^{(t)} \sigma(\widetilde{\mathcal Z}^{(t)}) \\ \widetilde{r}^{(t)} \mathcal W_{\mathcal H \Rightarrow r} \cdot h^{(t-1)} \mathcal W_{\mathcal X \Rightarrow r} \cdot x^{(t)} b_{r} \\ r^{(t)} \sigma(\widetilde{r}^{(t)}) \\ \widetilde{h}^{(t)} \text{Tanh} \left[\mathcal W_{\mathcal H \Rightarrow \widetilde{\mathcal H}} \cdot (r^{(t)} * h^{(t-1)}) \mathcal W_{\mathcal X \Rightarrow \widetilde{\mathcal H}} \cdot x^{(t)} b_{\widetilde{\mathcal H}}\right] \\ h^{(t)} (1 -\mathcal Z^{(t)}) * h^{(t-1)} \mathcal Z^{(t)} * \widetilde{h}^{(t)} \end{aligned} \end{cases} ⎩ ⎨ ⎧Z (t)WH⇒Z⋅h(t−1)WX⇒Z⋅x(t)bZZ(t)σ(Z (t))r (t)WH⇒r⋅h(t−1)WX⇒r⋅x(t)brr(t)σ(r (t))h (t)Tanh[WH⇒H ⋅(r(t)∗h(t−1))WX⇒H ⋅x(t)bH ]h(t)(1−Z(t))∗h(t−1)Z(t)∗h (t)
场景设计
上述仅描述的是 GRU \text{GRU} GRU关于序列信息 h ( t ) ( t 1 , 2 , ⋯ , T ) h^{(t)}(t1,2,\cdots,\mathcal T) h(t)(t1,2,⋯,T)的迭代过程。各时刻的输出特征以及损失函数于循环神经网络相同
使用 Softmax \text{Softmax} Softmax激活函数其输出结果作为模型对 t t t时刻的预测结果 { C ( t ) W H ⇒ C ⋅ h ( t ) b h y ^ ( t ) Softmax ( C ( t ) ) \begin{cases} \mathcal C^{(t)} \mathcal W_{\mathcal H \Rightarrow \mathcal C} \cdot h^{(t)} b_{h} \\ \hat y^{(t)} \text{Softmax}(\mathcal C^{(t)}) \end{cases} {C(t)WH⇒C⋅h(t)bhy^(t)Softmax(C(t))关于 t t t时刻预测结果 y ^ ( t ) \hat y^{(t)} y^(t)与真实分布 y ( t ) y^{(t)} y(t)之间的偏差信息使用交叉熵 ( CrossEntropy ) (\text{CrossEntropy}) (CrossEntropy)进行表示 其中 n Y n_{\mathcal Y} nY表示预测/真实分布的维数。 L ( t ) L [ y ^ ( t ) , y ( t ) ] − ∑ j 1 n Y y j ( t ) log [ y ^ j ( t ) ] \mathcal L^{(t)} \mathcal L \left[\hat y^{(t)},y^{(t)}\right] - \sum_{j1}^{n_\mathcal Y} y_j^{(t)} \log \left[\hat y_j^{(t)}\right] L(t)L[y^(t),y(t)]−j1∑nYyj(t)log[y^j(t)]所有时刻交叉熵结果的累加和构成完整的损失函数 L \mathcal L L L ∑ t 1 T L ( t ) ∑ t 1 T L [ y ^ ( t ) , y ( t ) ] \begin{aligned} \mathcal L \sum_{t1}^{\mathcal T} \mathcal L^{(t)}\\ \sum_{t1}^{\mathcal T} \mathcal L \left[\hat y^{(t)},y^{(t)}\right] \end{aligned} Lt1∑TL(t)t1∑TL[y^(t),y(t)]
反向传播过程 T \mathcal T T时刻的反向传播过程
以 T \mathcal T T时刻重置门 ∂ L ∂ W h ( T ) ⇒ Z ( T ) \begin{aligned}\frac{\partial \mathcal L}{\partial \mathcal W_{\mathcal h^{(\mathcal T)} \Rightarrow \mathcal Z^{(\mathcal T)}}}\end{aligned} ∂Wh(T)⇒Z(T)∂L的反向传播为例
计算梯度 ∂ L ∂ L ( T ) \begin{aligned}\frac{\partial \mathcal L}{\partial \mathcal L^{(\mathcal T)}}\end{aligned} ∂L(T)∂L 其中仅有 L ( T ) \mathcal L^{(\mathcal T)} L(T)一项存在梯度其余项均视作常数。 ∂ L ∂ L ( T ) ∂ ∂ L ( T ) [ ∑ t 1 T L ( t ) ] 0 0 ⋯ 1 1 \frac{\partial \mathcal L}{\partial \mathcal L^{(\mathcal T)}} \frac{\partial}{\partial \mathcal L^{(\mathcal T)}} \left[\sum_{t1}^{\mathcal T} \mathcal L^{(t)}\right] 0 0 \cdots 1 1 ∂L(T)∂L∂L(T)∂[t1∑TL(t)]00⋯11计算梯度 ∂ L ( T ) ∂ C ( T ) \begin{aligned}\frac{\partial \mathcal L^{(\mathcal T)}}{\partial \mathcal C^{(\mathcal T)}}\end{aligned} ∂C(T)∂L(T) 关于 Softmax \text{Softmax} Softmax激活函数与交叉熵组合的梯度描述见循环神经网络—— Softmax \text{Softmax} Softmax函数的反向传播过程一节这里不再赘述。 { L ( T ) − ∑ j 1 n Y y j ( T ) log [ y ^ j ( T ) ] y ^ ( T ) Softmax [ C ( T ) ] ⇒ ∂ L ( T ) ∂ C ( T ) y ^ ( T ) − y ( T ) \begin{aligned} \begin{cases} \mathcal L^{(\mathcal T)} -\sum_{j1}^{n_{\mathcal Y}}y_j^{(\mathcal T)} \log \left[\hat y_j^{(\mathcal T)}\right] \\ \hat y^{(\mathcal T)} \text{Softmax}[\mathcal C^{(\mathcal T)}] \end{cases} \\ \Rightarrow \frac{\partial \mathcal L^{(\mathcal T)}}{\partial \mathcal C^{(\mathcal T)}} \hat y^{(\mathcal T)} - y^{(\mathcal T)} \end{aligned} {L(T)−∑j1nYyj(T)log[y^j(T)]y^(T)Softmax[C(T)]⇒∂C(T)∂L(T)y^(T)−y(T)继续计算梯度 ∂ C ( T ) ∂ h ( T ) \begin{aligned}\frac{\partial \mathcal C^{(\mathcal T)}}{\partial h^{(\mathcal T)}}\end{aligned} ∂h(T)∂C(T) ∂ C ( T ) ∂ h ( T ) ∂ ∂ h ( T ) [ W H ⇒ C ⋅ h ( T ) b h ] W H ⇒ C \frac{\partial \mathcal C^{(\mathcal T)}}{\partial h^{(\mathcal T)}} \frac{\partial}{\partial h^{(\mathcal T)}} \left[\mathcal W_{\mathcal H \Rightarrow \mathcal C} \cdot h^{(\mathcal T)} b_{h}\right] \mathcal W_{\mathcal H \Rightarrow \mathcal C} ∂h(T)∂C(T)∂h(T)∂[WH⇒C⋅h(T)bh]WH⇒C
至此关于梯度 ∂ L ∂ h ( T ) \begin{aligned}\frac{\partial \mathcal L}{\partial h^{(\mathcal T)}}\end{aligned} ∂h(T)∂L可表示为 ∂ L ∂ h ( T ) ∂ L ∂ L ( T ) ⋅ ∂ L ( T ) ∂ C ( T ) ⋅ ∂ C ( T ) ∂ h ( T ) 1 ⋅ [ W H ⇒ C ] T ⋅ ( y ^ ( T ) − y ( T ) ) \begin{aligned} \frac{\partial \mathcal L}{\partial h^{(\mathcal T)}} \frac{\partial \mathcal L}{\partial \mathcal L^{(\mathcal T)}} \cdot \frac{\partial \mathcal L^{(\mathcal T)}}{\partial \mathcal C^{(\mathcal T)}} \cdot \frac{\partial \mathcal C^{(\mathcal T)}}{\partial h^{(\mathcal T)}} \\ 1 \cdot \left[\mathcal W_{\mathcal H \Rightarrow \mathcal C}\right]^T \cdot (\hat y^{(\mathcal T)} - y^{(\mathcal T)}) \end{aligned} ∂h(T)∂L∂L(T)∂L⋅∂C(T)∂L(T)⋅∂h(T)∂C(T)1⋅[WH⇒C]T⋅(y^(T)−y(T)) 观察从 h ( T ) h^{(\mathcal T)} h(T)开始从 h ( T ) ⇒ W h ( T ) ⇒ Z ( T ) h^{(\mathcal T)} \Rightarrow \mathcal W_{h^{(\mathcal T)} \Rightarrow \mathcal Z^{(\mathcal T)}} h(T)⇒Wh(T)⇒Z(T)的传播路径都有哪些。 只有唯一一条其前馈计算路径表示为 { h ( t ) ( 1 − Z ( t ) ) ∗ h ( t − 1 ) Z ( t ) ∗ h ~ ( t ) Z ( t ) σ ( Z ~ ( t ) ) Z ~ ( t ) W H ⇒ Z ⋅ h ( t − 1 ) W X ⇒ Z ⋅ x ( t ) b Z \begin{cases} \begin{aligned} h^{(t)} (1 -\mathcal Z^{(t)}) * h^{(t-1)} \mathcal Z^{(t)} * \widetilde{h}^{(t)} \\ \mathcal Z^{(t)} \sigma(\widetilde{\mathcal Z}^{(t)}) \\ \widetilde{\mathcal Z}^{(t)} \mathcal W_{\mathcal H \Rightarrow \mathcal Z} \cdot h^{(t-1)} \mathcal W_{\mathcal X \Rightarrow \mathcal Z} \cdot x^{(t)} b_{\mathcal Z} \end{aligned} \end{cases} ⎩ ⎨ ⎧h(t)(1−Z(t))∗h(t−1)Z(t)∗h (t)Z(t)σ(Z (t))Z (t)WH⇒Z⋅h(t−1)WX⇒Z⋅x(t)bZ 对应反向传播结果表示为 ∂ h ( T ) ∂ W h ( T ) ⇒ Z ( T ) ∂ h ( T ) ∂ Z ( T ) ⋅ ∂ Z ( T ) ∂ Z ~ ( T ) ⋅ ∂ Z ~ ( T ) ∂ W h ( T ) ⇒ Z ( T ) [ h ~ ( T ) − h ( T − 1 ) ] ⋅ [ Sigmoid ( Z ~ ( T ) ) ] ′ ⋅ h ( T − 1 ) \begin{aligned} \frac{\partial h^{(\mathcal T)}}{\partial \mathcal W_{h^{(\mathcal T)} \Rightarrow \mathcal Z^{(\mathcal T)}}} \frac{\partial h^{(\mathcal T)}}{\partial \mathcal Z^{(\mathcal T)}} \cdot \frac{\partial \mathcal Z^{(\mathcal T)}}{\partial \widetilde{\mathcal Z}^{(\mathcal T)}} \cdot \frac{\partial \widetilde{\mathcal Z}^{(\mathcal T)}}{\partial \mathcal W_{h^{(\mathcal T)} \Rightarrow \mathcal Z^{(\mathcal T)}}} \\ \left[\widetilde{h}^{(\mathcal T)} - h^{(\mathcal T - 1)}\right] \cdot \left[\text{Sigmoid}(\widetilde{\mathcal Z}^{(\mathcal T)})\right] \cdot h^{(\mathcal T - 1)} \end{aligned} ∂Wh(T)⇒Z(T)∂h(T)∂Z(T)∂h(T)⋅∂Z (T)∂Z(T)⋅∂Wh(T)⇒Z(T)∂Z (T)[h (T)−h(T−1)]⋅[Sigmoid(Z (T))]′⋅h(T−1) 最终关于 ∂ L ∂ W h ( T ) ⇒ Z ( T ) \begin{aligned}\frac{\partial \mathcal L}{\partial \mathcal W_{\mathcal h^{(\mathcal T)} \Rightarrow \mathcal Z^{(\mathcal T)}}}\end{aligned} ∂Wh(T)⇒Z(T)∂L的反向传播结果为 这里更主要的是描述它的反向传播路径它的具体展开在后续不再赘述。 ∂ L ∂ W h ( T ) ⇒ Z ( T ) ∂ L ∂ h ( T ) ⋅ ∂ h ( T ) ∂ W h ( T ) ⇒ Z ( T ) { [ W H ⇒ C ] T ⋅ ( y ^ ( T ) − y ( T ) ) } ⋅ { [ h ~ ( T ) − h ( T − 1 ) ] ⋅ [ Sigmoid ( Z ~ ( T ) ) ] ′ ⋅ h ( T − 1 ) } \begin{aligned} \begin{aligned}\frac{\partial \mathcal L}{\partial \mathcal W_{\mathcal h^{(\mathcal T)} \Rightarrow \mathcal Z^{(\mathcal T)}}}\end{aligned} \frac{\partial \mathcal L}{\partial h^{(\mathcal T)}} \cdot \frac{\partial h^{(\mathcal T)}}{\partial \mathcal W_{h^{(\mathcal T)}\Rightarrow \mathcal Z^{(\mathcal T)}}} \\ \left\{\left[\mathcal W_{\mathcal H \Rightarrow \mathcal C}\right]^T \cdot (\hat y^{(\mathcal T)} - y^{(\mathcal T)})\right\} \cdot \left\{ \left[\widetilde{h}^{(\mathcal T)} - h^{(\mathcal T - 1)}\right] \cdot \left[\text{Sigmoid}(\widetilde{\mathcal Z}^{(\mathcal T)})\right] \cdot h^{(\mathcal T - 1)}\right\} \end{aligned} ∂Wh(T)⇒Z(T)∂L∂h(T)∂L⋅∂Wh(T)⇒Z(T)∂h(T){[WH⇒C]T⋅(y^(T)−y(T))}⋅{[h (T)−h(T−1)]⋅[Sigmoid(Z (T))]′⋅h(T−1)} T − 1 \mathcal T - 1 T−1时刻的反向传播路径
关于 T − 1 \mathcal T - 1 T−1时刻的重置门梯度 ∂ L ∂ W h ( T − 1 ) ⇒ Z ( T − 1 ) \begin{aligned} \frac{\partial \mathcal L}{\partial \mathcal W_{h^{(\mathcal T - 1)} \Rightarrow \mathcal Z^{(\mathcal T - 1)}}} \end{aligned} ∂Wh(T−1)⇒Z(T−1)∂L它的路径主要包含两大类
第一类路径同 T \mathcal T T时刻路径从对应的 L ( T − 1 ) \mathcal L^{(\mathcal T - 1)} L(T−1)直接传至 W h ( T − 1 ) ⇒ Z ( T − 1 ) \mathcal W_{h^{(\mathcal T - 1)}\Rightarrow \mathcal Z^{(\mathcal T - 1)}} Wh(T−1)⇒Z(T−1) 该路径与上述 T \mathcal T T时刻的路径类型相同将对应的上标 T \mathcal T T改为 T − 1 \mathcal T - 1 T−1即可。 ∂ L ( T − 1 ) ∂ W h ( T − 1 ) ⇒ Z ( T − 1 ) ∂ L ( T − 1 ) ∂ h ( T − 1 ) ⋅ ∂ h ( T − 1 ) ∂ W h ( T − 1 ) ⇒ Z ( T − 1 ) \begin{aligned} \frac{\partial \mathcal L^{(\mathcal T - 1)}}{\partial \mathcal W_{h^{(\mathcal T - 1)} \Rightarrow \mathcal Z^{(\mathcal T - 1)}}} \frac{\partial \mathcal L^{(\mathcal T - 1)}}{\partial h^{(\mathcal T - 1)}} \cdot \frac{\partial h^{(\mathcal T - 1)}}{\partial \mathcal W_{h^{(\mathcal T - 1)} \Rightarrow \mathcal Z^{(\mathcal T - 1)}}} \end{aligned} ∂Wh(T−1)⇒Z(T−1)∂L(T−1)∂h(T−1)∂L(T−1)⋅∂Wh(T−1)⇒Z(T−1)∂h(T−1) 第二类路径重新观察 W h ( T − 1 ) ⇒ Z ( T − 1 ) \mathcal W_{h^{(\mathcal T - 1)}\Rightarrow \mathcal Z^{(\mathcal T - 1)}} Wh(T−1)⇒Z(T−1)只会出现在 Z ( T − 1 ) \mathcal Z^{(\mathcal T - 1)} Z(T−1)中并且 Z ( T − 1 ) \mathcal Z^{(\mathcal T - 1)} Z(T−1)只会出现在 h ( T − 1 ) h^{(\mathcal T - 1)} h(T−1)中。因此仅需要找出与 h ( T − 1 ) h^{(\mathcal T - 1)} h(T−1)相关的所有路径即可最终都可以使用 ∂ h ( T − 1 ) ∂ W h ( T − 1 ) ⇒ Z ( T − 1 ) \begin{aligned}\frac{\partial h^{(\mathcal T - 1)}}{\partial \mathcal W_{h^{(\mathcal T - 1)} \Rightarrow \mathcal Z^{(\mathcal T - 1)}}}\end{aligned} ∂Wh(T−1)⇒Z(T−1)∂h(T−1)将梯度传递给 W h ( T − 1 ) ⇒ Z ( T − 1 ) \mathcal W_{h^{(\mathcal T - 1)} \Rightarrow \mathcal Z^{(\mathcal T - 1)}} Wh(T−1)⇒Z(T−1)。 其中‘第一类路径’就是其中一种情况。只不过它是从当前 T − 1 \mathcal T - 1 T−1时刻直接传递得到的梯度结果。而第二类路径我们关注从 T \mathcal T T时刻传递产生的梯度信息。 从 T ⇒ T − 1 \mathcal T \Rightarrow \mathcal T - 1 T⇒T−1时刻中关于 h ( T − 1 ) h^{(\mathcal T - 1)} h(T−1)的梯度路径一共包含 4 4 4条
第一条通过 T \mathcal T T时刻 h ( T ) h^{(\mathcal T)} h(T)中的 h ( T − 1 ) h^{(\mathcal T - 1)} h(T−1)进行传递。 { Forword : h ( T ) ( 1 − Z ( T ) ) ∗ h ( T − 1 ) Z ( T ) ∗ h ~ ( T ) Backward : ∂ L ( T ) ∂ W h ( T − 1 ) ⇒ Z ( T − 1 ) ⇒ ∂ L ( T ) ∂ h ( T ) ⋅ ∂ h ( T ) ∂ h ( T − 1 ) ⋅ ∂ h ( T − 1 ) ∂ W h ( T − 1 ) ⇒ Z ( T − 1 ) \begin{cases} \text{Forword : } h^{(\mathcal T)} (1 -\mathcal Z^{(\mathcal T)}) * h^{(\mathcal T -1)} \mathcal Z^{(\mathcal T)} * \widetilde{h}^{(\mathcal T)} \\ \quad \\ \text{Backward : }\begin{aligned} \frac{\partial \mathcal L^{(\mathcal T)}}{\partial \mathcal W_{h^{(\mathcal T - 1)} \Rightarrow \mathcal Z^{(\mathcal T - 1)}}} \Rightarrow \frac{\partial \mathcal L^{(\mathcal T)}}{\partial h^{(\mathcal T)}} \cdot \frac{\partial h^{(\mathcal T)}}{\partial h^{(\mathcal T - 1)}} \cdot \frac{\partial h^{(\mathcal T - 1)}}{\partial \mathcal W_{h^{(\mathcal T - 1)} \Rightarrow \mathcal Z^{(\mathcal T - 1)}}} \end{aligned} \end{cases} ⎩ ⎨ ⎧Forword : h(T)(1−Z(T))∗h(T−1)Z(T)∗h (T)Backward : ∂Wh(T−1)⇒Z(T−1)∂L(T)⇒∂h(T)∂L(T)⋅∂h(T−1)∂h(T)⋅∂Wh(T−1)⇒Z(T−1)∂h(T−1)第二条通过 T \mathcal T T时刻 h ( T ) h^{(\mathcal T)} h(T)中的 Z ( T ) \mathcal Z^{(\mathcal T)} Z(T)向 h ( T − 1 ) h^{(\mathcal T-1)} h(T−1)进行传递。 { Forward : { h ( T ) ( 1 − Z ( T ) ) ∗ h ( T − 1 ) Z ( T ) ∗ h ~ ( T ) Z ( T ) σ [ W H ⇒ Z ⋅ h ( T − 1 ) W X ⇒ Z ⋅ x ( T ) b Z ] Backward : ∂ L ( T ) ∂ W h ( T − 1 ) ⇒ Z ( T − 1 ) ⇒ ∂ L ( T ) ∂ h ( T ) ⋅ ∂ h ( T ) ∂ Z ( T ) ⋅ ∂ Z ( T ) ∂ h ( T − 1 ) ⋅ ∂ h ( T − 1 ) ∂ W h ( T − 1 ) ⇒ Z ( T − 1 ) \begin{cases} \text{Forward : } \begin{cases} h^{(\mathcal T)} (1 -\mathcal Z^{(\mathcal T)}) * h^{(\mathcal T -1)} \mathcal Z^{(\mathcal T)} * \widetilde{h}^{(\mathcal T)} \\ \mathcal Z^{(\mathcal T)} \sigma \left[\mathcal W_{\mathcal H \Rightarrow \mathcal Z} \cdot h^{(\mathcal T -1)} \mathcal W_{\mathcal X \Rightarrow \mathcal Z} \cdot x^{(\mathcal T)} b_{\mathcal Z}\right] \end{cases} \quad \\ \text{Backward : } \begin{aligned} \frac{\partial \mathcal L^{(\mathcal T)}}{\partial \mathcal W_{h^{(\mathcal T - 1)} \Rightarrow \mathcal Z^{(\mathcal T - 1)}}} \Rightarrow \frac{\partial \mathcal L^{(\mathcal T)}}{\partial h^{(\mathcal T)}} \cdot \frac{\partial h^{(\mathcal T)}}{\partial \mathcal Z^{(\mathcal T)}} \cdot \frac{\partial \mathcal Z^{(\mathcal T)}}{\partial h^{(\mathcal T - 1)}} \cdot \frac{\partial h^{(\mathcal T - 1)}}{\partial \mathcal W_{h^{(\mathcal T - 1)} \Rightarrow \mathcal Z^{(\mathcal T - 1)}}} \end{aligned} \end{cases} ⎩ ⎨ ⎧Forward : {h(T)(1−Z(T))∗h(T−1)Z(T)∗h (T)Z(T)σ[WH⇒Z⋅h(T−1)WX⇒Z⋅x(T)bZ]Backward : ∂Wh(T−1)⇒Z(T−1)∂L(T)⇒∂h(T)∂L(T)⋅∂Z(T)∂h(T)⋅∂h(T−1)∂Z(T)⋅∂Wh(T−1)⇒Z(T−1)∂h(T−1)第三条通过 T \mathcal T T时刻 h ( T ) h^{(\mathcal T)} h(T)中的 h ~ ( T ) \widetilde{h}^{(\mathcal T)} h (T)向 h ( T − 1 ) h^{(\mathcal T - 1)} h(T−1)进行传递。 { Forward : { h ( T ) ( 1 − Z ( T ) ) ∗ h ( T − 1 ) Z ( T ) ∗ h ~ ( T ) h ~ ( T ) Tanh [ W H ⇒ H ~ ⋅ ( r ( T ) ∗ h ( T − 1 ) ) W X ⇒ H ~ ⋅ x ( T ) b H ~ ] Backward : ∂ L ( T ) ∂ W h ( T − 1 ) ⇒ Z ( T − 1 ) ⇒ ∂ L ( T ) ∂ h ( T ) ⋅ ∂ h ( T ) ∂ h ~ ( T ) ⋅ h ~ ( T ) ∂ h ( T − 1 ) ⋅ ∂ h ( T − 1 ) ∂ W h ( T − 1 ) ⇒ Z ( T − 1 ) \begin{cases} \text{Forward : } \begin{cases} h^{(\mathcal T)} (1 -\mathcal Z^{(\mathcal T)}) * h^{(\mathcal T -1)} \mathcal Z^{(\mathcal T)} * \widetilde{h}^{(\mathcal T)} \\ \widetilde{h}^{(\mathcal T)} \text{Tanh} \left[\mathcal W_{\mathcal H \Rightarrow \widetilde{\mathcal H}} \cdot (r^{(\mathcal T)} * h^{(\mathcal T -1)}) \mathcal W_{\mathcal X \Rightarrow \widetilde{\mathcal H}} \cdot x^{(\mathcal T)} b_{\widetilde{\mathcal H}}\right] \end{cases}\\ \quad \\ \text{Backward : } \begin{aligned} \frac{\partial \mathcal L^{(\mathcal T)}}{\partial \mathcal W_{h^{(\mathcal T - 1)} \Rightarrow \mathcal Z^{(\mathcal T - 1)}}} \Rightarrow \frac{\partial \mathcal L^{(\mathcal T)}}{\partial h^{(\mathcal T)}} \cdot \frac{\partial h^{(\mathcal T)}}{\partial \widetilde{h}^{(\mathcal T)}} \cdot \frac{\widetilde{h}^{(\mathcal T)}}{\partial h^{(\mathcal T - 1)}} \cdot \frac{\partial h^{(\mathcal T - 1)}}{\partial \mathcal W_{h^{(\mathcal T - 1)} \Rightarrow \mathcal Z^{(\mathcal T - 1)}}} \end{aligned} \end{cases} ⎩ ⎨ ⎧Forward : {h(T)(1−Z(T))∗h(T−1)Z(T)∗h (T)h (T)Tanh[WH⇒H ⋅(r(T)∗h(T−1))WX⇒H ⋅x(T)bH ]Backward : ∂Wh(T−1)⇒Z(T−1)∂L(T)⇒∂h(T)∂L(T)⋅∂h (T)∂h(T)⋅∂h(T−1)h (T)⋅∂Wh(T−1)⇒Z(T−1)∂h(T−1)第四条与第三条路径类似只不过从 h ~ ( T ) \widetilde{h}^{(\mathcal T)} h (T)中的 r ( T ) r^{(\mathcal T)} r(T)向 h ( T − 1 ) h^{(\mathcal T - 1)} h(T−1)进行传递。 { Forward : { h ( T ) ( 1 − Z ( T ) ) ∗ h ( T − 1 ) Z ( T ) ∗ h ~ ( T ) h ~ ( T ) Tanh [ W H ⇒ H ~ ⋅ ( r ( T ) ∗ h ( T − 1 ) ) W X ⇒ H ~ ⋅ x ( T ) b H ~ ] r ~ ( T ) W H ⇒ r ⋅ h ( T − 1 ) W X ⇒ r ⋅ x ( T ) b r Backward : ∂ L ( T ) ∂ W h ( T − 1 ) ⇒ Z ( T − 1 ) ⇒ ∂ L ( T ) ∂ h ( T ) ⋅ ∂ h ( T ) ∂ h ~ ( T ) ⋅ ∂ h ~ ( T ) ∂ r ( T ) ⋅ ∂ r ( T ) ∂ h ( T − 1 ) ⋅ ∂ h ( T − 1 ) ∂ W h ( T − 1 ) ⇒ Z ( T − 1 ) \begin{cases} \text{Forward : } \begin{cases} h^{(\mathcal T)} (1 -\mathcal Z^{(\mathcal T)}) * h^{(\mathcal T -1)} \mathcal Z^{(\mathcal T)} * \widetilde{h}^{(\mathcal T)} \\ \widetilde{h}^{(\mathcal T)} \text{Tanh} \left[\mathcal W_{\mathcal H \Rightarrow \widetilde{\mathcal H}} \cdot (r^{(\mathcal T)} * h^{(\mathcal T -1)}) \mathcal W_{\mathcal X \Rightarrow \widetilde{\mathcal H}} \cdot x^{(\mathcal T)} b_{\widetilde{\mathcal H}}\right] \\ \widetilde{r}^{(\mathcal T)} \mathcal W_{\mathcal H \Rightarrow r} \cdot h^{(\mathcal T -1)} \mathcal W_{\mathcal X \Rightarrow r} \cdot x^{(\mathcal T)} b_{r} \end{cases}\\ \quad \\ \text{Backward : } \begin{aligned} \frac{\partial \mathcal L^{(\mathcal T)}}{\partial \mathcal W_{h^{(\mathcal T - 1)} \Rightarrow \mathcal Z^{(\mathcal T - 1)}}} \Rightarrow \frac{\partial \mathcal L^{(\mathcal T)}}{\partial h^{(\mathcal T)}} \cdot \frac{\partial h^{(\mathcal T)}}{\partial \widetilde{h}^{(\mathcal T)}} \cdot \frac{\partial \widetilde{h}^{(\mathcal T)}}{\partial r^{(\mathcal T)}} \cdot \frac{\partial r^{(\mathcal T)}}{\partial h^{(\mathcal T - 1)}} \cdot \frac{\partial h^{(\mathcal T - 1)}}{\partial \mathcal W_{h^{(\mathcal T - 1)} \Rightarrow \mathcal Z^{(\mathcal T - 1)}}} \end{aligned} \end{cases} ⎩ ⎨ ⎧Forward : ⎩ ⎨ ⎧h(T)(1−Z(T))∗h(T−1)Z(T)∗h (T)h (T)Tanh[WH⇒H ⋅(r(T)∗h(T−1))WX⇒H ⋅x(T)bH ]r (T)WH⇒r⋅h(T−1)WX⇒r⋅x(T)brBackward : ∂Wh(T−1)⇒Z(T−1)∂L(T)⇒∂h(T)∂L(T)⋅∂h (T)∂h(T)⋅∂r(T)∂h (T)⋅∂h(T−1)∂r(T)⋅∂Wh(T−1)⇒Z(T−1)∂h(T−1)
至此 T ⇒ T − 1 \mathcal T \Rightarrow \mathcal T - 1 T⇒T−1时刻的 5 5 5条路径已全部找全。其中 1 1 1条是 T − 1 \mathcal T - 1 T−1时刻自身路径剩余 4 4 4条均是 T ⇒ T − 1 \mathcal T \Rightarrow \mathcal T - 1 T⇒T−1的传播路径。 T − 2 \mathcal T - 2 T−2时刻的反向传播路径
再往下走一步观察它路径传播数量的规律
第 1 1 1条依然是 L ( T − 2 ) \mathcal L^{(\mathcal T - 2)} L(T−2)向 W h ( T − 2 ) ⇒ Z ( T − 2 ) \mathcal W_{h^{(\mathcal T - 2)} \Rightarrow \mathcal Z^{(\mathcal T - 2)}} Wh(T−2)⇒Z(T−2)直接传递的梯度 ∂ L ( T − 2 ) ∂ W h ( T − 2 ) ⇒ Z ( T − 2 ) ∂ L ( T − 2 ) ∂ h ( T − 2 ) ⋅ ∂ h ( T − 2 ) ∂ W h ( T − 2 ) ⇒ Z ( T − 2 ) \begin{aligned} \frac{\partial \mathcal L^{(\mathcal T - 2)}}{\partial \mathcal W_{h^{(\mathcal T - 2)} \Rightarrow \mathcal Z^{(\mathcal T - 2)}}} \frac{\partial \mathcal L^{(\mathcal T - 2)}}{\partial h^{(\mathcal T - 2)}} \cdot \frac{\partial h^{(\mathcal T - 2)}}{\partial \mathcal W_{h^{(\mathcal T - 2)} \Rightarrow \mathcal Z^{(\mathcal T - 2)}}} \end{aligned} ∂Wh(T−2)⇒Z(T−2)∂L(T−2)∂h(T−2)∂L(T−2)⋅∂Wh(T−2)⇒Z(T−2)∂h(T−2)存在 4 4 4条是从 L ( T − 1 ) \mathcal L^{(\mathcal T - 1)} L(T−1)开始从 T − 1 ⇒ T − 2 \mathcal T - 1 \Rightarrow \mathcal T - 2 T−1⇒T−2时刻传递的路径 ∂ L ( T − 1 ) ∂ W h ( T − 2 ) ⇒ Z ( T − 2 ) { ∂ L ( T − 1 ) ∂ h ( T − 1 ) ⋅ ∂ h ( T − 1 ) ∂ h ( T − 2 ) ⋅ ∂ h ( T − 2 ) ∂ W h ( T − 2 ) ⇒ Z ( T − 2 ) ∂ L ( T − 1 ) ∂ h ( T − 1 ) ⋅ ∂ h ( T − 1 ) ∂ Z ( T − 1 ) ⋅ ∂ Z ( T − 1 ) ∂ h ( T − 2 ) ⋅ ∂ h ( T − 2 ) ∂ W h ( T − 2 ) ⇒ Z ( T − 2 ) ∂ L ( T − 1 ) ∂ h ( T − 1 ) ⋅ ∂ h ( T − 1 ) ∂ h ~ ( T − 1 ) ⋅ ∂ h ~ ( T − 1 ) ∂ h ( T − 2 ) ⋅ ∂ h ( T − 2 ) ∂ W h ( T − 2 ) ⇒ Z ( T − 2 ) ∂ L ( T − 1 ) ∂ h ( T − 1 ) ⋅ ∂ h ( T − 1 ) ∂ h ~ ( T − 1 ) ⋅ ∂ h ~ ( T − 1 ) ∂ r ( T − 1 ) ⋅ ∂ r ( T − 1 ) ∂ h ( T − 2 ) ⋅ ∂ h ( T − 2 ) ∂ W h ( T − 2 ) ⇒ Z ( T − 2 ) \frac{\partial \mathcal L^{(\mathcal T - 1)}}{\partial \mathcal W_{h^{(\mathcal T - 2)} \Rightarrow \mathcal Z^{(\mathcal T - 2)}}} \begin{cases} \begin{aligned} \frac{\partial \mathcal L^{(\mathcal T - 1)}}{\partial h^{(\mathcal T - 1)}} \cdot \frac{\partial h^{(\mathcal T - 1)}}{\partial h^{(\mathcal T - 2)}} \cdot \frac{\partial h^{(\mathcal T - 2)}}{\partial \mathcal W_{h^{(\mathcal T - 2)} \Rightarrow \mathcal Z^{(\mathcal T - 2)}}} \\ \frac{\partial \mathcal L^{(\mathcal T - 1)}}{\partial h^{(\mathcal T - 1)}} \cdot \frac{\partial h^{(\mathcal T - 1)}}{\partial \mathcal Z^{(\mathcal T - 1)}} \cdot \frac{\partial \mathcal Z^{(\mathcal T - 1)}}{\partial h^{(\mathcal T - 2)}} \cdot \frac{\partial h^{(\mathcal T - 2)}}{\partial \mathcal W_{h^{(\mathcal T - 2)} \Rightarrow \mathcal Z^{(\mathcal T - 2)}}}\\ \frac{\partial \mathcal L^{(\mathcal T - 1)}}{\partial h^{(\mathcal T - 1)}} \cdot \frac{\partial h^{(\mathcal T - 1)}}{\partial \widetilde{h}^{(\mathcal T - 1)}} \cdot \frac{\partial \widetilde{h}^{(\mathcal T - 1)}}{\partial h^{(\mathcal T - 2)}} \cdot \frac{\partial h^{(\mathcal T - 2)}}{\partial \mathcal W_{h^{(\mathcal T - 2)} \Rightarrow \mathcal Z^{(\mathcal T - 2)}}} \\ \frac{\partial \mathcal L^{(\mathcal T - 1)}}{\partial h^{(\mathcal T - 1)}} \cdot \frac{\partial h^{(\mathcal T - 1)}}{\partial \widetilde{h}^{(\mathcal T - 1)}} \cdot \frac{\partial \widetilde{h}^{(\mathcal T - 1)}}{\partial r^{(\mathcal T - 1)}} \cdot \frac{\partial r^{(\mathcal T - 1)}}{\partial h^{(\mathcal T - 2)}} \cdot \frac{\partial h^{(\mathcal T - 2)}}{\partial \mathcal W_{h^{(\mathcal T - 2)} \Rightarrow \mathcal Z^{(\mathcal T - 2)}}} \end{aligned} \end{cases} ∂Wh(T−2)⇒Z(T−2)∂L(T−1)⎩ ⎨ ⎧∂h(T−1)∂L(T−1)⋅∂h(T−2)∂h(T−1)⋅∂Wh(T−2)⇒Z(T−2)∂h(T−2)∂h(T−1)∂L(T−1)⋅∂Z(T−1)∂h(T−1)⋅∂h(T−2)∂Z(T−1)⋅∂Wh(T−2)⇒Z(T−2)∂h(T−2)∂h(T−1)∂L(T−1)⋅∂h (T−1)∂h(T−1)⋅∂h(T−2)∂h (T−1)⋅∂Wh(T−2)⇒Z(T−2)∂h(T−2)∂h(T−1)∂L(T−1)⋅∂h (T−1)∂h(T−1)⋅∂r(T−1)∂h (T−1)⋅∂h(T−2)∂r(T−1)⋅∂Wh(T−2)⇒Z(T−2)∂h(T−2)存在 4 × 4 4 \times 4 4×4条是从 L ( T ) \mathcal L^{(\mathcal T)} L(T)开始从 T ⇒ T − 2 \mathcal T \Rightarrow \mathcal T- 2 T⇒T−2时刻传递的路径。 这个就不写了太墨迹了。
这仅仅是 T ⇒ T − 2 \mathcal T \Rightarrow \mathcal T - 2 T⇒T−2时刻的路径数量 T − 3 \mathcal T - 3 T−3时刻关于 L ( T ) \mathcal L^{(\mathcal T)} L(T)相关的梯度路径有 4 × 4 × 4 64 4 \times 4 \times 4 64 4×4×464条以此类推。
总结
和 LSTM \text{LSTM} LSTM的反向传播路径相比 LSTM \text{LSTM} LSTM仅仅从 T ⇒ T − 2 \mathcal T \Rightarrow \mathcal T - 2 T⇒T−2时刻传递的路径就有 24 24 24条而 GRU \text{GRU} GRU仅有 16 16 16条相比之下极大地减小了反向传播路径的数量 降低了时间、空间复杂度;
其次 GRU \text{GRU} GRU相比 LSTM \text{LSTM} LSTM减少了模型参数的更新数量降低了过拟合 ( OverFitting ) (\text{OverFitting}) (OverFitting)的风险。
它的抑制梯度消失原理与 LSTM \text{LSTM} LSTM思想相同。首先随着反向传播深度的加深相关梯度路径依然会呈指数级别增长但规模明显小于 LSTM \text{LSTM} LSTM并且其梯度计算过程依然有更新门、重置门自身参与梯度运算从而调节各梯度分量的配置情况。
相关参考 GRU循环神经网络 —— LSTM的轻量级版本
