📝 权重初始化-01

lawlite19 · lawlite19 · commit 20661c26556e · 2016-12-21T09:59:11.000+08:00
diff --git a/readme.md b/readme.md
@@ -206,7 +206,7 @@
  - &#25152;&#20197;&#65306;![$$Var(wx) = Var(w)Var(x)$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24Var%28wx%29%20%3D%20Var%28w%29Var%28x%29%24%24)&#65292;&#20195;&#20837;&#19978;&#38754;&#21363;&#21487;&#12290;
 - &#22240;&#20026;&#20043;&#21069;![$${f^'}(s_k^i) \approx 1$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24%7Bf%5E%27%7D%28s_k%5Ei%29%20%5Capprox%201%24%24)&#65292;&#25152;&#20197;&#21487;&#20197;&#24471;&#21040;`&#10105;`&#65306;        
 ![$$V{\rm{ar}}[{{\partial Cost} \over {\partial {s^i}}}] = Var[{{\partial Cost} \over {\partial {s^n}}}]\prod\limits_{j = i}^n {{n_{j + 1}}Var[{W^j}]} $$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24V%7B%5Crm%7Bar%7D%7D%5B%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20%7Bs%5Ei%7D%7D%7D%5D%20%3D%20Var%5B%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20%7Bs%5En%7D%7D%7D%5D%5Cprod%5Climits_%7Bj%20%3D%20i%7D%5En%20%7B%7Bn_%7Bj%20&plus;%201%7D%7DVar%5B%7BW%5Ej%7D%5D%7D%20%24%24)
-- &#23558;`&#10104;&#10105;`&#20195;&#20837;&#21040;&#23545;&#26435;&#37325;w&#20559;&#23548;&#65288;&#21363;&#20026;&#26799;&#24230;&#65289;&#30340;&#26041;&#24046;&#20026;:![$$Var[{{\partial Cost} \over {\partial {w^i}}}] = \prod\limits_{j = 0}^{i - 1} {{n_j}Var[{W^j}]} \prod\limits_{j = i}^{n - 1} {{n_{j + 1}}Var[{W^j}]}  * Var[x]Var[{{\partial Cost} \over {\partial {s^n}}}]$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24Var%5B%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20%7Bw%5Ei%7D%7D%7D%5D%20%3D%20%5Cprod%5Climits_%7Bj%20%3D%200%7D%5E%7Bi%20-%201%7D%20%7B%7Bn_j%7DVar%5B%7BW%5Ej%7D%5D%7D%20%5Cprod%5Climits_%7Bj%20%3D%20i%7D%5E%7Bn%20-%201%7D%20%7B%7Bn_%7Bj%20&plus;%201%7D%7DVar%5B%7BW%5Ej%7D%5D%7D%20*%20Var%5Bx%5DVar%5B%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20%7Bs%5En%7D%7D%7D%5D%24%24)
+- &#23558;`&#10104;&#10105;`&#20195;&#20837;&#21040;&#23545;&#26435;&#37325;w&#20559;&#23548;`&#10103;`&#65288;&#21363;&#20026;&#26799;&#24230;&#65289;&#30340;&#26041;&#24046;&#20026;:![$$Var[{{\partial Cost} \over {\partial {w^i}}}] = \prod\limits_{j = 0}^{i - 1} {{n_j}Var[{W^j}]} \prod\limits_{j = i}^{n - 1} {{n_{j + 1}}Var[{W^j}]}  * Var[x]Var[{{\partial Cost} \over {\partial {s^n}}}]$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24Var%5B%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20%7Bw%5Ei%7D%7D%7D%5D%20%3D%20%5Cprod%5Climits_%7Bj%20%3D%200%7D%5E%7Bi%20-%201%7D%20%7B%7Bn_j%7DVar%5B%7BW%5Ej%7D%5D%7D%20%5Cprod%5Climits_%7Bj%20%3D%20i%7D%5E%7Bn%20-%201%7D%20%7B%7Bn_%7Bj%20&plus;%201%7D%7DVar%5B%7BW%5Ej%7D%5D%7D%20*%20Var%5Bx%5DVar%5B%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20%7Bs%5En%7D%7D%7D%5D%24%24)
 - &#23545;&#20110;&#27491;&#21521;&#20256;&#25773;&#65292;&#25105;&#20204;&#24076;&#26395;&#65306; ![$$\forall (i,j),Var[{z^i}] = Var[{z^j}]$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24%5Cforall%20%28i%2Cj%29%2CVar%5B%7Bz%5Ei%7D%5D%20%3D%20Var%5B%7Bz%5Ej%7D%5D%24%24)
  - &#23545;&#20110;&#21453;&#21521;&#20256;&#25773;&#65292;&#21516;&#26679;&#24076;&#26395;&#65306;![$$\forall (i,j),Var[{{\partial Cost} \over {\partial {s^i}}}] = Var[{{\partial Cost} \over {\partial {s^j}}}]$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24%5Cforall%20%28i%2Cj%29%2CVar%5B%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20%7Bs%5Ei%7D%7D%7D%5D%20%3D%20Var%5B%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20%7Bs%5Ej%7D%7D%7D%5D%24%24)
  - &#20004;&#31181;&#24773;&#20917;&#21487;&#20197;&#36716;&#21270;&#20026;&#65306;               

-Original file line number
+Diff line change
  - 所以：![$$Var(wx) = Var(w)Var(x)$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24Var%28wx%29%20%3D%20Var%28w%29Var%28x%29%24%24)，代入上面即可。
 - 因为之前![$${f^'}(s_k^i) \approx 1$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24%7Bf%5E%27%7D%28s_k%5Ei%29%20%5Capprox%201%24%24)，所以可以得到`❹`：
 ![$$V{\rm{ar}}[{{\partial Cost} \over {\partial {s^i}}}] = Var[{{\partial Cost} \over {\partial {s^n}}}]\prod\limits_{j = i}^n {{n_{j + 1}}Var[{W^j}]} $$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24V%7B%5Crm%7Bar%7D%7D%5B%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20%7Bs%5Ei%7D%7D%7D%5D%20%3D%20Var%5B%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20%7Bs%5En%7D%7D%7D%5D%5Cprod%5Climits_%7Bj%20%3D%20i%7D%5En%20%7B%7Bn_%7Bj%20&plus;%201%7D%7DVar%5B%7BW%5Ej%7D%5D%7D%20%24%24)
 -- 将`❸❹`代入到对权重w偏导（即为梯度）的方差为:![$$Var[{{\partial Cost} \over {\partial {w^i}}}] = \prod\limits_{j = 0}^{i - 1} {{n_j}Var[{W^j}]} \prod\limits_{j = i}^{n - 1} {{n_{j + 1}}Var[{W^j}]}  * Var[x]Var[{{\partial Cost} \over {\partial {s^n}}}]$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24Var%5B%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20%7Bw%5Ei%7D%7D%7D%5D%20%3D%20%5Cprod%5Climits_%7Bj%20%3D%200%7D%5E%7Bi%20-%201%7D%20%7B%7Bn_j%7DVar%5B%7BW%5Ej%7D%5D%7D%20%5Cprod%5Climits_%7Bj%20%3D%20i%7D%5E%7Bn%20-%201%7D%20%7B%7Bn_%7Bj%20&plus;%201%7D%7DVar%5B%7BW%5Ej%7D%5D%7D%20*%20Var%5Bx%5DVar%5B%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20%7Bs%5En%7D%7D%7D%5D%24%24)
 +- 将`❸❹`代入到对权重w偏导`❷`（即为梯度）的方差为:![$$Var[{{\partial Cost} \over {\partial {w^i}}}] = \prod\limits_{j = 0}^{i - 1} {{n_j}Var[{W^j}]} \prod\limits_{j = i}^{n - 1} {{n_{j + 1}}Var[{W^j}]}  * Var[x]Var[{{\partial Cost} \over {\partial {s^n}}}]$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24Var%5B%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20%7Bw%5Ei%7D%7D%7D%5D%20%3D%20%5Cprod%5Climits_%7Bj%20%3D%200%7D%5E%7Bi%20-%201%7D%20%7B%7Bn_j%7DVar%5B%7BW%5Ej%7D%5D%7D%20%5Cprod%5Climits_%7Bj%20%3D%20i%7D%5E%7Bn%20-%201%7D%20%7B%7Bn_%7Bj%20&plus;%201%7D%7DVar%5B%7BW%5Ej%7D%5D%7D%20*%20Var%5Bx%5DVar%5B%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20%7Bs%5En%7D%7D%7D%5D%24%24)
 - 对于正向传播，我们希望： ![$$\forall (i,j),Var[{z^i}] = Var[{z^j}]$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24%5Cforall%20%28i%2Cj%29%2CVar%5B%7Bz%5Ei%7D%5D%20%3D%20Var%5B%7Bz%5Ej%7D%5D%24%24)
  - 对于反向传播，同样希望：![$$\forall (i,j),Var[{{\partial Cost} \over {\partial {s^i}}}] = Var[{{\partial Cost} \over {\partial {s^j}}}]$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24%5Cforall%20%28i%2Cj%29%2CVar%5B%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20%7Bs%5Ei%7D%7D%7D%5D%20%3D%20Var%5B%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20%7Bs%5Ej%7D%7D%7D%5D%24%24)
  - 两种情况可以转化为：