📝 权重初始化-01

lawlite19 · lawlite19 · commit 62b0b830b7e5 · 2016-12-21T00:29:32.000+08:00
diff --git a/readme.md b/readme.md
@@ -186,15 +186,21 @@
 ![$${z^{i + 1}} = f({s^i})$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24%7Bz%5E%7Bi%20&plus;%201%7D%7D%20%3D%20f%28%7Bs%5Ei%7D%29%24%24)&#65292;
 - &#26681;&#25454;BP&#21453;&#21521;&#20256;&#25773;&#21487;&#20197;&#24471;&#21040;&#65306;    
 ![$${{\partial Cost} \over {\partial s_k^i}} = {f^'}(s_k^i)W_{k, \bullet }^{i + 1}{{\partial Cost} \over {\partial {s^{i + 1}}}}$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20s_k%5Ei%7D%7D%20%3D%20%7Bf%5E%27%7D%28s_k%5Ei%29W_%7Bk%2C%20%5Cbullet%20%7D%5E%7Bi%20&plus;%201%7D%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20%7Bs%5E%7Bi%20&plus;%201%7D%7D%7D%7D%24%24)
- - &#26435;&#37325;&#30340;&#20559;&#23548;&#65288;&#26799;&#24230;&#65289;&#23601;&#20026;&#65306;![$${{\partial Cost} \over {\partial w_{l,k}^i}} = z_l^i{{\partial Cost} \over {\partial s_k^i}}$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20w_%7Bl%2Ck%7D%5Ei%7D%7D%20%3D%20z_l%5Ei%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20s_k%5Ei%7D%7D%24%24)
+ - &#26435;&#37325;&#30340;&#20559;&#23548;&#65288;&#26799;&#24230;&#65289;&#23601;&#20026;&#65306;    
+ ![$${{\partial Cost} \over {\partial w_{l,k}^i}} = z_l^i{{\partial Cost} \over {\partial s_k^i}}$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20w_%7Bl%2Ck%7D%5Ei%7D%7D%20%3D%20z_l%5Ei%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20s_k%5Ei%7D%7D%24%24)
  - &#36824;&#26159;BP&#21453;&#21521;&#20256;&#25773;&#30340;&#25512;&#23548;&#65292;&#21487;&#20197;&#26597;&#30475;&#25105;&#20043;&#21069;&#32473;&#30340;BP&#21453;&#21521;&#20256;&#25773;&#30340;&#25512;&#23548;&#12290;
  - &#23427;&#36825;&#37324;`W`&#26159;&#20174;`0`&#24320;&#22987;&#30340;&#65292;&#25152;&#20197;&#23545;&#24212;&#21487;&#33021;&#19981;&#22826;&#19968;&#33268;&#12290;
-- `tanh`&#30340;&#23548;&#25968;&#20026;&#65306;![$${[\tanh (x)]^'} = 1 - {[\tanh (x)]^2}$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24%7B%5B%5Ctanh%20%28x%29%5D%5E%27%7D%20%3D%201%20-%20%7B%5B%5Ctanh%20%28x%29%5D%5E2%7D%24%24)
+- `tanh`&#30340;&#23548;&#25968;&#20026;&#65306;     
+![$${[\tanh (x)]^'} = 1 - {[\tanh (x)]^2}$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24%7B%5B%5Ctanh%20%28x%29%5D%5E%27%7D%20%3D%201%20-%20%7B%5B%5Ctanh%20%28x%29%5D%5E2%7D%24%24)
  - &#25152;&#20197;&#65306;![$${f^'}(0) = 1$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24%7Bf%5E%27%7D%280%29%20%3D%201%24%24)
- - &#24403;![$$s_k^i$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24s_k%5Ei%24%24)&#30340;&#24456;&#23567;&#26102;&#65292;&#21487;&#20197;&#24471;&#21040;&#65292;![$${f^'}(s_k^i) \approx 1$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24%7Bf%5E%27%7D%28s_k%5Ei%29%20%5Capprox%201%24%24)
+ - &#24403;![$$s_k^i$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24s_k%5Ei%24%24)&#30340;&#24456;&#23567;&#26102;&#65292;&#21487;&#20197;&#24471;&#21040;&#65306;     
+ ![$${f^'}(s_k^i) \approx 1$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24%7Bf%5E%27%7D%28s_k%5Ei%29%20%5Capprox%201%24%24)
  - &#36825;&#37324;&#23454;&#38469;&#19978;&#20182;&#26159;&#20551;&#35774;&#28608;&#21169;&#20989;&#25968;&#26159;&#32447;&#24615;&#30340;&#65292;&#19979;&#19968;&#31687;&#35770;&#25991;&#20013;&#20063;&#26377;&#25552;&#21040;&#12290;
-- &#26681;&#25454;**&#26041;&#24046;**&#20844;&#24335;&#65306;![$$Var(x) = E({x^2}) - {E^2}(x)$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24Var%28x%29%20%3D%20E%28%7Bx%5E2%7D%29%20-%20%7BE%5E2%7D%28x%29%24%24)&#21487;&#20197;&#24471;&#21040;&#65306;
-![$$Var[{z^i}] = Var[x]\prod\limits_{j = 0}^{i - 1} {{n_j}Var[{W^j}]} $$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24Var%5B%7Bz%5Ei%7D%5D%20%3D%20Var%5Bx%5D%5Cprod%5Climits_%7Bj%20%3D%200%7D%5E%7Bi%20-%201%7D%20%7B%7Bn_j%7DVar%5B%7BW%5Ej%7D%5D%7D%20%24%24)&#65292;&#25512;&#23548;&#22914;&#19979;&#65306;
+- &#26681;&#25454;**&#26041;&#24046;**&#20844;&#24335;&#65306;
+![$$Var(x) = E({x^2}) - {E^2}(x)$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24Var%28x%29%20%3D%20E%28%7Bx%5E2%7D%29%20-%20%7BE%5E2%7D%28x%29%24%24)    
+&#21487;&#20197;&#24471;&#21040;&#65306;     
+![$$Var[{z^i}] = Var[x]\prod\limits_{j = 0}^{i - 1} {{n_j}Var[{W^j}]} $$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24Var%5B%7Bz%5Ei%7D%5D%20%3D%20Var%5Bx%5D%5Cprod%5Climits_%7Bj%20%3D%200%7D%5E%7Bi%20-%201%7D%20%7B%7Bn_j%7DVar%5B%7BW%5Ej%7D%5D%7D%20%24%24)&#65292;     
+&#25512;&#23548;&#22914;&#19979;&#65306;
  - ![$$Var(s) = Var(\sum\limits_i^n {{w_i}{x_i}} ) = \sum\limits_i^n {Var({w_i}{x_i})} $$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24Var%28s%29%20%3D%20Var%28%5Csum%5Climits_i%5En%20%7B%7Bw_i%7D%7Bx_i%7D%7D%20%29%20%3D%20%5Csum%5Climits_i%5En%20%7BVar%28%7Bw_i%7D%7Bx_i%7D%29%7D%20%24%24)
  - ![enter description here][21]&#65288;&#24335;&#23376;&#22826;&#38271;&#65292;&#30452;&#25509;&#25130;&#22270;&#30340;&#65292;&#27809;&#29992;LaTex&#35299;&#26512;&#65289;
  - &#22240;&#20026;&#36755;&#20837;&#30340;**&#22343;&#20540;&#20026;0**&#65292;&#21487;&#20197;&#24471;&#21040;&#65306;![$$E(w) = E(x) = 0$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24E%28w%29%20%3D%20E%28x%29%20%3D%200%24%24)

-Original file line number
+Diff line change
 ![$${z^{i + 1}} = f({s^i})$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24%7Bz%5E%7Bi%20&plus;%201%7D%7D%20%3D%20f%28%7Bs%5Ei%7D%29%24%24)，
 - 根据BP反向传播可以得到：
 ![$${{\partial Cost} \over {\partial s_k^i}} = {f^'}(s_k^i)W_{k, \bullet }^{i + 1}{{\partial Cost} \over {\partial {s^{i + 1}}}}$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20s_k%5Ei%7D%7D%20%3D%20%7Bf%5E%27%7D%28s_k%5Ei%29W_%7Bk%2C%20%5Cbullet%20%7D%5E%7Bi%20&plus;%201%7D%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20%7Bs%5E%7Bi%20&plus;%201%7D%7D%7D%7D%24%24)
 - - 权重的偏导（梯度）就为：![$${{\partial Cost} \over {\partial w_{l,k}^i}} = z_l^i{{\partial Cost} \over {\partial s_k^i}}$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20w_%7Bl%2Ck%7D%5Ei%7D%7D%20%3D%20z_l%5Ei%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20s_k%5Ei%7D%7D%24%24)
 + - 权重的偏导（梯度）就为：
 + ![$${{\partial Cost} \over {\partial w_{l,k}^i}} = z_l^i{{\partial Cost} \over {\partial s_k^i}}$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20w_%7Bl%2Ck%7D%5Ei%7D%7D%20%3D%20z_l%5Ei%7B%7B%5Cpartial%20Cost%7D%20%5Cover%20%7B%5Cpartial%20s_k%5Ei%7D%7D%24%24)
  - 还是BP反向传播的推导，可以查看我之前给的BP反向传播的推导。
  - 它这里`W`是从`0`开始的，所以对应可能不太一致。
 -- `tanh`的导数为：![$${[\tanh (x)]^'} = 1 - {[\tanh (x)]^2}$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24%7B%5B%5Ctanh%20%28x%29%5D%5E%27%7D%20%3D%201%20-%20%7B%5B%5Ctanh%20%28x%29%5D%5E2%7D%24%24)
 +- `tanh`的导数为：
 +![$${[\tanh (x)]^'} = 1 - {[\tanh (x)]^2}$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24%7B%5B%5Ctanh%20%28x%29%5D%5E%27%7D%20%3D%201%20-%20%7B%5B%5Ctanh%20%28x%29%5D%5E2%7D%24%24)
  - 所以：![$${f^'}(0) = 1$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24%7Bf%5E%27%7D%280%29%20%3D%201%24%24)
 - - 当![$$s_k^i$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24s_k%5Ei%24%24)的很小时，可以得到，![$${f^'}(s_k^i) \approx 1$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24%7Bf%5E%27%7D%28s_k%5Ei%29%20%5Capprox%201%24%24)
 + - 当![$$s_k^i$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24s_k%5Ei%24%24)的很小时，可以得到：
 + ![$${f^'}(s_k^i) \approx 1$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24%7Bf%5E%27%7D%28s_k%5Ei%29%20%5Capprox%201%24%24)
  - 这里实际上他是假设激励函数是线性的，下一篇论文中也有提到。
 -- 根据**方差**公式：![$$Var(x) = E({x^2}) - {E^2}(x)$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24Var%28x%29%20%3D%20E%28%7Bx%5E2%7D%29%20-%20%7BE%5E2%7D%28x%29%24%24)可以得到：
 -![$$Var[{z^i}] = Var[x]\prod\limits_{j = 0}^{i - 1} {{n_j}Var[{W^j}]} $$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24Var%5B%7Bz%5Ei%7D%5D%20%3D%20Var%5Bx%5D%5Cprod%5Climits_%7Bj%20%3D%200%7D%5E%7Bi%20-%201%7D%20%7B%7Bn_j%7DVar%5B%7BW%5Ej%7D%5D%7D%20%24%24)，推导如下：
 +- 根据**方差**公式：
 +![$$Var(x) = E({x^2}) - {E^2}(x)$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24Var%28x%29%20%3D%20E%28%7Bx%5E2%7D%29%20-%20%7BE%5E2%7D%28x%29%24%24)
 +可以得到：
 +![$$Var[{z^i}] = Var[x]\prod\limits_{j = 0}^{i - 1} {{n_j}Var[{W^j}]} $$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24Var%5B%7Bz%5Ei%7D%5D%20%3D%20Var%5Bx%5D%5Cprod%5Climits_%7Bj%20%3D%200%7D%5E%7Bi%20-%201%7D%20%7B%7Bn_j%7DVar%5B%7BW%5Ej%7D%5D%7D%20%24%24)，
 +推导如下：
  - ![$$Var(s) = Var(\sum\limits_i^n {{w_i}{x_i}} ) = \sum\limits_i^n {Var({w_i}{x_i})} $$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24Var%28s%29%20%3D%20Var%28%5Csum%5Climits_i%5En%20%7B%7Bw_i%7D%7Bx_i%7D%7D%20%29%20%3D%20%5Csum%5Climits_i%5En%20%7BVar%28%7Bw_i%7D%7Bx_i%7D%29%7D%20%24%24)
  - ![enter description here][21]（式子太长，直接截图的，没用LaTex解析）
  - 因为输入的**均值为0**，可以得到：![$$E(w) = E(x) = 0$$](http://latex.codecogs.com/gif.latex?%5Clarge%20%24%24E%28w%29%20%3D%20E%28x%29%20%3D%200%24%24)