# 02.1 线性反向传播

## 2.1 线性反向传播⚓︎

### 2.1.1 正向计算的实例⚓︎

z = x \cdot y \tag{1}

x = 2w + 3b \tag{2}
y = 2b + 1 \tag{3}

$w = 3, b = 4$ 时，会得到图2-5的结果。

### 2.1.2 反向传播求解 $w$⚓︎

#### 求 $w$ 的偏导⚓︎

$z$ 开始一层一层向回看，图中各节点关于变量 $w$ 的偏导计算结果如下：

\frac{\partial{z}}{\partial{w}}=\frac{\partial{z}}{\partial{x}} \cdot \frac{\partial{x}}{\partial{w}}=y \cdot 2=18 \tag{4}

\frac{\partial{z}}{\partial{x}}=\frac{\partial{}}{\partial{x}}(x \cdot y)=y=9
\frac{\partial{x}}{\partial{w}}=\frac{\partial{}}{\partial{w}}(2w+3b)=2

z=x \cdot y=(2w+3b)(2b+1)=4wb+2w+6b^2+3b \tag{5}

\frac{\partial z}{\partial w}=4b+2=4 \cdot 4 + 2=18 \tag{6}

#### 求 $w$ 的具体变化值⚓︎

\Delta z = 18 \cdot \Delta w

\Delta w = {\Delta z \over 18}=\frac{162-150}{18}= 0.6667

w = w - 0.6667=2.3333
x=2w+3b=16.6667
z=x \cdot y=16.6667 \times 9=150.0003

### 2.1.3 反向传播求解 $b$⚓︎

#### 求 $b$ 的偏导⚓︎

1. $z$$x$$b$
2. $z$$y$$b$

\frac{\partial{z}}{\partial{b}}=\frac{\partial{z}}{\partial{x}} \cdot \frac{\partial{x}}{\partial{b}}+\frac{\partial{z}}{\partial{y}}\cdot\frac{\partial{y}}{\partial{b}}=y \cdot 3+x \cdot 2=63 \tag{7}

\frac{\partial{z}}{\partial{x}}=\frac{\partial{}}{\partial{x}}(x \cdot y)=y=9  \frac{\partial{z}}{\partial{y}}=\frac{\partial{}}{\partial{y}}(x \cdot y)=x=18  \frac{\partial{x}}{\partial{b}}=\frac{\partial{}}{\partial{b}}(2w+3b)=3  \frac{\partial{y}}{\partial{b}}=\frac{\partial{}}{\partial{b}}(2b+1)=2

z=x \cdot y=(2w+3b)(2b+1)=4wb+2w+6b^2+3b \tag{5}

\frac{\partial z}{\partial b}=4w+12b+3=12+48+3=63 \tag{8}

#### 求 $b$ 的具体变化值⚓︎

\Delta z = 63 \cdot \Delta b

\Delta b = \frac{\Delta z}{63}=\frac{162-150}{63}=0.1905

x=2w+3b=17.4285
y=2b+1=8.619
z=x \cdot y=17.4285 \times 8.619=150.2162

### 2.1.4 同时求解 $w$ 和 $b$ 的变化值⚓︎

\Delta b=\frac{\Delta z / 2}{63} = \frac{12/2}{63}=0.095
\Delta w=\frac{\Delta z / 2}{18} = \frac{12/2}{18}=0.333
• $w = w-\Delta w=3-0.333=2.667$
• $b = b - \Delta b=4-0.095=3.905$
• $x=2w+3b=2 \times 2.667+3 \times 3.905=17.049$
• $y=2b+1=2 \times 3.905+1=8.81$
• $z=x \times y=17.049 \times 8.81=150.2$

#### 【课堂练习】用Python代码实现以上双变量的反向传播计算过程⚓︎

1. 在检查 $\Delta z$ 时的值时，注意要用绝对值，因为有可能是个负数
2. 在计算 $\Delta b$$\Delta w$ 时，第一次时，它们对 $z$ 的贡献值分别是 $1/63$$1/18$，但是第二次时，由于 $b,w$ 值的变化，对 $z$ 的贡献值也会有微小变化，所以要重新计算。具体解释如下：
\frac{\partial{z}}{\partial{b}}=\frac{\partial{z}}{\partial{x}} \cdot \frac{\partial{x}}{\partial{b}}+\frac{\partial{z}}{\partial{y}}\cdot\frac{\partial{y}}{\partial{b}}=y \cdot 3+x \cdot 2=3y+2x
\frac{\partial{z}}{\partial{w}}=\frac{\partial{z}}{\partial{x}} \cdot \frac{\partial{x}}{\partial{w}}+\frac{\partial{z}}{\partial{y}}\cdot\frac{\partial{y}}{\partial{w}}=y \cdot 2+x \cdot 0 = 2y $$所以，在每次迭代中，要重新计算下面两个值：$$ \Delta b=\frac{\Delta z}{3y+2x}
\Delta w=\frac{\Delta z}{2y}

single variable: b -----
w=3.000000,b=4.000000,z=162.000000,delta_z=12.000000
delta_b=0.190476
w=3.000000,b=3.809524,z=150.217687,delta_z=0.217687
delta_b=0.003455
w=3.000000,b=3.806068,z=150.007970,delta_z=0.007970
delta_b=0.000127
w=3.000000,b=3.805942,z=150.000294,delta_z=0.000294
delta_b=0.000005
w=3.000000,b=3.805937,z=150.000011,delta_z=0.000011
delta_b=0.000000
w=3.000000,b=3.805937,z=150.000000,delta_z=0.000000
done!
final b=3.805937


single variable new: b -----
w=3.000000,b=4.000000,z=162.000000,delta_z=12.000000
factor_b=63.000000, delta_b=0.190476
w=3.000000,b=3.809524,z=150.217687,delta_z=0.217687
factor_b=60.714286, delta_b=0.003585
w=3.000000,b=3.805938,z=150.000077,delta_z=0.000077
factor_b=60.671261, delta_b=0.000001
w=3.000000,b=3.805937,z=150.000000,delta_z=0.000000
done!
final b=3.805937


1. factor_b第一次是63，以后每次都会略微降低一些
2. 第二个函数迭代了3次就结束了，而第一个函数迭代了5次，效率不一样
3. 最后得到的结果是一样的，因为这个问题只有一个解

double variable: w, b -----
w=3.000000,b=4.000000,z=162.000000,delta_z=12.000000
delta_b=0.095238, delta_w=0.333333
w=2.666667,b=3.904762,z=150.181406,delta_z=0.181406
delta_b=0.001440, delta_w=0.005039
w=2.661628,b=3.903322,z=150.005526,delta_z=0.005526
delta_b=0.000044, delta_w=0.000154
w=2.661474,b=3.903278,z=150.000170,delta_z=0.000170
delta_b=0.000001, delta_w=0.000005
w=2.661469,b=3.903277,z=150.000005,delta_z=0.000005
done!
final b=3.903277
final w=2.661469


double variable new: w, b -----
w=3.000000,b=4.000000,z=162.000000,delta_z=12.000000
factor_b=63.000000, factor_w=18.000000, delta_b=0.095238, delta_w=0.333333
w=2.666667,b=3.904762,z=150.181406,delta_z=0.181406
factor_b=60.523810, factor_w=17.619048, delta_b=0.001499, delta_w=0.005148
w=2.661519,b=3.903263,z=150.000044,delta_z=0.000044
factor_b=60.485234, factor_w=17.613053, delta_b=0.000000, delta_w=0.000001
w=2.661517,b=3.903263,z=150.000000,delta_z=0.000000
done!
final b=3.903263
final w=2.661517


### 参考资料⚓︎

http://colah.github.io/posts/2015-08-Backprop/

ch02, Level1