神经网络常见层Numpy封装参考：激活层（3）

目录前置代码激活层ReLU层公式封装LeakyReLU层公式封装Sigmoid层公式封装Tanh层公式封装测试前置代码- 神经网络常见层Numpy封装参考损失层1- 神经网络常见层Numpy封装参考线性层2激活层ReLU层公式forward R e L U ( i n p u t ) w h e r e ( i n p u t 0 , i n p u t , 0 ) g r a d g r a d i n × w h e r e ( i n p u t 0 , 1 , 0 ) \begin{array}{l} {\text{forward} {\mathop{\rm ReLU}\nolimits} (input) {\rm{where}}\left( {input{\rm{ 0, }}input{\rm{, 0}}} \right)}\\ {{\bf{grad}} {\bf{grad_{in}}} \times {\rm{where}}\left( {input{\rm{ 0, }}1{\rm{, 0}}} \right)} \end{array}forwardReLU(input)where(input0,input,0)gradgradin​×where(input0,1,0)​封装classReLU(Module):defforward(self,x:np.ndarray)-np.ndarray:# 储存输入数据用于反向传播self.xxreturnnp.where(self.x0,x,0)defbackward(self,grad:np.ndarray)-np.ndarray:returngrad*np.where(self.x0,1,0)def__repr__(self):returnself.__class__.__name__()LeakyReLU层公式forward L e a k y R e L U ( i n p u t ) w h e r e ( i n p u t 0 , i n p u t , α × i n p u t ) g r a d g r a d i n × w h e r e ( i n p u t 0 , 1 , α ) \begin{array}{l} {\text{forward} {\mathop{\rm LeakyReLU}\nolimits} (input) {\rm{where}}\left( {input{\rm{ 0, }}input{\rm{, }}\alpha \times input} \right)}\\ {{\bf{grad}} {\bf{grad_{in}}} \times {\rm{where}}\left( {input{\rm{ 0, 1, }}\alpha } \right)} \end{array}forwardLeakyReLU(input)where(input0,input,α×input)gradgradin​×where(input0,1,α)​封装classLeakyReLU(Module):defforward(self,x:np.ndarray,alpha:float0.01)-np.ndarray:# 微小常数self.alphaalpha# 储存输入数据用于反向传播self.xxreturnnp.where(self.x0,x,self.alpha*x)defbackward(self,grad:np.ndarray)-np.ndarray:returngrad*np.where(self.x0,1,self.alpha)def__repr__(self):returnself.__class__.__name__()Sigmoid层公式forward s i g m o i d ( i n p u t ) 1 1 e − i n p u t g r a d g r a d i n × o u t p u t ( 1 − o u t p u t ) \begin{array}{l} {\text{forward} {\mathop{\rm sigmoid}\nolimits} (input) \frac{1}{{1 {e^{ - input}}}}}\\ {{\bf{grad}} {\bf{grad_{in}}} \times output(1 - output)} \end{array}forwardsigmoid(input)1e−input1​gradgradin​×output(1−output)​封装classSigmoid(Module):defforward(self,x:np.ndarray)-np.ndarray:# 暂存前向传播输出用于反向传播self.output1/(1np.exp(-x))returnself.outputdefbackward(self,grad:np.ndarray)-np.ndarray:# 反向传播梯度计算returngrad*self.output*(1-self.output)def__repr__(self):returnself.__class__.__name__()Tanh层公式forward tanh ⁡ ( i n p u t ) e i n p u t − e − i n p u t e i n p u t e − i n p u t g r a d g r a d i n × ( 1 − o u t p u t 2 ) \begin{array}{l} \text{forward} \tanh (input) \frac{{{e^{input}} - {e^{ - input}}}}{{{e^{input}} {e^{ - input}}}}\\ {\bf{grad}} {\bf{grad_{in}}} \times (1 - outpu{t^2}) \end{array}forwardtanh(input)einpute−inputeinput−e−input​gradgradin​×(1−output2)​封装classTanh(Module):defforward(self,x:np.ndarray)-np.ndarray:# 暂存前向传播输出用于反向传播self.outputnp.tanh(x)returnself.outputdefbackward(self,grad:np.ndarray)-np.ndarray:# 反向传播梯度计算returngrad*(1-np.square(self.output))def__repr__(self):returnself.__class__.__name__()测试# 模拟多分类任务将具有5个特征的数据分为3个类别modelSequential(Linear(5,10),ReLU(),Linear(10,10),LeakyReLU(),Linear(10,5),Sigmoid(),Linear(5,3),Tanh())criterionCrossEntropyLoss()# 随机创建100个5维输入数据xnp.random.randn(100,5)# 前向传播产生预测值y_predmodel(x)# 随机生成分类类别类别数为3ynp.random.randint(0,3,100)# 计算损失、梯度loss,gradcriterion(y_pred,y)# 线性层反向传播model.backward(grad)print(损失,loss)损失 1.4188282892331157下一篇 - 神经网络常见层Numpy封装参考优化器4