Activation Function PReLu
The activation function PReLu is:
\[f(y_i)=\begin{cases} y_i,&\text{if }y_i>0 \\ a_iy_i,&\text{if }y_i\leq0\\ \end{cases}\]
\[\frac{\partial f}{\partial a_i}=\begin{cases} 0,&\text{if }y_i>0 \\ y_i,&\text{if }y_i\leq0\\ \end{cases}\] The \(a_i\) update method: \[\triangle a_i:=\mu \triangle a_i+\epsilon\frac{\partial\varepsilon}{\partial a_i} \] where \(\mu\) is the momentum, \(\epsilon\) is the learning rate and \({\varepsilon}\) is target function.
And: \[\frac{\partial\varepsilon}{\partial a_i}=\sum_{y_i}{\frac{\partial\varepsilon}{\partial f(y_i) }\frac{\partial f(y_i) }{\partial a_i}} \]
Conclusion
Leaky ReLu 的\(a_i\)值是固定的，而Parametric ReLu的\(a_i\)值会根据input data作修正。
Reference
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification