batch&stochasic gradient descent

版权声明：本文为博主原创文章，转载请注明出处。 https://cloud.tencent.com/developer/article/1437724

stochastic gradient descent 和 batch gradient descent

水平有限,如有错误,请指正!

本文不是对梯度下降进行推倒,只是介绍两者的区别.

以Linear Model 为例.

注:

x(i)_j_x_j^{(i)} 表示第i个样本的第j个特征的值

batch gradient descent

batch gradient descent 是考虑了batch中所有样本求出来的 ∂Loss∂θj\frac{\partial Loss}{\partial \theta_j}

更新公式:

Repeat until convergence{

θj:=θj−α_1_m∑mi=1(yi−hθ(x(i)))x(i)j\theta_j := \theta_j-\alpha\frac{1}{m}\sum_{i=1}^m(y^{i}-h_{\theta}(x^{(i)}))x_j^{(i)}

}

∑mi=1(yi−hθ(x(i)))x(i)j\sum_{i=1}^m(y^{i}-h_{\theta}(x^{(i)}))x_j^{(i)}:就是 ∑mi=1∂Loss(i)∂θj\sum_{i=1}^{m}\frac{\partial Loss^{(i)}}{\partial \theta_j}

stochastic gradient descent

stochastic gradient decent :首先从训练集中随机抽取一个样本，然后使用这个样本计算梯度 ∂Loss(i)∂θj\frac{\partial Loss^{(i)}}{\partial \theta_j}，之后更新一次参数。

Repeat until convergence{

for i = 1 to m{

θj:=θj−α(y(i)−hθ(x(i)))x(i)j\theta_j := \theta_j-\alpha(y^{(i)}-h_{\theta}(x^{(i)}))x_j^{(i)}

}

}

(y(i)−hθ(x(i)))x(i)j(y^{(i)}-h_{\theta}(x^{(i)}))x_j^{(i)} 就是 ∂Loss(i)∂θj\frac{\partial Loss^{(i)}}{\partial \theta_j}