深度学习中如何选择合适的初始化权重

YoungTimes

发布于 2022-04-28 11:06:35

1.6K00

代码可运行

运行总次数：0

代码可运行

不同的神经网络权重初始值会导致不同的神经网络训练结果，一个良好初始化权重可以对于神经网络的训练带来很大帮助，比如加速梯度下降(Gradient Descent)的收敛；增加梯度下降(Gradient Descent)收敛到低训练误差(Trainning Error)的概率；避免梯度消失或梯度爆炸等。

下面以一个简单的分类问题为例，比较3种不同的神经网络权重初始化方法对训练结果的影响。

1.简单的分类网络模型

假设我们需要一个分类器(classifier)将下图所示的红色的点和蓝色的点区分开来。

神经网络的模型已经构建好了，是如下一个三层的神经网络：

LINEAR->RELU->LINEAR->RELU->LINEAR->SIGMOID

模型的训练代码如下，学习率(learning_rate)默认为0.01，迭代次数默认为15000次。

def model(X, Y, learning_rate = 0.01, num_iterations = 15000, print_cost = True, initialization = "he"):
    grads = {}
    costs = [] # to keep track of the loss
    m = X.shape[1] # number of examples
    layers_dims = [X.shape[0], 10, 5, 1]
    
    # Initialize parameters dictionary.
    if initialization == "zeros":
        parameters = initialize_parameters_zeros(layers_dims)
    elif initialization == "random":
        parameters = initialize_parameters_random(layers_dims)
    elif initialization == "he":
        parameters = initialize_parameters_he(layers_dims)

    # Loop (gradient descent)

    for i in range(0, num_iterations):
        # Forward propagation
        a3, cache = forward_propagation(X, parameters)
        
        # Loss
        cost = compute_loss(a3, Y)

        # Backward propagation.
        grads = backward_propagation(X, Y, cache)
        
        # Update parameters.
        parameters = update_parameters(parameters, grads, learning_rate)
        
        # Print the loss every 1000 iterations
        if print_cost and i % 1000 == 0:
            print("Cost after iteration {}: {}".format(i, cost))
            costs.append(cost)
            
    # plot the loss
    plt.plot(costs)
    plt.ylabel('cost')
    plt.xlabel('iterations (per hundreds)')
    plt.title("Learning rate =" + str(learning_rate))
    plt.show()
    
    return parameters

模型的测试验证代码如下:

parameters = model(train_X, train_Y, initialization = "zeros")
print ("On the train set:")
predictions_train = predict(train_X, train_Y, parameters)
print ("On the test set:")
predictions_test = predict(test_X, test_Y, parameters)

万事俱备，下面我们分别验证不同的神经网络权重初始化方法对最终训练结果的影响。

2.不同权重初始化方法对比

我们使用如下3层神经网络对比3种不同的初始化方法对训练结果的影响。

def initialize_parameters_zeros(layers_dims): parameters = {} L = len(layers_dims) for l in range(1, L): parameters['W' + str(l)] = np.zeros((layers_dims[l], layers_dims[l-1])) parameters['b' + str(l)] = np.zeros((layers_dims[l], 1)) return parameters

神经网络的迭代过程输出如下:

Cost after iteration 0: 0.6931471805599453
Cost after iteration 1000: 0.6931471805599453
Cost after iteration 2000: 0.6931471805599453
Cost after iteration 3000: 0.6931471805599453
Cost after iteration 4000: 0.6931471805599453
Cost after iteration 5000: 0.6931471805599453
Cost after iteration 6000: 0.6931471805599453
Cost after iteration 7000: 0.6931471805599453
Cost after iteration 8000: 0.6931471805599453
Cost after iteration 9000: 0.6931471805599453
Cost after iteration 10000: 0.6931471805599455
Cost after iteration 11000: 0.6931471805599453
Cost after iteration 12000: 0.6931471805599453
Cost after iteration 13000: 0.6931471805599453
Cost after iteration 14000: 0.6931471805599453

迭代过程中的Cost可视化如下，可以看到，经过15000次的迭代，Cost没有任何下降。

为什么会这样呢？因为如果所有的参数都是0，那么所有神经元的输出都将是相同的，那在Back Propagation的时候同一层内所有神经元的行为也是相同的，Gradient相同，Weight Update也相同，所以训练过程的Cost不会下降。

训练后的神经网络在训练集(Trainning Set)和测试集(Test Set)上的Accuracy都为50%，基本都是在瞎猜。

2.2 Random Initialization

Random Initialization是对神经网络的初始权重采用随机初始化的方法，即为每个Weight的值赋予一个随机数。

def initialize_parameters_random(layers_dims):
    np.random.seed(3)
    parameters = {}
    L = len(layers_dims)
    
    for l in range(1, L):
        parameters['W' + str(l)] = np.random.randn(layers_dims[l], layers_dims[l-1]) * 10
        parameters['b' + str(l)] = np.zeros((layers_dims[l], 1))
        
    return parameters

训练过程的迭代输出如下:

Cost after iteration 0: inf
Cost after iteration 1000: 0.6237287551108738
Cost after iteration 2000: 0.5981106708339466
Cost after iteration 3000: 0.5638353726276827
Cost after iteration 4000: 0.550152614449184
Cost after iteration 5000: 0.5444235275228304
Cost after iteration 6000: 0.5374184054630083
Cost after iteration 7000: 0.47357131493578297
Cost after iteration 8000: 0.39775634899580387
Cost after iteration 9000: 0.3934632865981078
Cost after iteration 10000: 0.39202525076484457
Cost after iteration 11000: 0.38921493051297673
Cost after iteration 12000: 0.38614221789840486
Cost after iteration 13000: 0.38497849983013926
Cost after iteration 14000: 0.38278397192120406

迭代过程中的Cost变化的过程可视化如下。可以看到，相比于Zero Initialization，经过15000次的迭代，Cost在不断的下降。

训练后的神经网络在训练集(Trainning Set)上的Accuracy为83%，在测试集(Test Set)上的Accuracy为86%，相比于Zero Initialization好多了。

神经网络的初始Cost非常大，这是因为初始化的Weight非常大。如果随机初始化的权重比较大，神经网络的优化迭代过程就比较慢，甚至会出现梯度消失和梯度爆炸的情况。

Random Initialization的弊端在于一旦随机分布选择不当，就会导致网络优化陷入困境。

2.3 He initialization

def initialize_parameters_he(layers_dims):
    np.random.seed(3)
    parameters = {}
    L = len(layers_dims) - 1 # integer representing the number of layers
     
    for l in range(1, L + 1):
        parameters['W' + str(l)] = np.random.randn(layers_dims[l], layers_dims[l-1]) * np.sqrt(2/layers_dims[l-1])
        parameters['b' + str(l)] = np.zeros((layers_dims[l], 1))
        
    return parameters

训练过程的迭代输出如下:

Cost after iteration 0: 0.8830537463419761
Cost after iteration 1000: 0.6879825919728063
Cost after iteration 2000: 0.6751286264523371
Cost after iteration 3000: 0.6526117768893807
Cost after iteration 4000: 0.6082958970572938
Cost after iteration 5000: 0.5304944491717495
Cost after iteration 6000: 0.4138645817071794
Cost after iteration 7000: 0.3117803464844441
Cost after iteration 8000: 0.23696215330322562
Cost after iteration 9000: 0.18597287209206836
Cost after iteration 10000: 0.1501555628037182
Cost after iteration 11000: 0.12325079292273548
Cost after iteration 12000: 0.09917746546525937
Cost after iteration 13000: 0.0845705595402428
Cost after iteration 14000: 0.07357895962677366