GN(GroupNorm)
介于LN和IN之间,其首先将channel分为许多组(group),对每一组做归一化,及先将feature的维度由N, C, H, Wreshape为N, G,C//G ,...H, W,归一化的维度为C//G , H, W
如图一所示,GN是介于LN和IN之间,将C分为多个group,B,C,H,W转换为B*G,C/G,H,W然后对每个组进行归一化,也就是与batch和layer...GN代码实现:
def torch_gn_offical(x, num_groups):
""" 调用官方API
"""
gn = nn.GroupNorm(num_groups...def torch_gn(x, num_groups):
b = x.shape[0]
x1 = x.view(b, num_groups, -1)
mu = x1.mean...= x1_norm.reshape(x.shape)
return my_gn
GN的优缺点:
GN和BN对比,避开了batchsize对训练的影响,训练开销小;GN的num_group=1就是