基于Pytorch构建ResNet网络对cifar-10进行分类

何凯明等人在2015年提出的ResNet，在ImageNet比赛classification任务上获得第一名，获评CVPR2016最佳论文。

自从深度神经网络在ImageNet大放异彩之后，后来问世的深度神经网络就朝着网络层数越来越深的方向发展，从LeNet、AlexNet、VGG-Net、GoogLeNet。直觉上我们不难得出结论：增加网络深度后，网络可以进行更加复杂的特征提取，因此更深的模型可以取得更好的结果。

但事实并非如此，人们发现随着网络深度的增加，模型精度并不总是提升，并且这个问题显然不是由过拟合（overfitting）造成的，因为网络加深后不仅测试误差变高了，它的训练误差竟然也变高了。作者提出，这可能是因为更深的网络会伴随梯度消失/爆炸问题，从而阻碍网络的收敛。将这种加深网络深度但网络性能却下降的现象称为退化问题。例如传统神经网络的层数从20增加为56时，网络的训练误差和测试误差均出现了明显的增长。

何博士灵感爆发，他提出了残差学习来解决退化问题。对于一个堆积层结构（几层堆积而成）当输入为 时其学习到的特征记为 H(x) ，现在我们希望其可以学习到残差 F(x)=H(x)-x ，这样其实原始的学习特征是 F(x)+x 。之所以这样是因为残差学习相比原始特征直接学习更容易。当残差为0时，此时堆积层仅仅做了恒等映射，至少网络性能不会下降，实际上残差不会为0，这也会使得堆积层在输入特征基础上学习到新的特征，从而拥有更好的性能。残差学习的结构如图4所示。这有点类似与电路中的“短路”，所以是一种短路连接

而且网络的训练，也并不是越多轮次越好，这一点可以参考一下GoogLeNet网络的训练和验证数据，基本上到了60轮次，就出现上下波动了。

ResNet主要有五种主要形式：Res18，Res34，Res50，Res101，Res152；

如下图所示，每个网络都包括三个主要部分：输入部分、输出部分和中间卷积部分（中间卷积部分包括如图所示的Stage1到Stage4共计四个stage）。尽管ResNet的变种形式丰富，但都遵循上述的结构特点，网络之间的不同主要在于中间卷积部分的block参数和个数存在差异。

具体代码如下：

1. #定义ResNet网络
2. # 3x3卷积定义
3. def conv3x3(in_channels, out_channels, kernel_size = 3,stride=1, padding=1):
4. return nn.Conv2d(in_channels, out_channels, kernel_size=kernel_size,
5. stride=stride, padding=padding, bias=False)
8. # Resnet_50中的残差块
9. classResidualBlock(nn.Module):
10. def __init__(self, in_channels, out_channels, stride=1, downsample=None):
11. super(ResidualBlock, self).__init__()
12. self.mid_channels = out_channels//4
13. self.conv1 = conv3x3(in_channels, self.mid_channels, kernel_size=1, stride=stride, padding=0)#Resnet50中，从第二个残差块开始每个layer的第一个残差块都需要一次downsample
14. self.bn1 = nn.BatchNorm2d(self.mid_channels)
15. self.relu = nn.ReLU(inplace=True)
16. self.conv2 = conv3x3(self.mid_channels, self.mid_channels)
17. self.bn2 = nn.BatchNorm2d(self.mid_channels)
18. self.conv3 = conv3x3(self.mid_channels, out_channels,kernel_size=1,padding=0)
19. self.bn3 = nn.BatchNorm2d(out_channels)
20. self.downsample_0 = conv3x3(in_channels,out_channels,kernel_size=1,stride=1,padding=0)
21. self.downsample = downsample
23. def forward(self, x):
24. residual = x
25. out = self.conv1(x)
26. out = self.bn1(out)
27. out = self.relu(out)
28. out = self.conv2(out)
29. out = self.bn2(out)
30. out = self.conv3(out)
31. out = self.bn3(out)
32. if self.downsample:
33. residual = self.downsample(x)
34. else:
35. residual = self.downsample_0(x)
37. out += residual
38. out = self.bn3(out)
39. out = self.relu(out)
40. return out
43. # ResNet定义
44. classResNet(nn.Module):
45. def __init__(self, block, layers, num_classes=10):
46. super(ResNet, self).__init__()
47. self.conv = conv3x3(3, 64,kernel_size=7,stride=2)
48. self.bn = nn.BatchNorm2d(64)
49. self.max_pool = nn.MaxPool2d(3,2,padding=1)
50. self.layer1 = self.make_layer(block, 64, 256, layers[0])
51. self.layer2 = self.make_layer(block, 256, 512, layers[1], 2)
52. self.layer3 = self.make_layer(block, 512, 1024, layers[2], 2)
53. self.layer4 = self.make_layer(block, 1024, 2048, layers[3], 2)
54. self.avg_pool = nn.AvgPool2d(3,stride=1,padding=1)
55. self.fc = nn.Linear(math.ceil(img_height/32)*math.ceil(img_width/32)*2048, num_classes)
57. def make_layer(self, block, in_channels, out_channels, blocks, stride=1):
58. downsample = None
59. if(stride != 1):
60. downsample = nn.Sequential(
61. conv3x3(in_channels, out_channels, kernel_size=3,stride=stride),
62. nn.BatchNorm2d(out_channels))
63. layers = []
64. layers.append(block(in_channels, out_channels, stride, downsample))
66. for i in range(1, blocks):
67. layers.append(block(out_channels, out_channels))
68. return nn.Sequential(*layers)
70. def forward(self, x):
71. out = self.conv(x)
72. out = self.bn(out)
73. out = self.max_pool(out)
74. out = self.layer1(out)
75. out = self.layer2(out)
76. out = self.layer3(out)
77. out = self.layer4(out)
78. out = self.avg_pool(out)
79. out = out.view( -1,math.ceil(img_height/32)*math.ceil(img_width/32)*2048)
80. return out
82. # 定义模型输出模式，GPU和CPU均可
83. #Resnet-503-4-6-3总计(3+4+6+3)*3=48个conv层 加上开头的两个Conv一共50层
84. model = ResNet(ResidualBlock, [3, 4, 6, 3]).to(DEVICE)

输出ResNet网络架构

1. ResNet(
2. (conv): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(1, 1), bias=False)
3. (bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
4. (max_pool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
5. (layer1): Sequential(
6. (0): ResidualBlock(
7. (conv1): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
8. (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
9. (relu): ReLU(inplace=True)
10. (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
11. (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
12. (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
13. (bn3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
14. (downsample_0): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
15. )
16. (1): ResidualBlock(
17. (conv1): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
18. (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
19. (relu): ReLU(inplace=True)
20. (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
21. (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
22. (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
23. (bn3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
24. (downsample_0): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
25. )
26. (2): ResidualBlock(
27. (conv1): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
28. (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
29. (relu): ReLU(inplace=True)
30. (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
31. (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
32. (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
33. (bn3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
34. (downsample_0): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
35. )
36. )
37. (layer2): Sequential(
38. (0): ResidualBlock(
39. (conv1): Conv2d(256, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)
40. (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
41. (relu): ReLU(inplace=True)
42. (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
43. (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
44. (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
45. (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
46. (downsample_0): Conv2d(256, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
47. (downsample): Sequential(
48. (0): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
49. (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
50. )
51. )
52. (1): ResidualBlock(
53. (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
54. (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
55. (relu): ReLU(inplace=True)
56. (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
57. (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
58. (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
59. (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
60. (downsample_0): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
61. )
62. (2): ResidualBlock(
63. (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
64. (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
65. (relu): ReLU(inplace=True)
66. (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
67. (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
68. (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
69. (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
70. (downsample_0): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
71. )
72. (3): ResidualBlock(
73. (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
74. (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
75. (relu): ReLU(inplace=True)
76. (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
77. (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
78. (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
79. (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
80. (downsample_0): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
81. )
82. )
83. (layer3): Sequential(
84. (0): ResidualBlock(
85. (conv1): Conv2d(512, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)
86. (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
87. (relu): ReLU(inplace=True)
88. (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
89. (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
90. (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
91. (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
92. (downsample_0): Conv2d(512, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
93. (downsample): Sequential(
94. (0): Conv2d(512, 1024, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
95. (1): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
96. )
97. )
98. (1): ResidualBlock(
99. (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
100. (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
101. (relu): ReLU(inplace=True)
102. (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
103. (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
104. (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
105. (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
106. (downsample_0): Conv2d(1024, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
107. )
108. (2): ResidualBlock(
109. (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
110. (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
111. (relu): ReLU(inplace=True)
112. (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
113. (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
114. (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
115. (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
116. (downsample_0): Conv2d(1024, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
117. )
118. (3): ResidualBlock(
119. (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
120. (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
121. (relu): ReLU(inplace=True)
122. (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
123. (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
124. (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
125. (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
126. (downsample_0): Conv2d(1024, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
127. )
128. (4): ResidualBlock(
129. (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
130. (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
131. (relu): ReLU(inplace=True)
132. (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
133. (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
134. (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
135. (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
136. (downsample_0): Conv2d(1024, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
137. )
138. (5): ResidualBlock(
139. (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
140. (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
141. (relu): ReLU(inplace=True)
142. (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
143. (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
144. (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
145. (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
146. (downsample_0): Conv2d(1024, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
147. )
148. )
149. (layer4): Sequential(
150. (0): ResidualBlock(
151. (conv1): Conv2d(1024, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
152. (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
153. (relu): ReLU(inplace=True)
154. (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
155. (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
156. (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
157. (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
158. (downsample_0): Conv2d(1024, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
159. (downsample): Sequential(
160. (0): Conv2d(1024, 2048, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
161. (1): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
162. )
163. )
164. (1): ResidualBlock(
165. (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
166. (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
167. (relu): ReLU(inplace=True)
168. (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
169. (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
170. (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
171. (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
172. (downsample_0): Conv2d(2048, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
173. )
174. (2): ResidualBlock(
175. (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
176. (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
177. (relu): ReLU(inplace=True)
178. (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
179. (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
180. (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
181. (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
182. (downsample_0): Conv2d(2048, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
183. )
184. )
185. (avg_pool): AvgPool2d(kernel_size=3, stride=1, padding=1)
186. (fc): Linear(in_features=2048, out_features=10, bias=True)
187. )

ResNet的损失率和准确率变化

说实话，到了ResNet基本上已经无法理解其数学原理了，只知道增加了残差网络，降低深度网络学习退化问题。