测试常用curl工具详解

原创

sheazhang

修改于 2019-09-02 11:25:30

3.7K0

修改于 2019-09-02 11:25:30

文章被收录于专栏：加速视频

前言

linux中测试网络的访问工具

title:      CDN中测试常用curl工具详解
author:     sheazhang

1. 验证某节点ip缓存情况

<p style="line-height: 2em;">[root@VM_27_201_centos ~]# curl -I 'http://app613.imgcache.xxx.com/app613/frontend_swf/version.js' -x 106.74.23.144:80
HTTP/1.1 200 OK
Server: NWS_TCloud_S1
Connection: keep-alive
Date: Sun, 03 Jul 2016 08:50:34 GMT
Cache-Control: max-age=6000
Expires: Sun, 03 Jul 2016 10:30:34 GMT
Last-Modified: Thu, 30 Jun 2016 09:12:30 GMT
Content-Type: application/x-javascript; charset=utf-8
Content-Length: 69
X-Cache-Lookup: Hit From Disktank<br></p>

PS: 其中的-x绑定代理服务器ip的用法，类似于wget中的：

# wget -e http-proxy=119.97.137.178 http://sdxxx.cloudcdn.net/test008/6wireshark-win64-1.2.9.exe

get back

2. 验证referer头部白名单是否生效

场景

使用-H 来添加自定义的头部，模拟特殊条件下源站/cdn节点对特性是否支持，结果返回是否符合预期。下面的这个例子，就是用来验证在控制台上添加了referer白名单后，模拟该referer头部返回是否符合预期。

<p style="line-height: 2em;">[root@VM_27_201_centos ~]# curl -I 'http://res2.xxxxx.com/' -x 106.74.23.144:80 -H"referer:http://*.yqh5.cn"
HTTP/1.1 200 OK
Server: NWS_TCloud_S1
Connection: keep-alive
Date: Sun, 03 Jul 2016 08:59:50 GMT
Cache-Control: max-age=31536000
Expires: Mon, 03 Jul 2017 08:59:50 GMT
Last-Modified: Tue, 12 Apr 2016 01:59:32 GMT
Content-Type: text/html
Content-Length: 612
X-Cache-Lookup: Hit From Disktank
Access-Control-Allow-Origin: *
Content-Disposition: inline; filename=""
Accept-Ranges: bytes<br></p>

PS 上面的

curl -I 'http://res2.eqxxx.com/' -x 106.74.23.144:80 -H"referer:http://*.yqh5.cn"

也可以写为：

curl -I 'http://res2.exxx.com/' -x 106.74.23.144:80  -e "http://pachong.org/"

get back

3. 验证节点/源站对分片的支持情况

[root@VM_27_201_centos ~]# curl -Iv 'http://update01.xxxx-cloud.com/pup/8S70_E6200_V016.002.230_9.zip' -x 106.74.23.144:80 -H'range:bytes=0-1000'
* About to connect() to proxy 106.74.23.144 port 80 (#0)
*   Trying 106.74.23.144... connected
* Connected to 106.74.23.144 (106.74.23.144) port 80 (#0)
> HEAD http://update01.skyworth-cloud.com/pup/8S70_E6200_V016.002.230_9.zip HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: update01.skyworth-cloud.com
> Accept: */*
> Proxy-Connection: Keep-Alive
> range:bytes=0-1000
>
< HTTP/1.1 206 Partial Content
HTTP/1.1 206 Partial Content
< Server: nws_ocmid_hy
Server: nws_ocmid_hy
< Connection: close
Connection: close
< Date: Sun, 03 Jul 2016 09:02:24 GMT
Date: Sun, 03 Jul 2016 09:02:24 GMT
< Cache-Control: max-age=600
Cache-Control: max-age=600
< Expires: Sun, 03 Jul 2016 09:12:24 GMT
Expires: Sun, 03 Jul 2016 09:12:24 GMT
< Last-Modified: Mon, 23 May 2016 20:10:06 GMT
Last-Modified: Mon, 23 May 2016 20:10:06 GMT
< Content-Range: bytes 0-1000/437405243
Content-Range: bytes 0-1000/437405243
< Content-Type: application/octet-stream
Content-Type: application/octet-stream
< Content-Length: 1001
Content-Length: 1001
< X-Daa-Tunnel: hop_count=2
X-Daa-Tunnel: hop_count=2
< X-Cache-Lookup: Hit From Disktank Upstream
X-Cache-Lookup: Hit From Disktank Upstream
< X-Cache-Lookup: Hit From Disktank
X-Cache-Lookup: Hit From Disktank
< X-Daa-Tunnel: hop_count=2
X-Daa-Tunnel: hop_count=2
< X-Cache-Lookup: Hit From Inner Cluster
X-Cache-Lookup: Hit From Inner Cluster
 
<
* Closing connection #0<br><br></p>

备注

上面的

curl -Iv 'http://update01.xxx-cloud.com/pup/8S70_E6200_V016.002.230_9.zip' -x 106.74.23.144:80 -H'range:bytes=0-1000'

也可以写成

curl -Iv 'http://update01.xxx-cloud.com/pup/8S70_E6200_V016.002.230_9.zip' -x 106.74.23.144:80 -r 0-1000

get back

4. 验证文件在节点和源站上大小是否一致（通过md5判断）

[root@VM_27_201_centos ~]# curl -o pic1 'http://as.xxx.com/show/visitor/4a2d8aae508506ef0150a6fcebbf2f53' -x 106.74.23.144:80
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10126  100 10126    0     0  40629      0 --:--:-- --:--:-- --:--:-- 54149
[root@VM_27_201_centos ~]#
[root@VM_27_201_centos ~]# md5sum pic1
4b7671dbe1a634541726bd9473d14dcc  pic1
[root@VM_27_201_centos ~]#
[root@VM_27_201_centos ~]#
[root@VM_27_201_centos ~]# curl -o pic2 'http://as.xxx.com/show/visitor/4a2d8aae508506ef0150a6fcebbf2f53' -x 115.159.20.55:80
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10126  100 10126    0     0  4629k      0 --:--:-- --:--:-- --:--:-- 9888k
[root@VM_27_201_centos ~]#
[root@VM_27_201_centos ~]# md5sum pic2
4b7671dbe1a634541726bd9473d14dcc  pic2<br></p>

备注

上面的-o 代表把文件下载到指定的路径下，-x代表使用指定的ip或者域名作为代理。

get back

5. 模拟https访问验证https证书

[root@VM_117_214_centos ~]#  curl -sv 'https://xxx.qcloud.com/static/img/57c45bb3997a277e5996dcf13314b365/user-40.png' -o /dev/null -k
* About to connect() to mccdn.qcloud.com port 443 (#0)
*   Trying 113.207.16.63... connected
* Connected to mccdn.qcloud.com (113.207.16.63) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* warning: ignoring value of ssl.verifyhost
* skipping SSL peer certificate verification
* SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
*     subject: CN=*.qcloud.com,OU=R&D,O=Shenzhen Tencent Computer Systems Company Limited,L=shenzhen,ST=guangdong,C=CN
*     start date: May 25 00:00:00 2016 GMT
*     expire date: Jul 24 23:59:59 2019 GMT
*     common name: *.qcloud.com
*     issuer: CN=Symantec Class 3 Secure Server CA - G4,OU=Symantec Trust Network,O=Symantec Corporation,C=US
> GET /static/img/57c45bb3997a277e5996dcf13314b365/user-40.png HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: mccdn.qcloud.com
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nnws/1.7.3.8
< Date: Sat, 20 Aug 2016 12:43:55 GMT
< Content-Type: image/png
< Content-Length: 3558
< Connection: keep-alive
< Cache-Control: max-age=2592000
< Expires: Mon, 19 Sep 2016 12:43:54 GMT
< Last-Modified: Wed, 25 May 2016 08:06:29 GMT
< X-Cache-Lookup: Hit From Disktank
< 
{ [data not shown]
* Connection #0 to host mccdn.qcloud.com left intact
* Closing connection #0

get back

6. curl中绑定host测试方法

等同于在windows中绑定host进行测试

[root@VM_230113centos ~]# curl -I -H 'host:7u2q8y.com3.z0.glb.xxxcdn.com' http://xxx.com.z0.glb.xxxdns.com/M-_WoLgVqduR7VFfrhnmWwL8YgA=/luI-
 
lideKDodDAYi55H6D2HV6HcO/000000.ts
HTTP/1.1 200 OK
Server: nginx/1.4.4
Date: Sat, 02 Jan 2016 11:39:23 GMT
Content-Type: video/mp2t
Content-Length: 673416
Connection: keep-alive
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Access-Control-Max-Age: 2592000
Cache-Control: public, max-age=31536000
Content-Disposition: inline; filename="000000.ts"
Content-Transfer-Encoding: binary
Etag: "Fs–ekImTQ2ZMjO_MdTeTN0Yew_5"
Last-Modified: Sun, 22 Nov 2015 06:28:55 GMT
X-Log: mc.g;IO:1
X-Reqid: iToAAMIdW1cTmSUU
X-Qiniu-Zone: 0

[root@VM_230113centos ~]# curl -I -H 'host:7u2q8y.com3.z0.glb.xxxcdn.com' http://211.161.127.147/M-_WoLgVqduR7VFfrhnmWwL8YgA=/luI-lideKDodDAYi55H6D2HV6HcO/000000.ts
HTTP/1.1 200 OK
Server: NWS_Appimg_HY
Connection: keep-alive
Date: Sat, 02 Jan 2016 11:45:50 GMT
Cache-Control: max-age=31536000
Expires: Sun, 01 Jan 2017 11:45:50 GMT
Last-Modified: Sun, 22 Nov 2015 06:28:55 GMT
Content-Type: video/mp2t
Content-Length: 673416
X-Cache-Lookup: Hit From Disktank
Access-Control-Allow-Origin: *
Accept-Ranges: bytes
Etag: "Fs–ekImTQ2ZMjO_MdTeTN0Yew_5"
X-ReqId: ATUAABGu8oBVjyUU
Content-Disposition: inline; filename="000000.ts"

其中：

7u2q8y.com3.z0.glb.qiniucdn.com 为用户的域名
src.com.z0.glb.qiniudns.com 为源站或者回源host地址，也可以为ip地址形式
211.161.127.147 为域名接入cdn之后解析到的cdn节点ip地址

通过验证源站url和cdn节点ip地址后的curl反馈可以判断出到底是源站有问题，还是cdn节点有问题

get back

7. 验证TCP握手和ssl握手的延时

curl -w "TCP handshake: %{time_connect}, SSL handshake: %{time_appconnect}\n" -so /dev/null https://www.alipay.com

8. 使用curl抓取返回时间

判断网页访问各耗时时间

curl -o /dev/null -w "Connect: %{time_connect} TTFB: %{time_starttransfer} Total time: %{time_total} \n" http://www.itts-union.com
This command will output this:
[root@VM_117_214_centos ~]#  curl -o /dev/null -w "Connect: %{time_connect} TTFB: %{time_starttransfer} Total time: %{time_total} \n" http://www.itts-union.com/1326.html
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 74470    0 74470    0     0  52368      0 --:--:--  0:00:01 --:--:--  116k
Connect: 0.799 TTFB: 1.309 Total time: 1.422

get back

9. 查询各个阶段的耗时时间：

-w 可以指定特殊文件 也可以指定耗时**%{time_namelookup}**

[root@VM_49_231_centos ~]# curl -Iv "http://img.yxp01.com/
images/20/2017/07/VXunsmRXXU355lC5lM8xc5sXmFNfus.jpg?v=4" -x 
119.6.229.37:80 -so /dev/null -w "@curl-time"

特殊文件：

[root@VM_49_231_centos ~]# cat curl-time 
    Time-DNS-lookup:  %{time_namelookup}s\n
   Time-TCP-connect:  %{time_connect}s\n
   Time-SSL-connect:  %{time_appconnect}s\n
      Time-redirect:  %{time_redirect}s\n
  Time-Pre-transfer:  %{time_pretransfer}s\n
Time-First-transfer:  %{time_starttransfer}s\n
                    ----------\n
         Time-total:  %{time_total}s\n

get back

10. -L实现301或者302的跳转

11. 缓存文件建立时间

Pragma: X-Get-Last-Update-Info

get back

12. 查看cdn节点回源的链路

x_upstream_info=on

curl -o 100 "http://res.hoisin.xxxtv.com/video/20150731164143_960.ts?x_upstream_info=on" -x 61.140.13.201:80 -H "Pragma: X-Get-Last-Update-Info" -v

[root@sheazhang ~]# curl -o /dev/null -v "http://zl.winyp.cn/test.jpg?x_upstream_info=on"
> GET /test.jpg?x_upstream_info=on HTTP/1.1
> User-Agent: curl/7.29.0
> Host: zl.winyp.cn
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: NWS_TCloud_S1
< Connection: keep-alive
< Date: Tue, 01 Jan 2019 02:46:29 GMT
< Cache-Control: max-age=600
< Expires: Tue, 01 Jan 2019 02:56:29 GMT
< Last-Modified: Thu, 22 Jun 2017 18:27:09 GMT
< Content-Type: image/jpeg
< Content-Length: 201557
< X-NWS-LOG-UUID: 4243021142118093756 4c4df2ad367f7a7282b8c634e5bb5346
< Access-Control-Allow-Origin: *
< X-Daa-Tunnel: hop_count=3
< X-Cache-Lookup: Hit From Upstream 125.39.46.83, cost 33 ms
< X-Upstream-Info: l2.cloudhy.tcdn.qq.com
< X-Cache-Lookup: Hit From Disktank3
< Accept-Ranges: bytes
< X-Cache-Lookup: Hit From Inner Cluster

13. 指定访问的来源客户端ua

-A 'ua'

[root@sheaBJ ~]# curl -Iv "http://www.xxx.com/ckkhj/index.shtml" -x 1.xxx:80 -A 'Smartphone'
* About to connect() to proxy 1.180.204.168 port 80 (#0)
*   Trying 1.180.204.168...
* Connected to 1.180.204.168 (1.180.204.168) port 80 (#0)
> HEAD http://www.dongao.com/ckkhj/index.shtml HTTP/1.1
> User-Agent: Smartphone
> Host: www.dongao.com
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 301 Moved Permanently
HTTP/1.1 301 Moved Permanently
< Server: NWS_TCloud_S1
Server: NWS_TCloud_S1
< Connection: keep-alive
Connection: keep-alive
< Date: Fri, 08 Mar 2019 03:51:38 GMT
Date: Fri, 08 Mar 2019 03:51:38 GMT
< Content-Length: 39
Content-Length: 39
< Location: http://m.dongao.com/ckkhj/index.shtml
Location: http://m.dongao.com/ckkhj/index.shtml

< 
* Excess found in a non pipelined read: excess = 39 url = /ckkhj/index.shtml (zero-length body)
* Connection #0 to host 1.180.204.168 left intact

14. curl常用头部集合

 -H 'Accept-Encoding:gzip'   测试压缩
 -L                 实现301或者302的跳转
 -H 'Pragma: X-Get-Last-Update-Info'     缓存文件建立时间
 -H 'x_upstream_info=on'			cdn节点回源的链路
 -H 'host:7u2q8y.com3.z0.glb.qiniucdn.com'   绑定hosts
 -H 'range:bytes=0-1000'  或 -r 0-0   测试分片支持
 -H "referer:http://*.yqh5.cn"     验证reffer头部
 -d "x=123"  	模拟post请求
 -H "Origin:http://sheazhang.winyp.cn/"  验证跨域的来源情况，其中O大写
 -w "@curl-time"				各个阶段的耗时时间

特殊文件：

[root@VM_49_231_centos ~]# cat curl-time 
    Time-DNS-lookup:  %{time_namelookup}s\n
   Time-TCP-connect:  %{time_connect}s\n
   Time-SSL-connect:  %{time_appconnect}s\n
      Time-redirect:  %{time_redirect}s\n
  Time-Pre-transfer:  %{time_pretransfer}s\n
Time-First-transfer:  %{time_starttransfer}s\n
                    ----------\n
         Time-total:  %{time_total}s\n

15. httpstat的脚本-time

参考git链接

16. 测试源站是否吐出304的情况

-H 'If-Modified-Since:xxx'

[root@VM_14_42_centos ~]# curl -o 1 -v "http://j.xxx.cc/daobo/js/fabric.js" -x 139.xxx.204:80 -H 'If-Modified-Since: Mon, 23 Apr 2018 03:38:55 GMT'
* About to connect() to proxy 139.199.214.204 port 80 (#0)
*   Trying 139.xxx...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
				 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 139.199.214.204 (139.199.214.204) port 80 (#0)
> GET http://j.xxx.cc/daobo/js/fabric.js HTTP/1.1
> User-Agent: curl/7.29.0
> Host: xxx
> Accept: */*
> Proxy-Connection: Keep-Alive
> If-Modified-Since: Mon, 23 Apr 2018 03:38:55 GMT
>
< HTTP/1.1 304 Not Modified
< Server: nginx/1.14.2
< Date: Mon, 15 Apr 2019 07:40:50 GMT
< Connection: keep-alive
< Last-Modified: Mon, 23 Apr 2018 03:38:55 GMT
< ETag: "5add554f-da835"
< Access-Control-Allow-Origin: *
<
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host 139.199.214.204 left intact

17. 其他的不常见用法，仅做了解

1、抓取页面内容到一个文件中

[root@www.itts-union.com]# curl -o home.html  http://www.itts-union.com

2、模拟表单信息，模拟登录，保存cookie信息

[root@www.itts-union.com]# curl -c ./cookie_c.txt -F log=aaaa -F pwd=****** http://www.itts-union.com/wp-login.php

3、使用cookie文件

[root@www.itts-union.com]# curl -b ./cookie_c.txt  http://www.itts-union.com/wp-admin

使用cookie头部：

[root@www.itts-union.com]# curl -o /dev/null -v http://www.itts-union.com/wp-admin -H "Cookie: uid=999;sex=2"

4、显示下载进度条

[root@VM_117_214_centos ~]# curl -o smokeping.tar.gz http://oss.oetiker.ch/smokeping/pub/smokeping-2.6.9.tar.gz -#
######################################################################## 100.0%

5、通过ftp下载文件

[zhangy@www.itts-union.com ~]$ curl -u 用户名:密码 -O http://www.itts-union.com/demo/curtain/bbstudy_files/style.css
　　% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
　　Dload  Upload   Total   Spent    Left  Speed
　　101  1934  101  1934    0     0   3184      0 –:–:– –:–:– –:–:–  7136

　　或者用下面的方式

[zhangy@www.itts-union.com ~]$ curl -O ftp://用户名:密码@ip:port/demo/curtain/bbstudy_files/style.css
[zhangy@www.itts-union.com ~]$ curl -O ftp://用户名:密码@ip:port/demo/curtain/bbstudy_files/style.css

6、通过ftp上传

[zhangy@www.itts-union.com ~]$ curl -T test.sql ftp://用户名:密码@ip:port/demo/curtain/bbstudy_files/

7、如下载网页发现乱码，可以使用iconv转码：

[root@VM_117_214_centos
 ~]# curl -o smokeping.tar.gz 
http://oss.oetiker.ch/smokeping/pub/smokeping-2.6.9.tar.gz -# | iconv 
-fgb2312
######################################################################## 100.0%

8、验证https访问

curl -sv 'https://mccdn.qcloud.com/static/img/57c45bb3997a277e5996dcf13314b365/user-40.png' -o/dev/null --resolve "mccdn.qcloud.com:443:221.204.60.11"

get back

原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。

如有侵权，请联系 cloudcommunity@tencent.com 删除。

http