前往小程序,Get更优阅读体验!
立即前往
发布
社区首页 >专栏 >本地使用ClusterFuzz

本地使用ClusterFuzz

作者头像
用户1423082
发布2024-12-31 20:13:17
发布2024-12-31 20:13:17
6500
代码可运行
举报
文章被收录于专栏:giantbranch's bloggiantbranch's blog
运行总次数:0
代码可运行

获取代码

代码语言:javascript
代码运行次数:0
复制
git clone https://github.com/google/clusterfuzz
cd clusterfuzz
git pull

处于稳定性考虑,建议使用最新的发行版本,而不是master分支

代码语言:javascript
代码运行次数:0
复制
git checkout tags/vX.Y.Z

通过git tag -l或者在github上面可以看到发行版本

代码语言:javascript
代码运行次数:0
复制
$ git tag -l
1.2.0
1.2.1
reproduce-tool-stable
v1.0.0
v1.0.1
v1.1.0
v1.3.0
v1.4.0
v1.5.0
v1.5.1
v1.6.0
v1.6.1
v1.7.0
v1.7.1
v1.8.0
v1.9.0
v2.0.0
v2.0.1

我用v2.0.1

代码语言:javascript
代码运行次数:0
复制
git checkout tags/v2.0.1`

依赖

首先谷歌建议使用python 3.7,python2已经不支持

代码语言:javascript
代码运行次数:0
复制
$ python butler.py
Traceback (most recent call last):
  File "butler.py", line 33, in <module>
    guard.check()
  File "src/local/butler/guard.py", line 41, in check
    check_virtualenv()
  File "src/local/butler/guard.py", line 30, in check_virtualenv
    'You are not in a virtual env environment. Please install it with'
Exception: You are not in a virtual env environment. Please install it with `./local/install_deps.bash` or load it with `pipenv shell`. Then, you can re-run this command.

安装python 3.7:

代码语言:javascript
代码运行次数:0
复制
apt-get install -y apt-transport-https software-properties-common build-essential git curl \
      libssl-dev zlib1g-dev libncurses5-dev libncursesw5-dev libreadline-dev libsqlite3-dev \
      libgdbm-dev libdb5.3-dev libbz2-dev libexpat1-dev liblzma-dev libffi-dev uuid-dev
	  
curl -sS https://www.python.org/ftp/python/3.7.7/Python-3.7.7.tgz | tar -C /tmp -xzv && \
      cd /tmp/Python-3.7.7 && \
      ./configure --enable-optimizations && make altinstall && \
      rm -rf /tmp/Python-3.7.7

安装golang

代码语言:javascript
代码运行次数:0
复制
sudo add-apt-repository ppa:longsleep/golang-backports
sudo apt update
sudo apt install golang-go

修改./local/install_deps_linux.bash中的 bower installbower install --allow-root(因为bower install的时候默认不允许root用户)

最后执行./local/install_deps.bash

代码语言:javascript
代码运行次数:0
复制
./local/install_deps.bash

环境搭建

运行python3.7 -m pipenv shell即可进入clusterfuzzer需要的环境

我们运行python butler.py,就看到所有功能了

代码语言:javascript
代码运行次数:0
复制
usage: butler.py [-h]
                 {bootstrap,py_unittest,js_unittest,format,lint,package,deploy,run_server,run,run_bot,remote,clean_indexes,create_config,integration_tests,reproduce}
                 ...

Butler is here to help you with command-line tasks.

positional arguments:
  {bootstrap,py_unittest,js_unittest,format,lint,package,deploy,run_server,run,run_bot,remote,clean_indexes,create_config,integration_tests,reproduce}
    bootstrap           Install all required dependencies for running an
                        appengine, a bot,and a mapreduce locally.
    py_unittest         Run Python unit tests.
    js_unittest         Run Javascript unit tests.
    format              Format changed code in current branch.
    lint                Lint changed code in current branch.
    package             Package clusterfuzz with a staging revision
    deploy              Deploy to Appengine
    run_server          Run the local Clusterfuzz server.
    run                 Run a one-off script against a datastore (e.g.
                        migration).
    run_bot             Run a local clusterfuzz bot.
    remote              Run command-line tasks on a remote bot.
    clean_indexes       Clean up undefined indexes (in index.yaml).
    create_config       Create a new deployment config.
    integration_tests   Run end-to-end integration tests.
    reproduce           Reproduce a crash or error from a test case.

optional arguments:
  -h, --help            show this help message and exit

初始化

第一次运行,初始化数据,试了下跟python butler.py bootstrap执行的功能一样。。。

代码语言:javascript
代码运行次数:0
复制
python butler.py run_server --bootstrap

假如不是第一次就直接

代码语言:javascript
代码运行次数:0
复制
python butler.py run_server

如果依赖早已安装好,使用下面的就可以跳过依赖的再次安装

代码语言:javascript
代码运行次数:0
复制
python butler.py run_server --skip-install-deps

最后终于起来了,访问9000端口即可

代码语言:javascript
代码运行次数:0
复制
$ python butler.py run_server --skip-install-deps
Running: pkill -KILL -f "dev_appserver.py"
| Return code is non-zero (-9).
Running: pkill -KILL -f "CloudDatastore.jar"
| Return code is non-zero (-9).
Running: pkill -KILL -f "pubsub-emulator"
| Return code is non-zero (-9).
Running: pkill -KILL -f "run_bot"
| Return code is non-zero (-9).
Created symlink: source: /root/clusterfuzz/configs/test, target /root/clusterfuzz/src/appengine/config.
Created symlink: source: /root/clusterfuzz/src/protos, target /root/clusterfuzz/src/appengine/protos.
Created symlink: source: /root/clusterfuzz/src/python, target /root/clusterfuzz/src/appengine/python.
Running: python polymer_bundler.py (cwd='local')
| App Engine templates are up to date.
Created symlink: source: /root/clusterfuzz/local/storage/local_gcs, target /root/clusterfuzz/src/appengine/local_gcs.
Running: gunicorn -b :9000 main:app (cwd='src/appengine')
| [2020-05-18 22:51:55 +0800] [31835] [INFO] Starting gunicorn 20.0.4
| [2020-05-18 22:51:55 +0800] [31835] [INFO] Listening at: http://0.0.0.0:9000 (31835)
| [2020-05-18 22:51:55 +0800] [31835] [INFO] Using worker: sync
| [2020-05-18 22:51:55 +0800] [31841] [INFO] Booting worker with pid: 31841

访问9000端口如下:

之后启动botpython butler.py run_bot --name my-bot /path/to/my-bot,比如下面

代码语言:javascript
代码运行次数:0
复制
python butler.py run_bot --name my-fuzzing-bot `pwd`/my-fuzzing-bot

有时候空闲运行时间过长就退出了

代码语言:javascript
代码运行次数:0
复制
[1]    1446 killed     python butler.py run_bot --name my-fuzzing-bot `pwd`/my-fuzzing-bot

我们执行

代码语言:javascript
代码运行次数:0
复制
python butler.py run_bot `pwd`/my-fuzzing-bot
代码语言:javascript
代码运行次数:0
复制
$ python butler.py run_bot `pwd`/my-fuzzing-bot
Created symlink: source: /root/clusterfuzz/configs/test, target /root/clusterfuzz/src/appengine/config.
Bot directory already exists. Re-using...
| /root/clusterfuzz/my-fuzzing-bot/clusterfuzz/src/python/crash_analysis/stack_parsing/stack_analyzer.py:84: FutureWarning: Possible nested set at position 4
|   r'\s*[[][^]]*[:]([^](]*).*[]].*Check failed[:]\s*(.*)')
| /root/clusterfuzz/my-fuzzing-bot/clusterfuzz/src/python/crash_analysis/stack_parsing/stack_analyzer.py:202: FutureWarning: Possible nested set at position 3
|   r'.*[[][^]]*[:]([^](]*).*[]].*Security CHECK failed[:]\s*(.*)\.\s*')
| /root/clusterfuzz/my-fuzzing-bot/clusterfuzz/src/python/crash_analysis/stack_parsing/stack_analyzer.py:204: FutureWarning: Possible nested set at position 3
|   r'.*[[][^]]*[:]([^](]*).*[]].*Security DCHECK failed[:]\s*(.*)\.\s*')

在网页上也可以看bot的状态

可以看到bot应该是将clusterfuzz复制了一份

查看bot的log

代码语言:javascript
代码运行次数:0
复制
cd /path/to/my-bot/clusterfuzz/bot/logs
tail -f bot.log

可以看到先由于没有fuzzing任务所以显示Failed to get any fuzzing tasks

代码语言:javascript
代码运行次数:0
复制
$ tail bot.log
2020-05-19 11:16:49,456 - run_bot - INFO - Using local source, skipping source code update.
2020-05-19 11:16:49,457 - run_bot - INFO - Running platform initialization scripts.
2020-05-19 11:16:49,993 - run_bot - INFO - Completed running platform initialization scripts.
2020-05-19 11:16:50,374 - run_bot - ERROR - Failed to get any fuzzing tasks. This should not happen.
NoneType: None
2020-05-19 11:21:50,478 - run_bot - INFO - Using local source, skipping source code update.
2020-05-19 11:21:50,478 - run_bot - INFO - Running platform initialization scripts.
2020-05-19 11:21:51,009 - run_bot - INFO - Completed running platform initialization scripts.
2020-05-19 11:21:51,194 - run_bot - ERROR - Failed to get any fuzzing tasks. This should not happen.
NoneType: None

实际fuzz实例

ClusterFuzz支持覆盖率指导的模糊测试(libfuzzer和afl)和黑盒测试

接下来我们以心脏出血漏洞Heartbleed为例

使用这个平台,我们首先的编译出自己的fuzzer或者二进制程序

文档已经提供了针对OpenSSL的构建libfuzzer的代码,这样我们就得到了openssl-fuzzer-build.zip

代码语言:javascript
代码运行次数:0
复制
# Download and unpack a vulnerable version of OpenSSL:
curl -O https://ftp.openssl.org/source/old/1.0.1/openssl-1.0.1f.tar.gz
tar xf openssl-1.0.1f.tar.gz

# Build OpenSSL with ASan and fuzzer instrumentation:
cd openssl-1.0.1f/
./config

# $CC must be pointing to clang binary, see the "compiler section" link above.
make CC="$CC -g -fsanitize=address,fuzzer-no-link"
cd ..

# Download the fuzz target and its data dependencies:
curl -O https://raw.githubusercontent.com/google/clusterfuzz/master/docs/setting-up-fuzzing/heartbleed/handshake-fuzzer.cc
curl -O https://raw.githubusercontent.com/google/clusterfuzz/master/docs/setting-up-fuzzing/heartbleed/server.key
curl -O https://raw.githubusercontent.com/google/clusterfuzz/master/docs/setting-up-fuzzing/heartbleed/server.pem

# Build OpenSSL fuzz target for ClusterFuzz ($CXX points to clang++ binary):
$CXX -g handshake-fuzzer.cc -fsanitize=address,fuzzer openssl-1.0.1f/libssl.a \
  openssl-1.0.1f/libcrypto.a -std=c++17 -Iopenssl-1.0.1f/include/ -lstdc++fs   \
  -ldl -lstdc++ -o handshake-fuzzer

zip openssl-fuzzer-build.zip handshake-fuzzer server.key server.pem

之后回到平台(9000端口那个),来到Jobs,看到ADD NEW JOB表单

依次填写:

代码语言:javascript
代码运行次数:0
复制
“libfuzzer_asan_linux_openssl” for the “Name”.
“LINUX” for the “Platform”.
“libfuzzer” and “engine_asan” for the “Templates”.
CORPUS_PRUNE = True for the “Environment String”.

其中CORPUS_PRUNE = True是开启语料库修剪

最后将我们openssl-fuzzer-build.zip选择文件,再add即可

下面这个是之前忘记装go,导致模板没有导入

代码语言:javascript
代码运行次数:0
复制
但是报错,Invalid template name

不过好像确实我这个搭建完一个template也没有啊,不填template又上传不了

不过看页面确实没有template,我在源码中找到了设置template的代码`src/local/butler/scripts/setup.py`

可能是初始化失败了,导致没有template,所以我手动添加了libfuzzer和engine_asan

![](http://pic.giantbranch.cn/pic/1589876155414.png)

![](http://pic.giantbranch.cn/pic/1589876185250.png)

到后面是上传压缩包失败,抓包好像是请求是发给localhost的gcs的,但是服务器在远程,那当然失败了

所以在服务器那边装个图形界面,终于ko了

还有一个解决方案,就是修改代码,让gcs监听0.0.0.0,请求的url也相应修改

https://github.com/lookfwd/getting-started-clusterfuzz-local-in-aws

上传成功后,就可以来到Fuzzer页面,选择libfuzzer那行的edit,将我们新建的jobs勾选上,之后提交即可

可以看到libfuzzer这里也有updated

查看bot这里,可以看到my-bot2先拿到了任务

从log也可以看到

代码语言:javascript
代码运行次数:0
复制
$ tail ./my-bot2/clusterfuzz/bot/logs/bot.log
2020-05-26 11:16:22,389 - run_bot - INFO - Completed running platform initialization scripts.
2020-05-26 11:16:23,509 - run_bot - INFO - Executing command 'fuzz libFuzzer libfuzzer_asan_linux_openssl'
2020-05-26 11:16:28,551 - run_bot - INFO - Setting up fuzzer and data bundles.
2020-05-26 11:16:29,779 - run_bot - INFO - Retrieving custom binary build r1.
2020-05-26 11:17:47,563 - run_bot - INFO - Unpacked 3/3.
2020-05-26 11:17:47,578 - run_bot - INFO - Picked fuzz target handshake-fuzzer for fuzzing.
2020-05-26 11:17:47,579 - run_bot - INFO - Retrieved custom binary build r1.
2020-05-26 11:17:47,579 - run_bot - INFO - Setup application path.
2020-05-26 11:17:48,461 - run_bot - INFO - Checking for bad build.
2020-05-26 11:17:49,769 - run_bot - INFO - Recorded use of fuzz target libFuzzer_handshake-fuzzer.

但是报错了,ERROR - libFuzzer: engine encountered an error (target=handshake-fuzzer).

代码语言:javascript
代码运行次数:0
复制
$ tail bot.log
2020-05-26 11:36:19,370 - run_bot - INFO - Strategy pool was generated according to default parameters. Chosen strategies: value_profile, corpus_mutations_ml_rnn, corpus_subset
2020-05-26 11:36:19,583 - run_bot - INFO - Corpus is empty. Skip generation.
2020-05-26 11:36:19,822 - run_bot - ERROR - libFuzzer: engine encountered an error (target=handshake-fuzzer).
NoneType: None
2020-05-26 11:36:19,824 - run_bot - INFO - Skipped corpus merge since no new units added by fuzzing.
2020-05-26 11:36:19,824 - run_bot - INFO - Extracting and analyzing recommended dictionary for handshake-fuzzer.
2020-05-26 11:36:19,825 - run_bot - INFO - No recommended dictionary in output from handshake-fuzzer.
2020-05-26 11:36:19,825 - run_bot - INFO - Used strategies.
2020-05-26 11:36:20,287 - run_bot - INFO - Uploaded file to logs bucket.
2020-05-26 11:36:20,288 - run_bot - INFO - Uploaded file to logs bucket.

后来又分配给第一个bot了

一直没结果,我就换了一个简单的程序,命名项目为libfuzzer_asan_my_project

代码语言:javascript
代码运行次数:0
复制
#include <stdint.h>
#include <stdio.h>
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
  uint8_t tmp = Data[10];
  return 0;  // Non-zero return values are reserved for future use.
}

提交后,不一会就有minimize任务了

这个简单程序的结果就出来了

点进去可以查看更多信息

参考

https://google.github.io/clusterfuzz/ https://github.com/lookfwd/getting-started-clusterfuzz-local-in-aws

本文参与 腾讯云自媒体同步曝光计划,分享自作者个人站点/博客。
原始发表:2020-05-18,如有侵权请联系 cloudcommunity@tencent.com 删除

本文分享自 作者个人站点/博客 前往查看

如有侵权,请联系 cloudcommunity@tencent.com 删除。

本文参与 腾讯云自媒体同步曝光计划  ,欢迎热爱写作的你一起参与!

评论
登录后参与评论
0 条评论
热度
最新
推荐阅读
目录
  • 获取代码
  • 依赖
  • 环境搭建
  • 初始化
  • 实际fuzz实例
  • 参考
领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档