首页
学习
活动
专区
圈层
工具
发布
精选内容/技术社群/优惠产品,尽在小程序
立即前往
30 篇文章
1
ElasticSearch高级功能:Cross Cluster Replication实战
2
ElasticSearch压测工具:esrally离线使用详解
3
ElasticSearch实战:Kibana可视化
4
ElasticSearch实战:IK中文分词插件
5
ElasticSearch实战:将文本文件导入kibana
6
ElasticSearch实战:Linux日志对接Kibana
7
Elasticsearch:flattened 数据类型 (7.3 发行版新功能)
8
Elasticsearch: range 数据类型及基于range的聚合 (7.4发行版新功能)
9
腾讯云ES索引生命周期管理使用教程(视频)
10
腾讯云ES数据备份恢复使用教程(视频)
11
Elasticsearch:透彻理解 Elasticsearch 中的 Bucket aggregation
12
Elasticsearch:pipeline aggregation 介绍
13
Elasticsearch:Index 生命周期管理入门
14
Elastic Stack 7.7 Observability 新功能介绍
15
腾讯云中 Elastic Stack 的 Beats 部署最佳实践
16
海量挑战:腾讯云ES可用性及性能优化实践
17
Elasticsearch: Reindex接口
18
Elasticsearch:Java 运用示例
19
如何安装 Elastic 栈中的 Logstash
20
Logstash: 应用实践 - 装载 CSV 文档到 Elasticsearch
21
2分钟快速了解Elasticsearch
22
Elastic:Elasticsearch 的分片管理策略
23
Elasticsearch:如何把 Elasticsearch 中的数据导出为 CSV 格式的文件
24
Elasticsearch:Elasticsearch 中的 refresh 和 flush 操作指南
25
Kibana: 如何使用 Search Bar
26
Kibana:如何开始使用 Kibana
27
10分钟快速入门海量数据搜索分析引擎 Elasticsearch
28
腾讯万亿级 Elasticsearch 内存效率提升解密
29
腾讯Elasticsearch海量规模背后的内核优化剖析
30
【ElasticSearch性能测试】esrally最新版本的编译、安装与使用

【ElasticSearch性能测试】esrally最新版本的编译、安装与使用

【背景】

在使用esrally 1.4.1进行性能测试的过程中发现,如果使用geonames数据集,且ES集群版本是7.x,则测试进行到一半,就会报错,如:

代码语言:shell
复制

[ERROR] Cannot race. Error in load generator [0]
        ("Cannot execute [user-defined context-manager enabled runner for [query]]. Provided parameters are: ['index', 'type', 'cache', 'request-params', 'body']. Error: ['total'].", None)

Getting further help:
*********************
* Check the log files in /root/.rally/logs for errors.
* Read the documentation at https://esrally.readthedocs.io/en/1.4.1/
* Ask a question on the forum at https://discuss.elastic.co/c/elasticsearch/rally
* Raise an issue at https://github.com/elastic/rally/issues and include the log files in /root/.rally/logs.

查了很多资料,未找到解决办法。

遂怀疑,可能是esrally版本过老,无法适配ES 7.x。

因此,决定手工安装esrally最新版本。

下面以CentOS 7.4为例进行操作。

【安装】

1,下载

github上官方项目中提供最新版的下载链接:

代码语言:javascript
复制
https://github.com/elastic/rally/releases/

esrally最新版本下载

2,编译与安装

下载后,解压安装:

代码语言:shell
复制
 tar -zxvf esrally-dist-linux-2.0.2.tar.gz
 
 cd esrally-dist-2.0.2/
 
 ./install.sh

1)执行 ./install.sh

代码语言:shell
复制
[root@VM-0-115-centos /opt/esrally-dist-2.0.2]# ./install.sh 
Installing Rally 2.0.2...
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.
Collecting esrally==2.0.2
esrally requires Python '>=3.8,<3.9' but the running Python is 3.6.8

报错提示我们,esrally依赖 python 3.8.x 版本,而当前机器是3.6.8,因此,需要我们安装对应版本的python。

python安装教程很多,这里不多讲,命令供参考:

代码语言:shell
复制
yum install yum-utils
yum-builddep python
curl -O https://www.python.org/ftp/python/3.8.0/Python-3.8.0.tgz
tar xf Python-3.8.0.tgz
cd Python-3.8.0
./configure
make
make install

# 配置全局命令
rm /usr/bin/python3
ln -s /usr/local/bin/python3.8 /usr/bin/python3

# 检查是否安装完成

[root@VM-0-115-centos /usr/bin]# python3
Python 3.8.6 (default, Jan  5 2021, 20:26:42) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
[root@VM-0-115-centos /usr/bin]# 

完成python 3.8.6 安装后,重新执行 ./install.sh

代码语言:shell
复制
[root@VM-0-115-centos /opt]# cd esrally-dist-2.0.2/
[root@VM-0-115-centos /opt/esrally-dist-2.0.2]# ./install.sh 
Installing Rally 2.0.2...
Looking in links: file:///opt/esrally-dist-2.0.2/bin
Processing ./bin/esrally-2.0.2-py3-none-any.whl
Processing ./bin/ijson-2.6.1.tar.gz
Processing ./bin/tabulate-0.8.7-py3-none-any.whl
Processing ./bin/elasticsearch-7.9.1-py2.py3-none-any.whl
Processing ./bin/thespian-3.10.1.zip
Processing ./bin/py-cpuinfo-7.0.0.tar.gz
Processing ./bin/google_auth-1.22.1-py2.py3-none-any.whl
Processing ./bin/psutil-5.7.0.tar.gz
Processing ./bin/boto3-1.10.32-py2.py3-none-any.whl
Processing ./bin/Jinja2-2.10.3-py2.py3-none-any.whl
Processing ./bin/certifi-2020.6.20-py2.py3-none-any.whl
Processing ./bin/google_resumable_media-1.1.0-py2.py3-none-any.whl
Processing ./bin/yappi-1.2.3.tar.gz
Processing ./bin/jsonschema-3.1.1-py2.py3-none-any.whl
Processing ./bin/urllib3-1.25.11-py2.py3-none-any.whl
Processing ./bin/yarl-1.6.2-cp38-cp38-manylinux2014_x86_64.whl
Processing ./bin/aiohttp-3.7.2-cp38-cp38-manylinux2014_x86_64.whl
Processing ./bin/cachetools-4.1.1-py3-none-any.whl
Processing ./bin/six-1.15.0-py2.py3-none-any.whl
Processing ./bin/pyasn1_modules-0.2.8-py2.py3-none-any.whl
Requirement already satisfied: setuptools>=40.3.0 in /usr/local/lib/python3.8/site-packages (from google-auth==1.22.1->esrally==2.0.2) (49.2.1)
Processing ./bin/rsa-4.6-py3-none-any.whl
Processing ./bin/botocore-1.13.50-py2.py3-none-any.whl
Processing ./bin/jmespath-0.10.0-py2.py3-none-any.whl
Processing ./bin/s3transfer-0.2.1-py2.py3-none-any.whl
Processing ./bin/MarkupSafe-1.1.1.tar.gz
Processing ./bin/google_crc32c-1.0.0-cp38-cp38-manylinux2010_x86_64.whl
Processing ./bin/attrs-20.2.0-py2.py3-none-any.whl
Processing ./bin/pyrsistent-0.17.3.tar.gz
Processing ./bin/importlib_metadata-2.0.0-py2.py3-none-any.whl
Processing ./bin/idna-2.10-py2.py3-none-any.whl
Processing ./bin/multidict-5.0.0-cp38-cp38-manylinux2014_x86_64.whl
Processing ./bin/chardet-3.0.4-py2.py3-none-any.whl
Processing ./bin/typing_extensions-3.7.4.3-py3-none-any.whl
Processing ./bin/async_timeout-3.0.1-py3-none-any.whl
Processing ./bin/pyasn1-0.4.8-py2.py3-none-any.whl
Processing ./bin/python_dateutil-2.8.1-py2.py3-none-any.whl
Processing ./bin/docutils-0.15.2-py3-none-any.whl
Processing ./bin/cffi-1.14.3-cp38-cp38-manylinux1_x86_64.whl
Processing ./bin/zipp-3.4.0-py3-none-any.whl
Processing ./bin/pycparser-2.20-py2.py3-none-any.whl
Using legacy 'setup.py install' for ijson, since package 'wheel' is not installed.
Using legacy 'setup.py install' for thespian, since package 'wheel' is not installed.
Using legacy 'setup.py install' for py-cpuinfo, since package 'wheel' is not installed.
Using legacy 'setup.py install' for psutil, since package 'wheel' is not installed.
Using legacy 'setup.py install' for yappi, since package 'wheel' is not installed.
Using legacy 'setup.py install' for MarkupSafe, since package 'wheel' is not installed.
Using legacy 'setup.py install' for pyrsistent, since package 'wheel' is not installed.
Installing collected packages: ijson, tabulate, certifi, urllib3, idna, multidict, yarl, attrs, chardet, typing-extensions, async-timeout, aiohttp, elasticsearch, thespian, py-cpuinfo, cachetools, six, pyasn1, pyasn1-modules, rsa, google-auth, psutil, python-dateutil, jmespath, docutils, botocore, s3transfer, boto3, MarkupSafe, Jinja2, pycparser, cffi, google-crc32c, google-resumable-media, yappi, pyrsistent, zipp, importlib-metadata, jsonschema, esrally
    Running setup.py install for ijson ... done
    Running setup.py install for thespian ... done
    Running setup.py install for py-cpuinfo ... done
    Running setup.py install for psutil ... done
    Running setup.py install for MarkupSafe ... done
    Running setup.py install for yappi ... done
    Running setup.py install for pyrsistent ... done
Successfully installed Jinja2-2.10.3 MarkupSafe-1.1.1 aiohttp-3.7.2 async-timeout-3.0.1 attrs-20.2.0 boto3-1.10.32 botocore-1.13.50 cachetools-4.1.1 certifi-2020.6.20 cffi-1.14.3 chardet-3.0.4 docutils-0.15.2 elasticsearch-7.9.1 esrally-2.0.2 google-auth-1.22.1 google-crc32c-1.0.0 google-resumable-media-1.1.0 idna-2.10 ijson-2.6.1 importlib-metadata-2.0.0 jmespath-0.10.0 jsonschema-3.1.1 multidict-5.0.0 psutil-5.7.0 py-cpuinfo-7.0.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pyrsistent-0.17.3 python-dateutil-2.8.1 rsa-4.6 s3transfer-0.2.1 six-1.15.0 tabulate-0.8.7 thespian-3.10.1 typing-extensions-3.7.4.3 urllib3-1.25.11 yappi-1.2.3 yarl-1.6.2 zipp-3.4.0
[root@VM-0-115-centos /opt/esrally-dist-2.0.2]# 
[root@VM-0-115-centos /opt/esrally-dist-2.0.2]# 

至此,esrally 2.0.2完成安装。

我们可以使用下列命令查看安装是否成功:

代码语言:shell
复制
[root@VM-0-115-centos /opt/esrally-dist-2.0.2]# esrally --version
Traceback (most recent call last):
  File "/usr/local/bin/esrally", line 5, in <module>
    from esrally.rally import main
  File "/usr/local/lib/python3.8/site-packages/esrally/rally.py", line 27, in <module>
    from esrally import PROGRAM_NAME, BANNER, FORUM_LINK, SKULL, check_python_version, doc_link, telemetry
  File "/usr/local/lib/python3.8/site-packages/esrally/telemetry.py", line 25, in <module>
    from esrally import metrics, time, exceptions
  File "/usr/local/lib/python3.8/site-packages/esrally/metrics.py", line 33, in <module>
    from esrally import time, exceptions, config, version, paths
  File "/usr/local/lib/python3.8/site-packages/esrally/config.py", line 27, in <module>
    from esrally.utils import io, git, console, convert
  File "/usr/local/lib/python3.8/site-packages/esrally/utils/io.py", line 18, in <module>
    import bz2
  File "/usr/local/lib/python3.8/bz2.py", line 19, in <module>
    from _bz2 import BZ2Compressor, BZ2Decompressor
ModuleNotFoundError: No module named '_bz2'

很明显,又报错了。提示缺少 _bz2 之类的python包。

这里可以使用一个巧妙的办法---用python 3.6的包。

代码语言:shell
复制
[root@VM-0-115-centos /opt/esrally-dist-2.0.2]# find / -name _bz2*
/usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so
…
…

我们可以将上面这个包,拷贝到python 3.8的对应路径并重命名:

代码语言:shell
复制
[root@VM-0-115-centos /]# cp /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so /usr/local/lib/python3.8/lib-dynload
[root@VM-0-115-centos /]# 
[root@VM-0-115-centos /]# cd /usr/local/lib/python3.8/lib-dynload
[root@VM-0-115-centos /usr/local/lib/python3.8/lib-dynload]# mv _bz2.cpython-36m-x86_64-linux-gnu.so _bz2.cpython-38-x86_64-linux-gnu.so 
[root@VM-0-115-centos /usr/local/lib/python3.8/lib-dynload]# ll | grep _bz2
-rwxr-xr-x 1 root root   23056 Jan  5 21:01 _bz2.cpython-38-x86_64-linux-gnu.so 

这样一来,python3.8就不再缺少_bz2这个包了。

我们继续使用下列命令查看 esrally是否安装完成:

代码语言:shell
复制
[root@VM-0-115-centos /usr/local/lib/python3.8/lib-dynload]# esrally --version
esrally 2.0.2

这里可以看出,命令回显正常,可以开始使用esrally的最新版本进行性能测试了。

【使用】

可参考文章《ElasticSearch压测工具:esrally离线使用详解》

【最终效果】

使用esrally 2.0.2对ElasticSearch 7.5.1集群进行性能测试结果如下:

代码语言:javascript
复制
时间:

20210105 1912


组网方式:

CVM ---> VPC ---> ES

ES集群配置:
略


数据集:

geonames


启动命令:

esrally race --pipeline=benchmark-only --target-hosts=172.16.0.40:9200 --track=geonames  --client-options="basic_auth_user:'elastic',basic_auth_password:'yourpassword'" --challenge=append-no-conflicts  --offline


[root@VM-0-72-centos /usr/lib64]# esrally race --pipeline=benchmark-only --target-hosts=172.16.0.40:9200 --track=geonames  --client-options="basic_auth_user:'elastic',basic_auth_password:'yourpassword'" --challenge=append-no-conflicts  --offline

    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

[INFO] You did not provide an explicit timeout in the client options. Assuming default of 10 seconds.
[INFO] Racing on track [geonames], challenge [append-no-conflicts] and car ['external'] with version [7.5.1].

[WARNING] merges_total_time is 405627 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] merges_total_throttled_time is 87629 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] indexing_total_time is 1062620 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] refresh_total_time is 74113 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] flush_total_time is 18703 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
Running delete-index                                                           [100% done]
Running create-index                                                           [100% done]
Running check-cluster-health                                                   [100% done]
Running index-append                                                           [100% done]
Running refresh-after-index                                                    [100% done]
Running force-merge                                                            [100% done]
Running refresh-after-force-merge                                              [100% done]
Running wait-until-merges-finish                                               [100% done]
Running index-stats                                                            [100% done]
Running node-stats                                                             [100% done]
Running default                                                                [100% done]
Running term                                                                   [100% done]
Running phrase                                                                 [100% done]
Running country_agg_uncached                                                   [100% done]
Running country_agg_cached                                                     [100% done]
Running scroll                                                                 [100% done]
Running expression                                                             [100% done]
Running painless_static                                                        [100% done]
Running painless_dynamic                                                       [100% done]
Running decay_geo_gauss_function_score                                         [100% done]
Running decay_geo_gauss_script_score                                           [100% done]
Running field_value_function_score                                             [100% done]
Running field_value_script_score                                               [100% done]
Running large_terms                                                            [100% done]
Running large_filtered_terms                                                   [100% done]
Running large_prohibited_terms                                                 [100% done]
Running desc_sort_population                                                   [100% done]
Running asc_sort_population                                                    [100% done]
Running asc_sort_with_after_population                                         [100% done]
Running desc_sort_geonameid                                                    [100% done]
Running desc_sort_with_after_geonameid                                         [100% done]
Running asc_sort_geonameid                                                     [100% done]
Running asc_sort_with_after_geonameid                                          [100% done]

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------
            
|                                                         Metric |                           Task |      Value |    Unit |
|---------------------------------------------------------------:|-------------------------------:|-----------:|--------:|
|                     Cumulative indexing time of primary shards |                                |    18.1194 |     min |
|             Min cumulative indexing time across primary shards |                                |          0 |     min |
|          Median cumulative indexing time across primary shards |                                |          0 |     min |
|             Max cumulative indexing time across primary shards |                                |    3.83658 |     min |
|            Cumulative indexing throttle time of primary shards |                                |          0 |     min |
|    Min cumulative indexing throttle time across primary shards |                                |          0 |     min |
| Median cumulative indexing throttle time across primary shards |                                |          0 |     min |
|    Max cumulative indexing throttle time across primary shards |                                |          0 |     min |
|                        Cumulative merge time of primary shards |                                |    7.03953 |     min |
|                       Cumulative merge count of primary shards |                                |        612 |         |
|                Min cumulative merge time across primary shards |                                |          0 |     min |
|             Median cumulative merge time across primary shards |                                |          0 |     min |
|                Max cumulative merge time across primary shards |                                |    1.29192 |     min |
|  
|                                        50th percentile latency |  asc_sort_with_after_geonameid |    65.8707 |      ms |
|                                        90th percentile latency |  asc_sort_with_after_geonameid |    69.9126 |      ms |
|                                        99th percentile latency |  asc_sort_with_after_geonameid |    79.9802 |      ms |
|                                       100th percentile latency |  asc_sort_with_after_geonameid |     105.06 |      ms |
|                                   50th percentile service time |  asc_sort_with_after_geonameid |    64.7016 |      ms |
|                                   90th percentile service time |  asc_sort_with_after_geonameid |    68.9187 |      ms |
|                                   99th percentile service time |  asc_sort_with_after_geonameid |    77.6204 |      ms |
|                                  100th percentile service time |  asc_sort_with_after_geonameid |    104.128 |      ms |
|                                                     error rate |  asc_sort_with_after_geonameid |          0 |       % |
……
……
# 内容较多,略
……
……[WARNING] No throughput metrics available for [index-append]. Likely cause: The benchmark ended already during warmup.

----------------------------------
[INFO] SUCCESS (took 3896 seconds)
----------------------------------
[root@VM-0-72-centos /usr/lib64]# 
举报
领券