本文描述问题及解决方法同样适用于 腾讯云 Elasticsearch Service(ES)。
Elasticsearch 源端版本:7.17.7
Elasticsearch 目标版本:7.14.2
一般情况下,ES 迁移我们会选择等于或者是大于源端的版本,这是因为版本差距只要不是特别大,高版本对低版本通常都是兼容的。
不过在实际使用中,可能还会有其他特殊的情况。比如客户需要迁移上云,但目标版本的选择性有限,客户的自建版本比较高,而腾讯云 Elaticsearch Service 7.x 的最高版本为 7.14.2,所以就涉及到高版本迁移到低版本的需求。
有两个方案可以实施,我们带着实践的目的和大家一起验证这两个方案。
正常情况下,kibana 自带的对象导出/导入功能是 kibana 迁移的最佳选择,该功能的 UI 路径为:Management - Stack Management - Kibana - Saved Objects
这个功能操作起来很简单,我们只需在源端集群的 kibana 该界面勾选我们需要同步的对象,然后一键导出成文件,再到目标集群的 kibana 界面导入文件即可。
但很显然,不出意外的话,就出意外了。
对于 ES 7.17.7 这个版本导出的 kibana 对象,ES 7.14.1 是无法读取的,这个实际上是有两个原因:
综上,kibana 对象的导出/导入功能,无法实现 kibana 高版本迁移到低版本的需求。
实现该方案有两个办法:
下面我们来分别验证一下这两个方案。
1. 同步索引属性
首先我们使用elasticsearch-dump同步索引 mapping 属性
elasticsearch-dump 是一款开源工具,安装方式很简单:
npm install elasticdump -g
安装完成之后,就可以进行 mapping 同步了:
# 同步mapping
elasticdump --input='1.1.1.1:9200/.kibana_7.17.7_001' --output='2.2.2.2:9200/.kibana_7.17.7_001' --type=mapping
mapping 同步完之后,我们再同步一下数据:
# 同步data
elasticdump --input='1.1.1.1:9200/.kibana_7.17.7_001' --output='2.2.2.2:9200/.kibana_7.17.7_001' --type=data --limit 5000
elasticdump 更多细节问题可以参考另一篇详细介绍数据迁移的文档:Elasticsearch数据离线迁移方案(含冷热分层场景)
2. reindex 索引数据
POST _reindex?slices=auto
{
"source": {
"index": ".kibana_7.17.7_001"
},
"dest": {
"index": ".kibana_7.14.2_001"
}
}
不出意外的话,又出意外了:
{ -
"took": 876,
"timed_out": false,
"total": 7150,
"updated": 1,
"created": 284,
"deleted": 0,
"batches": 1,
"version_conflicts": 0,
"noops": 0,
"retries": { -
"bulk": 0,
"search": 0
},
"throttled_millis": 0,
"requests_per_second": -1,
"throttled_until_millis": 0,
"failures": [ -
{ -
"index": ".kibana_7.14.2_001",
"type": "_doc",
"id": "url:ae3f48e0-c3ea-11ed-b60f-f7d2a21ef73b",
"cause": { -
"type": "strict_dynamic_mapping_exception",
"reason": "mapping set to strict, dynamic introduction of [slug] within [url] is not allowed"
},
"status": 400
},
// 此处省略7130行
{ -
"index": ".kibana_7.14.2_001",
"type": "_doc",
"id": "url:9e2adac0-acff-11ed-b244-01878a062024",
"cause": { -
"type": "mapper_parsing_exception",
"reason": "doc early terminate:org.elasticsearch.index.mapper.StrictDynamicMappingException: mapping set to strict, dynamic introduction of [slug] within [url] is not allowed"
},
"status": 400
}
]
}
前面提到过,因为 kibana 系统索引默认是不允许增加字段,所以我们还需要设置一个动态参数:
PUT .kibana_7.14.2_001/_mapping
{
"dynamic": "true"
}
设置完之后,重新 reindex 成功:
{ -
"took": 3323,
"timed_out": false,
"total": 7150,
"updated": 324,
"created": 6826,
"deleted": 0,
"batches": 8,
"version_conflicts": 0,
"noops": 0,
"retries": { -
"bulk": 0,
"search": 0
},
"throttled_millis": 0,
"requests_per_second": -1,
"throttled_until_millis": 0,
"failures": [ -
]
}
打开 kibana 访问也都正常:
事情到这里好像已经圆满的画上了句号,直到 —— kibana 重启。
kibana 的重启在正常使用中无法避免,比如我们需要修改界面语言、调整 kibana 的 csv 导出限制参数,升级硬件配置,再或者是 kibana 自身进程崩溃发生的重启等等,这些都可能经常遇到。而一旦 kibana 发生重启,我们上面做的一切,都将功亏一篑。
为了搞清楚其中的原因,我们对 kibana 的日志进行了分析。我们发现 kibana 在启动时,kibana 索引会有迁移环节,我们也可以把这个环节理解为自检环节。该环节 kibana 会对存量系统索引别名 .kibana 进行 reindex,在 6.x 中 该环节不会进行元数据检查,而在高版本 7.14 中,该环节会先对 .kibana 进行元数据检查,当索引结构不符合预期时,会 block reindex 操作,kibana 也就会处于宕机状态。
重启之后就是因为元数据自检这个环节通不过,所以导致kibana启动失败,在 kibana 日志中具体的报错为:
{"type":"log","@timestamp":"2023-06-06T10:42:12+08:00","tags":["error","savedobjects-service"],"pid":746,"message":"[.kibana] migration failed, dumping execution log:"}
{"type":"log","@timestamp":"2023-06-06T10:42:12+08:00","tags":["fatal","root"],"pid":746,"message":"Error: Unable to complete saved object migrations for the [.kibana] index: Migrations failed. Reason: 382 transformation errors were encountered:\n - visualization:228b1e70-282f-11ed-9c10-0d1563a9512a: Error: Document \"228b1e70-282f-11ed-9c10-0d1563a9512a\" has property \"visualization\" which belongs to a more recent version of Kibana [7.17.0]. The last known version is [7.14.0]\n at /usr/local/service/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:602:27\n at Array.find (<anonymous>)\n at nextUnmigratedProp (/usr/local/service/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:593:21)\n at applyMigrations (/usr/local/service/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:340:18)\n at DocumentMigrator.transformAndValidate [as transformDoc] at processImmediate (internal/timers.js:464:21)\n\nTo allow migrations to proceed, please delete or fix these documents.\n at migrationStateActionMachine (/usr/local/service/kibana/src/core/server/saved_objects/migrationsv2/migrations_state_action_machine.js:155:29)\n at processTicksAndRejections (internal/process/task_queues.js:95:5)\n at async Promise.all (index 0)\n at SavedObjectsService.start (/usr/local/service/kibana/src/core/server/saved_objects/saved_objects_service.js:164:7)\n at Server.start (/usr/local/service/kibana/src/core/server/server.js:274:31)\n at Root.start (/usr/local/service/kibana/src/core/server/root/index.js:55:14)\n at bootstrap (/usr/local/service/kibana/src/core/server/bootstrap.js:98:5)\n at Command.<anonymous> (/usr/local/service/kibana/src/cli/serve/serve.js:224:5)"}
通过日志统计以及元数据的对比,得出以下结论:
为了解决这两个问题,我们需要做以下操作:
1. remove 这些字段
migrationVersion.visualization
migrationVersion.dashboard
migrationVersion.canvas-workpad
migrationVersion.lens
migrationVersion.map
migrationVersion.query
migrationVersion.search
migrationVersion.upgrade-assistant-telemetry
reindex 的同时实现 remove 字段,则需要改造一下 reindex 参数:
POST _reindex?slices=auto
{
"source": {
"index": ".kibana_7.17.7_001",
"_source": {
"excludes": [
"migrationVersion.visualization",
"migrationVersion.dashboard",
"migrationVersion.canvas-workpad",
"migrationVersion.lens",
"migrationVersion.map",
"migrationVersion.query",
"migrationVersion.search",
"migrationVersion.upgrade-assistant-telemetry"
]
}
},
"dest": {
"index": ".kibana_7.14.2_001"
}
}
2. 调整字段的值
coreMigrationVersion
字段的值全部修改 为 7.14.2。
实现将一个索引的字段全部改成一个值,只需要 update_by_query + script 即可:
POST .kibana_7.14.2_001/_update_by_query
{
"query": {
"match_all": {}
},
"script": {
"source": "ctx._source.coreMigrationVersion = params.newValue",
"params": {
"newValue": "7.14.2"
}
}
}
最后梳理一下 reindex 方案的整个流程:
1. es dump 同步 mapping 以及 data;
2. reindex 并 exclude 8个字段;
3. update_by_query 修改字段值。
有了上面的经验,那 logstash 就简单多了,我们配置一个 logstash pipeline即可:
input {
elasticsearch {
hosts => "1.1.1.1:9200"
index => ".kibana_7.17.7_001"
docinfo => true
size => 5000
scroll => "5m"
}
}
filter {
mutate {
replace => { "coreMigrationVersion" => "7.14.2" }
}
mutate {
remove_field => [ "migrationVersion.visualization", "migrationVersion.dashboard", "migrationVersion.canvas-workpad", "migrationVersion.lens", "migrationVersion.map", "migrationVersion.query", "migrationVersion.search", "migrationVersion.upgrade-assistant-telemetry" ]
}
}
output {
elasticsearch {
hosts => ["http://2.2.2.2:9200"]
index => ".kibana_7.14.2_001"
document_type => "%{[@metadata][_type]}"
document_id => "%{[@metadata][_id]}"
}
}
实际操作中也可能会遇到如下报错:
{"type":"log","@timestamp":"2023-07-10T16:10:58+08:00","tags":["fatal","root"],"pid":16472,"message":"Error: Unable to complete saved object migrations for the [.kibana] index: Migrations failed. Reason: 1 transformation errors were encountered:\n - lens:xx-xx-11ed-xx-xx: Error: Migration function for version 7.12.0 threw an error\n at tryTransformDoc (/usr/local/service/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:564:13)\n at migrateProp (/usr/local/service/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:633:22)\n at applyMigrations (/usr/local/service/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:353:20)\n at DocumentMigrator.transformAndValidate [as transformDoc] (/usr/local/service/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:293:22)\n at /usr/local/service/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:101:16\n at Immediate._onImmediate (/usr/local/service/kibana/src/core/server/saved_objects/migrations/core/migrate_raw_docs.js:158:17)\n at processImmediate (internal/timers.js:464:21)\nCaused by:\nTypeError: Cannot read property '0' of undefined\n at transformTableState (/usr/local/service/kibana/x-pack/plugins/lens/server/migrations/saved_object_migrations.js:244:32)\n at tryTransformDoc (/usr/local/service/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:551:22)\n at migrateProp (/usr/local/service/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:633:22)\n at applyMigrations (/usr/local/service/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:353:20)\n at DocumentMigrator.transformAndValidate [as transformDoc] (/usr/local/service/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:293:22)\n at /usr/local/service/kibana/src/core/server/saved_objects/migrations/core/document_migrator.js:101:16\n at Immediate._onImmediate (/usr/local/service/kibana/src/core/server/saved_objects/migrations/core/migrate_raw_docs.js:158:17)\n at processImmediate (internal/timers.js:464:21)\n\nTo allow migrations to proceed, please delete or fix these documents.\n at migrationStateActionMachine (/usr/local/service/kibana/src/core/server/saved_objects/migrationsv2/migrations_state_action_machine.js:155:29)\n at processTicksAndRejections (internal/process/task_queues.js:95:5)\n at async Promise.all (index 0)\n at SavedObjectsService.start (/usr/local/service/kibana/src/core/server/saved_objects/saved_objects_service.js:164:7)\n at Server.start (/usr/local/service/kibana/src/core/server/server.js:274:31)\n at Root.start (/usr/local/service/kibana/src/core/server/root/index.js:55:14)\n at bootstrap (/usr/local/service/kibana/src/core/server/bootstrap.js:98:5)\n at Command.<anonymous> (/usr/local/service/kibana/src/cli/serve/serve.js:224:5)"}我们需要关注的这一段报错信息:
这个同样会导致kibana无法启动:
我们只需要关注关键报错信息:
1 transformation errors were encountered:\n - lens:2692d220-xx-11ed-xx-xxx: Error: Migration function for version 7.12.0 threw an error
解决方案是删除报错的 doc id:
DELETE .kibana_7.14.2_001/_doc/lens:2692d220-xx-11ed-xx-xxx
reindex 和 logstash 都可以完美解决元数据冲突导致的 kibana 自检不通过问题,支持重启自检。
通过以上的实践,我们了解了 kibana 的启动逻辑,并使用 ES 数据迁移的知识对有冲突的元数据进行修复处理,最终达到了 kibana 高版本迁移到低版本的目的。
原创声明:本文系作者授权腾讯云开发者社区发表,未经许可,不得转载。
如有侵权,请联系 cloudcommunity@tencent.com 删除。
原创声明:本文系作者授权腾讯云开发者社区发表,未经许可,不得转载。
如有侵权,请联系 cloudcommunity@tencent.com 删除。