MongoDB 非分片集合转分片集合

好的分片键通常满足如下特征:
   1、分片键基数高、低频率
   2、写请求能够均衡分布
   3、大部分查询路由到目标分片而非广播

1、非空集合的分片键需要预先创建索引,否则无法将非分片集合转成分片集合，
此操作不可逆,分片集合不能转成非分片集合
2、非分片集合转成分片,根据采用chunk size以及文档平均大小来决定非分片集合
最大值,例如分片键平均是64字节时采用默认64M chunk,支持最大8TB的集合.参考
如下图.
3、原应用操作的是非分片集合,需要注意插入、更新、删除分片键问题,否则转换后会
导致应用报错，例如插入不带分片键的文档，更新采用upsert方式以及
findandmodify必须带分片键才能执行.
4、如果非分片转分片时,提示couldn't find valid index for shard key:
1、分片键是索引可能是多key索引，例如数组 2、遇到bug.
5、4.4版本支持插入不带分片键的文档,分片键对应值为null.4.4版本之前必须
带完整的分片键.
6、非分片转换成分片集合,mongo使用writeConcern是majority级别.

1、执行过程会检查分片键索引是否存在
2、执行checkShardingIndex来检查索引是否满足完整分片键 
如下命令就是检查索引是否满足条件的语句
db.runCommand({"checkShardingIndex":"xiaoxu.POCCOLL",
"keyPattern":{"callP":"hashed"}})
3、对每一条记录都进行验证(如果大表可能执行时间比较久且存在表大小限制问题)
4、更新config.collections信息
5、创建初始化chunk
其中checkShardingIndex是很关键步骤,需要对对每一条记录都要检查.

1、集合中分片键不存在索引,需要预先创建好分片键索引.
2、非空集合根据chunk大小不同,对原集合有大小限制
3、mongos执行sh.shardCollection(namespace,{分片键信息})
例如hash分片
sh.shardCollection("xiaoxu.POCCOLL",{"callP":"hashed"})
例如range分片
sh.shardCollection("xiaoxu.POCCOLL",{"callP":1})
备注:如果是range分片,通常采用组合分片键，4.4版本支持range+hash组合方式

"Please create an index that starts with the proposed shard 
key before sharding the collection"

【解决方案】
db.POCCOLL.createIndex({"callP":"hashed"})
sh.shardCollection("xiaoxu.POCCOLL",{"callP":"hashed"})

”There are documents which have missing or incomplete shard 
key fields ({ : 2338878944348059895 }). Please ensure that 
all documents in the collection include all fields from 
the shard key."

【解决方案--处理有问题都数据】
备注：对应应用必须更改,后续插入文档中包括分片字段,否则转换后应用报错
1、找到文档填充分片字段或者删除文档
db.POCCOLL.find({callP:{$exists:false}})
{ "_id" : ObjectId("607f830825795fbf8cc40f7b"), "name" : "xiaojing", "address" : "shanghai" }
{ "_id" : ObjectId("607f831125795fbf8cc40f7c"), "name" : "xiaoxing", "address" : "shanghai" }

【例如删除记录】
mongos> db.POCCOLL.remove({callP:{$exists:false}})
WriteResult({ "nRemoved" : 2 })

【例如更新记录】--更新需要按需
mongos>db.POCCOLL.update({callP:{$exists:false}},
{$set:{callP:""})

2、如果索引创建有问题或者分片j键不符合要求，比如是多key索引直接提示如下错误
"couldn't find valid index for shard key"

{multi:false} update on a sharded collection must either 
contain an exact match on _id or must target a single shard 
but this update targeted _id (and have the collection default 
collation) or must target a single shard (and have the simple 
collation), but this update targeted 2 shards.
备注:如果_id不是分片键,更新操作同样是下发到所有分片,可能会存在更新多条问题
,即使是multi:false也不生效.
因为_id是非分片键时,只能保证单个分片内值是唯一,而不是全局唯一。

Shard key update is not allowed without specifying the full 
shard key in the query

mongos>db.POCCOLL.update({"address":"jiangsu",callP:null},
{callP:1771235175,name:"xiaoxu",address:"jiuting"},
{upsert:true})
WriteResult({
  "nMatched" : 0,
  "nUpserted" : 1,
  "nModified" : 0,
  "_id" : ObjectId("607fb561be529318222c78cf")
})

"document { _id: ObjectId('607fb8ebd432266ecaf9e44a'), 
name: \"nanjing\", address: \"nnajing\" } 
does not contain shard key for 
pattern { callP: \"hashed\" }"

mongos> db.students2.remove({},{justOne:true})
WriteResult({
  "nRemoved" : 0,
  "writeError" : {
    "code" : 61,
    "errmsg" : "A single delete on a sharded collection must 
    contain an exact match on _id (and have the collection 
    default collation) or contain the shard key (and have the 
    simple collation). Delete request: { q: {}, limit: 1 }, 
    shard key pattern: { _id: 1.0 }"
  }
})

mongos>db.POCCOLL.findAndModify({query:{name:"xiaojing"},
update:{$set:{address:"jiuting"}}})
2021-04-23T10:15:33.316+0800 E  QUERY  
[js] uncaught exception: Error: 
findAndModifyFailed failed: {
"ok" : 0,
"errmsg" : "Query for sharded findAndModify must
contain the shard key",
"code" : 61,
"codeName" : "ShardKeyNotFound",

备注: 4.2版本支持修改分片键,但分片键是_id不能修改.
1、通过mongs去操作
2、必须在事务内或使用retryWrites参数
3、multi等于false，一次执行修改一条记录
4、必须包括完整shard key在查询条件中

mongos>db.POCCOLL.update({callP:{$gt:18849795174}}，
{$set:{callP:123456789}})
WriteResult({
"nMatched" : 0,
"nUpserted" : 0,
"nModified" : 0,
"writeError" : {
"code" : 72,
"errmsg" : "A {multi:false} update on a sharded 
collection must either contain an exact match 
on _id or must target a single shard but this update 
targeted _id (and have the collection default 
collation) or must target a single shard 
(and have the simple collation), 
but this update targeted 2 shards. 
Update request: { q: { callP: 
{ $gt: 18849795174.0 } }, 
u: { $set: { callP: 123456789.0 } }, 
multi: false, upsert: false }, 
shard key pattern: { callP: \"hashed\" }"
}
})
mongos>db.POCCOLL.update({callP:18849795174},{$set:
{callP:123456789}})
WriteResult({
"nMatched" : 0,
"nUpserted" : 0,
"nModified" : 0,
"writeError" : {
"code" : 20,
"errmsg" : "Must run update to shard key field 
in a multi-statement transaction or with 
retryWrites: true."
}
})


mongos>db.POCCOLL.update({"_id" : ObjectId(
"607fd5d8d2a6e53cbdfd21fd")},
{$set:{callP:1889785174}})
WriteResult({
"nMatched" : 0,
"nUpserted" : 0,
"nModified" : 0,
"writeError" : {
"code" : 31025,
"errmsg" : "Shard key update is not allowed 
without specifying the full shard key in 
the query"}
})


mongo --port 21051 -uadmin --retryWrites
mongos>db.POCCOLL.update({callP:18849795174},
{$set:{callP:123456789}})
WriteResult({ "nMatched" : 1, "nUpserted" : 0, 
"nModified" : 1 })

mongos>db.POCCOLL.update({callP:18849795174},
{$set:{callP:123456789}},{multi:true})
WriteResult({
"nMatched" : 0,
"nUpserted" : 0,
"nModified" : 0,
"writeError" : {
"code" : 72,
"errmsg" : "Multi-update operations are not 
allowed when updating the shard key field."
}
})