I set up a mongodb with 3 shards by using bitnami's chart: mongodb-sharded and I got two problems.
My sharding collection has more than 38 millions records.
The first problem is some mongodb pods take a lot of memory, more than 10G from the whole memory of 32G in a node. I found some similar issues from stackoverflow and I would try to tackle this problem.
The other problem is: when I do a search which matches the sharding key, it's very slow and sometimes it will be time out.
Here is my collection's indexes:
ice-shard-mongodb-sharded-shard-0:PRIMARY> db.LogInfo.getIndexes()
[
{
"v" : 2,
"key" : {
"_id" : 1
},
"name" : "_id_",
"collation" : {
"locale" : "en_US",
"caseLevel" : false,
"caseFirst" : "off",
"strength" : 2,
"numericOrdering" : false,
"alternate" : "non-ignorable",
"maxVariable" : "punct",
"normalization" : false,
"backwards" : false,
"version" : "57.1"
}
},
{
"v" : 2,
"key" : {
"Properties.Time" : 1
},
"name" : "Properties.Time_1",
"collation" : {
"locale" : "en_US",
"caseLevel" : false,
"caseFirst" : "off",
"strength" : 2,
"numericOrdering" : false,
"alternate" : "non-ignorable",
"maxVariable" : "punct",
"normalization" : false,
"backwards" : false,
"version" : "57.1"
}
},
{
"v" : 2,
"key" : {
"Properties.Plot" : 1,
"Properties.Time" : 1
},
"name" : "Properties.Plot_1_Properties.Time_1",
"collation" : {
"locale" : "en_US",
"caseLevel" : false,
"caseFirst" : "off",
"strength" : 2,
"numericOrdering" : false,
"alternate" : "non-ignorable",
"maxVariable" : "punct",
"normalization" : false,
"backwards" : false,
"version" : "57.1"
}
},
{
"v" : 2,
"key" : {
"Properties.Scenario" : 1,
"Properties.Time" : 1
},
"name" : "Properties.Scenario_1_Properties.Time_1",
"collation" : {
"locale" : "en_US",
"caseLevel" : false,
"caseFirst" : "off",
"strength" : 2,
"numericOrdering" : false,
"alternate" : "non-ignorable",
"maxVariable" : "punct",
"normalization" : false,
"backwards" : false,
"version" : "57.1"
}
},
{
"v" : 2,
"key" : {
"Properties.SecsFunction" : 1,
"Properties.Time" : 1
},
"name" : "Properties.SecsFunction_1_Properties.Time_1",
"collation" : {
"locale" : "en_US",
"caseLevel" : false,
"caseFirst" : "off",
"strength" : 2,
"numericOrdering" : false,
"alternate" : "non-ignorable",
"maxVariable" : "punct",
"normalization" : false,
"backwards" : false,
"version" : "57.1"
}
},
{
"v" : 2,
"key" : {
"Properties.EapId" : "hashed",
"Properties.Time" : 1
},
"name" : "Properties.EapId_hashed_Properties.Time_1"
}
]
The key "Properties.EapId_hashed_Properties.Time_1" is the sharding key.
If I search
db.LogInfo.find({"Properties.SecsFunction": "s6f11", "Properties.Time": {$gte: ISODate("2021-03-10")}})
This search matches an index and it will search all the 3 shards. It's very fast within 0.01 seconds.
But if I search
db.LogInfo.find(
{
"Properties.EapId": "12eap012",
"Properties.Time": {
"$gte": ISODate("2021-03-17")
}
}
)
which should use the sharding key, I think, it's very flow. Sometimes it will take more than 5 seconds and sometimes it will be time out. I've checked the indexes several times but I can't figure out why.
Please advise. Thanks.
update: Mongodb version: 4.4.3
update: shards are built as replica set = 3.
update: below is the query plan for the problematic search. (I drop the executionStat() due to stackoverflow's 30000 characters limit.)
{
"queryPlanner" : {
"mongosPlannerVersion" : NumberInt(1),
"winningPlan" : {
"stage" : "SHARD_MERGE",
"shards" : [
{
"shardName" : "ice-shard-mongodb-sharded-shard-1",
"connectionString" : "ice-shard-mongodb-sharded-shard-1/ice-shard-mongodb-sharded-shard1-data-0.ice-shard-mongodb-sharded-headless.ice-system.svc.cluster.local:27017,ice-shard-mongodb-sharded-shard1-data-1.ice-shard-mongodb-sharded-headless.ice-system.svc.cluster.local:27017,ice-shard-mongodb-sharded-shard1-data-2.ice-shard-mongodb-sharded-headless.ice-system.svc.cluster.local:27017",
"serverInfo" : {
"host" : "ice-shard-mongodb-sharded-shard1-data-1",
"port" : NumberInt(27017),
"version" : "4.4.3",
"gitVersion" : "913d6b62acfbb344dde1b116f4161360acd8fd13"
},
"plannerVersion" : NumberInt(1),
"namespace" : "LogCenter.LogInfo",
"indexFilterSet" : false,
"parsedQuery" : {
"$and" : [
{
"Properties.EapId" : {
"$eq" : "12eap012"
}
},
{
"Properties.Time" : {
"$gte" : ISODate("2021-03-17T00:00:00.000+0000")
}
}
]
},
"collation" : {
"locale" : "en_US",
"caseLevel" : false,
"caseFirst" : "off",
"strength" : NumberInt(2),
"numericOrdering" : false,
"alternate" : "non-ignorable",
"maxVariable" : "punct",
"normalization" : false,
"backwards" : false,
"version" : "57.1"
},
"winningPlan" : {
"stage" : "SHARDING_FILTER",
"inputStage" : {
"stage" : "FETCH",
"filter" : {
"Properties.EapId" : {
"$eq" : "12eap012"
}
},
"inputStage" : {
"stage" : "IXSCAN",
"keyPattern" : {
"Properties.Time" : 1.0
},
"indexName" : "Properties.Time_1",
"collation" : {
"locale" : "en_US",
"caseLevel" : false,
"caseFirst" : "off",
"strength" : NumberInt(2),
"numericOrdering" : false,
"alternate" : "non-ignorable",
"maxVariable" : "punct",
"normalization" : false,
"backwards" : false,
"version" : "57.1"
},
"isMultiKey" : false,
"multiKeyPaths" : {
"Properties.Time" : [
]
},
"isUnique" : false,
"isSparse" : false,
"isPartial" : false,
"indexVersion" : NumberInt(2),
"direction" : "forward",
"indexBounds" : {
"Properties.Time" : [
"[new Date(1615939200000), new Date(9223372036854775807)]"
]
}
}
}
},
"rejectedPlans" : [
]
},
{
"shardName" : "ice-shard-mongodb-sharded-shard-0",
"connectionString" : "ice-shard-mongodb-sharded-shard-0/ice-shard-mongodb-sharded-shard0-data-0.ice-shard-mongodb-sharded-headless.ice-system.svc.cluster.local:27017,ice-shard-mongodb-sharded-shard0-data-1.ice-shard-mongodb-sharded-headless.ice-system.svc.cluster.local:27017,ice-shard-mongodb-sharded-shard0-data-2.ice-shard-mongodb-sharded-headless.ice-system.svc.cluster.local:27017",
"serverInfo" : {
"host" : "ice-shard-mongodb-sharded-shard0-data-0",
"port" : NumberInt(27017),
"version" : "4.4.3",
"gitVersion" : "913d6b62acfbb344dde1b116f4161360acd8fd13"
},
"plannerVersion" : NumberInt(1),
"namespace" : "LogCenter.LogInfo",
"indexFilterSet" : false,
"parsedQuery" : {
"$and" : [
{
"Properties.EapId" : {
"$eq" : "12eap012"
}
},
{
"Properties.Time" : {
"$gte" : ISODate("2021-03-17T00:00:00.000+0000")
}
}
]
},
"collation" : {
"locale" : "en_US",
"caseLevel" : false,
"caseFirst" : "off",
"strength" : NumberInt(2),
"numericOrdering" : false,
"alternate" : "non-ignorable",
"maxVariable" : "punct",
"normalization" : false,
"backwards" : false,
"version" : "57.1"
},
"winningPlan" : {
"stage" : "SHARDING_FILTER",
"inputStage" : {
"stage" : "FETCH",
"filter" : {
"Properties.EapId" : {
"$eq" : "12eap012"
}
},
"inputStage" : {
"stage" : "IXSCAN",
"keyPattern" : {
"Properties.Time" : 1.0
},
"indexName" : "Properties.Time_1",
"collation" : {
"locale" : "en_US",
"caseLevel" : false,
"caseFirst" : "off",
"strength" : NumberInt(2),
"numericOrdering" : false,
"alternate" : "non-ignorable",
"maxVariable" : "punct",
"normalization" : false,
"backwards" : false,
"version" : "57.1"
},
"isMultiKey" : false,
"multiKeyPaths" : {
"Properties.Time" : [
]
},
"isUnique" : false,
"isSparse" : false,
"isPartial" : false,
"indexVersion" : NumberInt(2),
"direction" : "forward",
"indexBounds" : {
"Properties.Time" : [
"[new Date(1615939200000), new Date(9223372036854775807)]"
]
}
}
}
},
"rejectedPlans" : [
]
},
{
"shardName" : "ice-shard-mongodb-sharded-shard-2",
"connectionString" : "ice-shard-mongodb-sharded-shard-2/ice-shard-mongodb-sharded-shard2-data-0.ice-shard-mongodb-sharded-headless.ice-system.svc.cluster.local:27017,ice-shard-mongodb-sharded-shard2-data-1.ice-shard-mongodb-sharded-headless.ice-system.svc.cluster.local:27017,ice-shard-mongodb-sharded-shard2-data-2.ice-shard-mongodb-sharded-headless.ice-system.svc.cluster.local:27017",
"serverInfo" : {
"host" : "ice-shard-mongodb-sharded-shard2-data-0",
"port" : NumberInt(27017),
"version" : "4.4.3",
"gitVersion" : "913d6b62acfbb344dde1b116f4161360acd8fd13"
},
"plannerVersion" : NumberInt(1),
"namespace" : "LogCenter.LogInfo",
"indexFilterSet" : false,
"parsedQuery" : {
"$and" : [
{
"Properties.EapId" : {
"$eq" : "12eap012"
}
},
{
"Properties.Time" : {
"$gte" : ISODate("2021-03-17T00:00:00.000+0000")
}
}
]
},
"collation" : {
"locale" : "en_US",
"caseLevel" : false,
"caseFirst" : "off",
"strength" : NumberInt(2),
"numericOrdering" : false,
"alternate" : "non-ignorable",
"maxVariable" : "punct",
"normalization" : false,
"backwards" : false,
"version" : "57.1"
},
"winningPlan" : {
"stage" : "SHARDING_FILTER",
"inputStage" : {
"stage" : "FETCH",
"filter" : {
"Properties.EapId" : {
"$eq" : "12eap012"
}
},
"inputStage" : {
"stage" : "IXSCAN",
"keyPattern" : {
"Properties.Time" : 1.0
},
"indexName" : "Properties.Time_1",
"collation" : {
"locale" : "en_US",
"caseLevel" : false,
"caseFirst" : "off",
"strength" : NumberInt(2),
"numericOrdering" : false,
"alternate" : "non-ignorable",
"maxVariable" : "punct",
"normalization" : false,
"backwards" : false,
"version" : "57.1"
},
"isMultiKey" : false,
"multiKeyPaths" : {
"Properties.Time" : [
]
},
"isUnique" : false,
"isSparse" : false,
"isPartial" : false,
"indexVersion" : NumberInt(2),
"direction" : "forward",
"indexBounds" : {
"Properties.Time" : [
"[new Date(1615939200000), new Date(9223372036854775807)]"
]
}
}
}
},
"rejectedPlans" : [
]
}
]
}
},
"serverInfo" : {
"host" : "ice-shard-mongodb-sharded-mongos-59576fb8b8-n47n5",
"port" : NumberInt(27017),
"version" : "4.4.3",
"gitVersion" : "913d6b62acfbb344dde1b116f4161360acd8fd13"
},
"ok" : 1.0,
"operationTime" : Timestamp(1616372015, 12),
"$clusterTime" : {
"clusterTime" : Timestamp(1616372015, 14),
"signature" : {
"hash" : BinData(0, "vVxiZGic6un9QIInadoHCfBCX2Y="),
"keyId" : NumberLong(6924833040434724866)
}
}
}
I found the root cause why my search didn't use the sharding key. It's all about the collation.
Sharding key always use collation : { locale : "simple" } to do a binary comparison, while in my case my collection and indexes are designed to use "collation" : { "locale" : "en_US", "strength" : 2 }.
After redesigning my collection without any collation, now everything is fine.