Search code examples
mongodbsharding

Mongodb choose shard key

I have a mongodb collection which I want to shard. This collection holds messages from users and a document from the collection has the following properties

{
     _id : ObjectId,
     conversationId: ObjectId,
     created: DateTime         
}

All queries will be done using the converstionId property and sorter by created.

  • Sharding by _id obviously won't work because I need to query by conversationId (plus _id is of type ObjectId which won't scale very well to many inserts)

  • Sharding by conversationId would be a logical choice in terms of query isolation but I'm afraid that it won't scale very well many inserts (even if I use a hashed shard key on conversationId or if I change the type of the property from ObjectId to some other type which isn't incremental like GUID) because some conversation might be much more active than others (i.e.: have many more message added to them)

From what I see in the mongo documentation The shard key is either an indexed field or an indexed compound field that exists in every document in the collection.

Does this mean that I can create a shard key on a compound index ?

Bottom line is that:

  • creating a hashed shard key from the _id property would offer good distribution of the data

  • creating a shard key on conversationId would offer good query isolation

So a combination of these two things would be great, if it could be done.

Any ideas?

Thanks

Solution

  • For your case, neither of fields look good choice for sharding. For instance, if you shard on conversationId, it will result in hot spotting, i.e. most of your inserts will happen to the last shard as conversationId would monotonically increase over time. Same problem with other two fields as well.

    Also, conversationId will not offer high degree of isolation as conversationId would monotonically increase over time. (Since newer conversations will get updated much more frequently than very old ones)

    In your case, a "hashed shard key"(version 2.4 onwards) over conversationId would be the smart choice as one would imagine that there can be tons of conversations going on in parallel.

    Refer following link for details on creating hashed shard key: [ http://docs.mongodb.org/manual/tutorial/shard-collection-with-a-hashed-shard-key/ ]