Search code examples
mongodbpymongoexternal

Pymongo external harddrive to store collection

I googled quite a bit for this but could not find anything on this:

I am using mongodb/Pymongo on my local machine. Now I want to create a collection which is too big for my internal hard drive.

Is there a way to indicate MongoDB that I want to store a specific collection to a different path or would I have to move the entire database to the external hard-drive?

Any comments would be welcomed!

Solution

  • No, there is no straightforward way to save a single collection's data in a different location.

    There are a limited number of ways that you can keep different data in different places, but none of them are granular enough to work on different collections:

    1. The storage.directoryPerDB config option allows you to keep each database in a different directory - but that would still keep all collection data in the same folder.
    2. The storage.wiredTiger.engineConfig.directoryForIndexes option allows you to keep index data in a separate directory from the collection data - but that would still keep all collection data in the same folder.

    In both of these cases, and also for the journal, it is possible to use a symbolic link to a folder on a different drive - but this is not possible for a single collection's data.

    The standard solution for splitting a single very large collection's data across different locations is by sharding; but that requires multiple different servers, each with its own MongoDB installation. In your case, running just on a local machine, I don't think sharding would help you much; a better solution is probably just to buy a much bigger hard drive.