Search code examples
meteoramazon-s3data-migrationcollectionfs

Migrating from cfs:gridfs to cfs:s3

In a meteor project I maintain there has been cause to look at moving away from gridfs as the backend to CollectionFS and moving towards s3.

One thing I would be keen to do is migrate images / files currently stored using the gridfs collections.

Has anyone attempted this before? I can't find any guides or even suggestions.

My thinking right now is along the lines of;

  1. Create a new collection backed by s3
  2. Iterate over old collection pushing the data into s3
  3. Update code to point to new collection

Does this seem sound?

Solution

  • I just did this!

    You're basically right, here's how I did it. Migration is a pretty easy process. I've gone from GridFS to S3.

    1) By adding new FS.Store.S3("store_name",{}), CollectionFS automatically clones the meta data of existing files in the old store for your new store. But, all file sizes are zero in this new store.

    Images = new FS.Collection("images", {
    stores: [
        new FS.Store.S3("s3images", {}),
        new FS.Store.GridFS("images", {})
    ]
});

    2) While you have both stores in place, you need to manually migrate the content using the piping as referenced here https://github.com/CollectionFS/Meteor-CollectionFS/wiki/How-to:-Convert-a-file-already-stored.

    if(Meteor.isServer) {
Images.find().forEach(function (fileObj) {
var readStream = fileObj.createReadStream('images');
var writeStream = fileObj.createWriteStream('s3images');
readStream.pipe(writeStream);
});
}

    Hopefully after this you will now see your new store's file sizes matches the old one!

    3) Optionally, remove the old store. If you keep both, inserted files are added to both, with priority given to the first store in the array.

    Reference: https://github.com/CollectionFS/Meteor-CollectionFS/issues/747