Search code examples
google-app-enginegoogle-cloud-datastoredatabase-backups

GAE data upload (seed and backup/restore)

Due to the frequent changes in GAE platform, I read different articles and bit confused what is the best way to export/upload data in year 2013. Bulk Upload tool is not really meant for High Replication Store as per the doc.

This document applies to apps that use the master/slave datastore. If your app uses the High Replication datastore, it is possible to copy data from the app, but Google does not currently support this use case.

Please comment on below specific scenarion. I plan to use golang runtime, but hope this doesn't affect backup/restore.

  1. How to load the datastore with initial(seed/test) data. Say, I can login as admin, then need to load data into datastore from csv/json files to a namespace.

  2. Datastore Admin: I think this experimental feature is meant for backup/restore. Does it work for GO runtime? What format the backup file is stored. Can I do it in json ?

Thanks.

Solution

  • I have a few apps running on AppEngine and from my experience, the easiest way to initially populate the DB with the existing data would be just include .json files along with your application files, and then use:

    with open( 'yourdata.json', 'r') as file :
    data = json.loads( file.read() )

    to get your data and then recreate necessary DB entities. You may delete .json files next time you upload your application, since these are not needed anymore.

    Regarding backups, there's Datastore Admin, which allows you to create backups and store them to BlobStore or some other cloud space, but I've found them really inconvenient to download and reupload again -- there are dozens of different files, very easy to forget or miss something.

    Therefore I'd suggest you write your own backup routines, that also save your data to the BlobStore, but in one single file, so you can easily download and reupload it back without the worrying about missing/forgotten files.

    The creation of backup file would be quite straightforward -- open BlobStore file for writing, then iterate over your models and convert them to .json and write (optionally compress on the fly with (g)zip) to the BlobStore file.