Search code examples
google-cloud-sqlgoogle-cloud-datastore

Migration from Google cloud sql to datastore

Is there an easy way to migrate a large (100G) Google cloud sql database to Google datastore?

The way that comes to mind is to write a python appengine script for each database and table and then put it into the datastore. That sounds tedious but maybe it has to be done?

Side note, the reason I'm leaving cloud sql is because I have jsp pages with multiple queries on them and they are incredibly slow even with a d32 sql instance. I hope that putting it in the datastore will be faster?

There seems to be a ton of questions about moving away from the datastore to cloud sql, but I couldn't find this one.

Thanks

Solution

  • Here are a few options:

    1. Write an App Engine mapreduce [1] program that pulls data in appropriate chunks from Cloud SQL and write is to Datastore .
    2. Spin up a VM on Google Compute Engine and write a program that fetches the data from Cloud SQL and write to Datastore using the Datastore external API [2].
    3. Use the Datastore restore [3]. I'm not familiar with the format so I don't know how much work is to get produce something that the restore will accept.

    [1] https://cloud.google.com/appengine/docs/python/dataprocessing/
    [2] https://cloud.google.com/datastore/docs/apis/overview
    [3] https://cloud.google.com/appengine/docs/adminconsole/datastoreadmin?csw=1#restoring_data