Search code examples
bigtablegoogle-cloud-bigtable

Maintain data in Google Bigtable for longer periods


We have use-cases where we would like to store a large volume of data in Google Bigtable for long periods:

  • during product development
  • for performance tuning
  • for demos

We need to store the data but we don't really need it to be "online" all the time. The current cost bottleneck seems to be the cost of nodes which in these cases are idle for long periods.

How is Google Bigtable being used during product development? I am aware of the development mode (and the emulator) and they are fine for some use-cases but we still need the production environment for other use-cases.

Really, what would be ideal is the ability to switch "off" Bigtable (while still paying for data stored but not for nodes) and bring up the nodes when needed. I don't believe this feature exists. In its absence are there other possible workarounds/alternatives?


Solution

  • It's an interesting question. I've done it with smaller projects using Datastore with much smaller sizes (~2Gb) that have hung around for years after disabling billing. Given how much it costs to do backup/restores on those projects, I could imagine this would be cost prohibitive solution in the BigTable world. It is disappointing that Google hasn't provided a better solution for this. They do talk about different storage classes so I'd imagine disabling a project would move its assets to coldline - but that is just rank speculation on my part.