Cloud Spanner: Unexpected CPU utilization
We are running a small part of our production data in Google Cloud Spanner since 8months for testing and learning things about Cloud Spanner.
Rough characteristics:
- 1 table, 4 billion rows, around 600GB storage size, 25mio lines added per day, nothing is removed from the table.
- We do write lot's of data with low and simple reads.
- The amount of data written daily is rising constantly but not by much
- Node-Count: 1 (This is not recommended for production but what's interesting to me is the relative change in CPU utilization - using 3 cuts utilization to not more than a third)
- We do have some but not many pretty large splits that may exceed the recommended split-size of 'some GB' as they are around 20GB or so
What we first noticed was nightly CPU utilization that did not directly correspond to the load we were producing. I thought about this as some kind of reorganization-task like in Cassandra. So no big thing we thought ...
One week ago things started to change and the CPU utilization went up and up. The storage size started to shrink. That's interesting as we only add data. (see images)
What I was thinking about:
- Maybe some new release of cloud spanner that triggers some global reorganization
- Our big splits produce problems during reorganization
Does anyone have information what could be behind this ?
Thank you for the detailed post. Yes, Cloud Spanner has recently made some optimizations for reorganization/compaction. Cloud Spanner regularly releases optimizations in different areas that run as background tasks that use idle cpu resources. Cloud Spanner ensures that background tasks do not take up resources needed for serving user requests.