We have been using Cloud Spanner with three nodes and getting good performance
9,010 mutations
in 0.168 seconds
across 106 rows and 85 columns
or 53,630 mutations per second
Since we're still developing, we decided to only use a single node, to save development costs. Unfortunately, we're experiencing very poor performance. Much less than simply reducing the above by 66%. We're seeing
85 mutations mutation
in 1.7 seconds
across 1 row and 85 columns
or 50 mutations per second
We go from about 53,630 mutations a second to 50 mutations a second. Which is a decrease in performance by over 1/1000, instead of the predicted 1/3.
We did not change a single line of code, and only changed the number of nodes. Does anyone have suggestions or ideas as to why we're seeing such a slow down when going from 3 to 1 cloud spanner nodes?
EDIT: Just to be clear, we're using batch insert and when we "reduced" from 3 instances to 1, we blew away the nodes and started over with 1.
EDIT: corrected semantics ("node" instead of "instance")
GCP solved what went wrong in this issue. Copying the relevant portion in case the URL goes down:
Reducing the number of nodes from 3 to 1 is quite drastic for Cloud Spanner (is comparable to 300 to 100). The product is delivering high availability, data are replicated in different zones. So what happened in the background is that all the data from all replicas have to be restructured in the background to 1 node. That is relatively big operation that is supposed to take some time. In time background operation finishes, the performance should go back to expected level.