Search code examples
google-cloud-datalab

CPU swamped by node and zip in cloud datalab instance


I have recurring performance issue for my cloud datalab GCE instances, which over time seem to get swamped by root-level node and zip processes that are unrelated to my notebooks. (see image)

I have 4 CPUs and now 15GB of RAM, but making machines bigger does not solve the problem. I do have ~30GB of images stored on the datalab persistent disk, which may somehow (?) be contributing to the problem.

Creating brand new CDL instances does help, but this is an inconvenient approach.

Suggestions for resolutions, or the start of a diagnosis of the problem would be much appreciated.

top and console output


Solution

  • Those process seems to come from disk backups performed in background with file listeners. I've been dealing with this issue when adding a bunch of small datasets and could solve it by disabling backups on datalab creation.

    https://cloud.google.com/datalab/docs/reference/command-line/create

    --no-backups