Recently I was running a container under Compute Engine's container OS, and my data (my TLS certificate specifically) wasn't getting persisted outside of the container across reboots because I was writing to /etc. After a bit of time, I stumbled upon Disks and file system overview - File system, which explains how their are two types of writable partitions: stateful and stateless. /etc is stateless, and I needed to move my persisted files to /var for stateful storage.
But I'm left wondering about the purpose of writable, stateless partitions. Deploying Containers - Limitations explains how a container OS (on a VM instance) can only run one container. What does a writable but stateless partition enable compared to just writing data within the docker container, since both of those writable locations would be lost on host OS reboot anyway? Only benefit I could see would be sharing data across containers on the same host OS, but the limitation above invalidates that.
The main purpose of COS images is the security: a minimal OS, without useless system libraries and binaries and able to run containers.
So that, the /etc is stateless to not persist changes and updates (backdoors) in the most important executable library of the COS.
On the container side, it lives in memory. You can write what you want on it, it's written in memory (except if you have mount volume in your container, but it's not the purpose here). And you are limited by the amount of memory available in the container. And finally, when you stop the container, it is offloaded from the memory and of course, you lost all the data written in the container.
So now, you need to have in mind that the /etc of your container isn't the same as your /etc of your VM. Same for the /var. The /var of your container is always stateless (if not mounted from a VM volume), the /var of your VM is statefull.
In addition, the lifecycle isn't the same: You can start and stop several containers on your COS VM, without stopping and restarting it. So the VM /etc will live all the VM life, and maybe "view" several containers' life.
Eventually, the COS image is used on a Compute Engine to run a container, and only one at a time. However, this COS image is also used for Kubernetes node pools (GKE on GCP) and, typically with Kubernetes, you can run several Pod (1+ containers) on the same Node (Compute Engine instance).
All this use cases can show you the meaning and the usefulness (or not) of these restrictions and features (and I hope I was clear in my explanations!)