Search code examples
mongodbdockergoogle-compute-enginegce-persistent-disk

GCE "create-with-container --container-mount-disk" flag mounts disk as read-only


I am trying to use a Percona Docker image for MongoDB on GCE, however I'm running into an issue with Mongo saying the mounted path is read-only. I looked around as much as I could, but im stumped at what could be the issue.

gcloud compute instances create-with-container mongo-svr \
--create-disk name=disk-1,size=1GB \
--container-mount-disk mount-path="/data/mongodb",mode=rw \
--container-image=docker.io/percona/percona-server-mongodb:4.2

I used the above command and it creates my instance. I then SSH into the server, connect to the running mongo instance to shutdown, then I run: docker exec -it [NAME] mongod --configsvr --replSet rs0 --dbpath=/data/mongodb --bind_ip localhost

This spits out an error stating:

CONTROL  [initandlisten] options: { net: { bindIp: "localhost" }, replication: { replSet: "rs0" }, sharding: { clusterRole: "configsvr" }, storage: { dbPath: "/data/mongodb" } }
STORAGE  [initandlisten] exception in initAndListen: IllegalOperation: Attempted to create a lock file on a read-only directory: /data/mongodb, terminating

At this point, I've been recreating instances with different params, but nothing has worked so far. Anyone have an idea what I'm missing?

Updated with command output

gcloud compute instances create-with-container mongo-config-f --zone us-central1-f --create-disk name=disk-1,size=1GB --container-mount-disk mount-path="/data/mongodb" --container-image=docker.io/percona/percona-server-mongodb:4.2 --machine-type=f1-micro
WARNING: Default device-name for disk name [disk-1] will be [disk-1] because it is being mounted to a container with [`--container-mount-disk`]
Created [https://www.googleapis.com/compute/v1/projects/[PROJECT_NAME]/zones/us-central1-f/instances/mongo-config-f].
NAME            ZONE           MACHINE_TYPE  PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS
mongo-config-f  us-central1-f  f1-micro                   xx.xx.xx.xx  xx.xx.xx.xx     RUNNING

Solution

  • I've tried to replicate your issue on my test project and found that:

    • persistent disk was created and mounted in read-write mode as expected;

      bash-4.2$ mount 
      ...
      /dev/sdb on /data/mongodb type ext4 (rw,relatime)
      
    • docker runs containers inside our VM properly;

    • the cause of the error while running docker exec -it [NAME] mongod --configsvr --replSet rs0 --dbpath=/data/mongodb --bind_ip localhost is permissions inside mongodb container:

      bash-4.2$ ls -l /data/        
      ...
      drwxr-xr-x 3 root    root 4096 Feb 19 15:33 mongodb
      

    As a workaround commands could be executed with root permissions:

    $ docker exec -it --user root klt-mongo-svr-upd-wowt mongod --configsvr --replSet rs0 --db path=/data/mongodb
    

    Please find more details and my steps below:

    1. create VM:

      $ gcloud compute instances create-with-container mongo-svr \
      --create-disk name=disk-1,size=1GB \                                                                                 
      --container-image docker.io/percona/percona-server-mongodb:4.2 \
      --container-mount-disk mount-path="/data/mongodb"                               
      WARNING: Default device-name for disk name [disk-1] will be [disk-1] because it is being mounted to a container with [`--container-mount-disk`]
      Created [https://www.googleapis.com/compute/v1/projects/test-prj/zones/europe-west3-a/instances/mongo-svr].
      NAME           ZONE            MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS
      mongo-svr-upd  europe-west3-a  n1-standard-1               10.156.0.9   35.XXX.155.XXX  RUNNING
      
    2. SSH to instance;

    3. check if container is running:

      $ docker ps
      CONTAINER ID        IMAGE                                                                COMMAND                  CREATED              STATUS              PORTS               NAMES
      dfad9c10235d        percona/percona-server-mongodb:4.2                                   "/entrypoint.sh mong…"   About a minute ago   Up About a minute                       klt-mongo-svr-upd-wowt
      bbe02c8e8621        gcr.io/stackdriver-agents/stackdriver-logging-agent:0.2-1.5.33-1-1   "/entrypoint.sh /usr…"   About a minute ago   Up About a minute                       stackdriver-logging-agent
      

      everything looks good at this point;

    4. try to run command as user:

       $ docker exec -it klt-mongo-svr-upd-wowt mongod --configsvr --replSet rs0 --dbpath=/data/mongodb --bind_ip localhost
      

      and observe the same error:

      2020-02-19T15:37:53.176+0000 I  STORAGE  [initandlisten] exception in initAndListen: IllegalOperation: Attempted to create a lock file on a read-only directory: /data/mongodb, terminating
      

      here key read-only directory: /data/mongodb;

    5. check mounts and permissions inside of the container:

      $ docker exec -it klt-mongo-svr-upd-wowt /bin/bash
      bash-4.2$ mount 
      ...
      /dev/sdb on /data/mongodb type ext4 (rw,relatime)
      ...
      

      as we expected disk was created and mounted in read-write mode to the container

      bash-4.2$ ls -l /data/        
      total 8
      drwxr-xr-x 4 mongodb root 4096 Feb 19 15:36 db
      drwxr-xr-x 3 root    root 4096 Feb 19 15:33 mongodb
      bash-4.2$ 
      

      but to work with /data/mongodb you need root permissions;

    6. try to run command as root:

      $ docker exec -it --user root klt-mongo-svr-upd-wowt mongod --configsvr --replSet rs0 --dbpath=/data/mongodb
      2020-02-19T15:45:24.970+0000 I  CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
      2020-02-19T15:45:24.973+0000 I  CONTROL  [initandlisten] MongoDB starting : pid=119 port=27019 dbpath=/data/mongodb 64-bit host=mongo-svr-upd
      2020-02-19T15:45:24.974+0000 I  CONTROL  [initandlisten] db version v4.2.2-3
      2020-02-19T15:45:24.974+0000 I  CONTROL  [initandlisten] git version: 2cdb6e50913583f627acc5de35dc4e04dbfe196f
      2020-02-19T15:45:24.974+0000 I  CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2k-fips  26 Jan 2017
      2020-02-19T15:45:24.974+0000 I  CONTROL  [initandlisten] allocator: tcmalloc
      2020-02-19T15:45:24.974+0000 I  CONTROL  [initandlisten] modules: none
      2020-02-19T15:45:24.974+0000 I  CONTROL  [initandlisten] build environment:
      2020-02-19T15:45:24.975+0000 I  CONTROL  [initandlisten]     distarch: x86_64
      2020-02-19T15:45:24.975+0000 I  CONTROL  [initandlisten]     target_arch: x86_64
      2020-02-19T15:45:24.975+0000 I  CONTROL  [initandlisten] options: { replication: { replSet: "rs0" }, sharding: { clusterRole: "configsvr" }, storage: { dbPath: "/data/mongodb" } }
      2020-02-19T15:45:24.976+0000 I  STORAGE  [initandlisten] Detected data files in /data/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      ...
      

      and it's working with root permissions.