GCP instance group doesn't start containers

Question

I have an instance template that is supposed to run my app in a container running on Google Cloud's Container-Optimized OS. When I create a single VM from this template it runs just fine, but when I use it to create an instance group the containers don't start.

According to the logs the machine didn't even try to start them.

I tried to compare the output from gcloud compute instances describe <instance-name> for the instance that works OK against one of the instances in the MIG, but other than some differences in the network interfaces and some that are due to the fact that one instance is managed by an instance group and the other one isn't I don't see anything unusual.

I also noticed that when I SSH to the instance that works, I get this message:

  ########################[ Welcome ]########################
  #  You have logged in to the guest OS.                    #
  #  To access your containers use 'docker attach' command  #
  ###########################################################

but when I SSH to one of the instances in the MIG, I don't see it.

Is there a problem with using the container-optimized OS in an instance group?

My instance template is defined as follows:

creationTimestamp: '2022-11-09T03:25:29.896-08:00'
description: ''
id: '757769630202081478'
kind: compute#instanceTemplate
name: server-using-docker-hub-1
properties:
  canIpForward: false
  confidentialInstanceConfig:
    enableConfidentialCompute: false
  description: ''
  disks:
  - autoDelete: true
    boot: true
    deviceName: server-using-docker-hub
    index: 0
    initializeParams:
      diskSizeGb: '10'
      diskType: pd-balanced
      sourceImage: projects/cos-cloud/global/images/cos-stable-101-17162-40-20
    kind: compute#attachedDisk
    mode: READ_WRITE
    type: PERSISTENT
  keyRevocationActionType: NONE
  labels:
    container-vm: cos-stable-101-17162-40-20
  machineType: e2-micro
  metadata:
    fingerprint: 76mZ3i--POo=
    items:
    - key: gce-container-declaration
      value: |-
        spec:
          containers:
          - name: server-using-docker-hub-1
            image: docker.io/rinbar/kwik-e-mart
            env:
            - name: AWS_ACCESS_KEY_ID
              value: <redacted>
            - name: AWS_SECRET_ACCESS_KEY
              value: <redacted>
            - name: SECRET_FOR_SESSION
              value: <redacted>
            - name: SECRET_FOR_USER
              value: <redacted>
            - name: MONGODBURL
              value: mongodb+srv://<redacted>@cluster0.<redacted>.mongodb.net/kwik-e-mart
            - name: DEBUG
              value: server:*
            - name: PORT
              value: '80'
            stdin: false
            tty: false
          restartPolicy: Always
        # This container declaration format is not public API and may change without notice. Please
        # use gcloud command-line tool or Google Cloud Console to run Containers on Google Compute Engine.
    kind: compute#metadata
  networkInterfaces:
  - kind: compute#networkInterface
    name: nic0
    network: https://www.googleapis.com/compute/v1/projects/rons-project-364411/global/networks/default
    stackType: IPV4_ONLY
    subnetwork: https://www.googleapis.com/compute/v1/projects/rons-project-364411/regions/me-west1/subnetworks/default
  reservationAffinity:
    consumeReservationType: ANY_RESERVATION
  scheduling:
    automaticRestart: true
    onHostMaintenance: MIGRATE
    preemptible: false
    provisioningModel: STANDARD
  serviceAccounts:
  - email: 629139871582-compute@developer.gserviceaccount.com
    scopes:
    - https://www.googleapis.com/auth/devstorage.read_only
    - https://www.googleapis.com/auth/logging.write
    - https://www.googleapis.com/auth/monitoring.write
    - https://www.googleapis.com/auth/servicecontrol
    - https://www.googleapis.com/auth/service.management.readonly
    - https://www.googleapis.com/auth/trace.append
  shieldedInstanceConfig:
    enableIntegrityMonitoring: true
    enableSecureBoot: false
    enableVtpm: true
  tags:
    items:
    - http-server
selfLink: https://www.googleapis.com/compute/v1/projects/rons-project-364411/global/instanceTemplates/server-using-docker-hub-1

Answer 1

Since the instances in the group have no external IP addresses, you need to enable Private Google Access or Cloud NAT to allow the instances to pull the container image from Container Registry / Artifact Registry / Docker Hub / any other container registry.