I have a container I'm deploying to Kubernetes (GKE), and the image I have built locally is good, and runs as expected, but it appears that the image being pulled from Google Container Registry, when the run command is changed to pwd && ls returns the output shown here:
I 2020-06-17T16:24:54.222382706Z /app
I 2020-06-17T16:24:54.226108583Z lost+found
I 2020-06-17T16:24:54.226143620Z package-lock.json
and the output of the same commands when running in the container locally, with docker run -it <container:tag> bash is this:
#${API_CONTAINER} resolves to gcr.io/<project>/container: I.E. tag gets appended
.../# docker run -it ${API_CONTAINER}latest bash
root@362737147de4:/app# pwd
/app
root@362737147de4:/app# ls
Dockerfile dist files node_modules package.json ssh.bat stop_forever.bat test tsconfig.json
cloudbuild.yaml environments log package-lock.json src startApi.sh swagger.json test.pdf tsconfig.test.json
root@362737147de4:/app#
My thoughts on this start with, either the push to the registry is literally failing to work, or I'm not pulling the right one, i.e. pulling some off latest tag that was build by cloud build in a previous attempt to get this going.
What could be the potential issue? what could potentially fix this issue?
Edit: After using differing tags in deployment, using --no-cache during build, and pulling from the registry on another machine, my inclination is that GKE is having an issue pulling the image from GCR. Is there a way I can put this somewhere else, or get visibility on what's going on with the pull?
EDIT 2:
So Yes, I have a docker file I can share, but please be aware that I have inherited it, and don't understand the process that went into building this, or why some steps were necessary to the other developer. (I am definitely interested in refactoring this as much as possible.
FROM node:8.12.0
RUN mkdir /app
WORKDIR /app
ENV PATH /app/node_modules/.bin:$PATH
RUN apt-get update && apt-get install snmp -y
RUN npm install --unsafe-perm=true
RUN apt-get update \
&& apt-get install -y \
gconf-service \
libasound2 \
libatk1.0-0 \
libatk-bridge2.0-0 \
libc6 \
libcairo2 \
libcups2 \
libdbus-1-3 \
libexpat1 \
libfontconfig1 \
libgcc1 \
libgconf-2-4 \
libgdk-pixbuf2.0-0 \
libglib2.0-0 \
libgtk-3-0 \
libnspr4 \
libpango-1.0-0 \
libpangocairo-1.0-0 \
libstdc++6 \
libx11-6 \
libx11-xcb1 \
libxcb1 \
libxcomposite1 \
libxcursor1 \
libxdamage1 \
libxext6 \
libxfixes3 \
libxi6 \
libxrandr2 \
libxrender1 \
libxss1 \
libxtst6 \
ca-certificates \
fonts-liberation \
libappindicator1 \
libnss3 \
lsb-release \
xdg-utils \
wget
COPY . /app
# Installing puppeteer and chromium for generating PDF of the invoices.
# Install latest chrome dev package and fonts to support major charsets (Chinese, Japanese, Arabic, Hebrew, Thai and a few others)
# Note: this installs the necessary libs to make the bundled version of Chromium that Puppeteer
# installs, work.
RUN apt-get update \
&& apt-get install -y wget gnupg libpam-cracklib \
&& wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
&& sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
&& apt-get update \
&& apt-get install -y google-chrome-unstable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-freefont-ttf \
--no-install-recommends \
&& rm -rf /var/lib/apt/lists/*
# Uncomment to skip the chromium download when installing puppeteer. If you do,
# you'll need to launch puppeteer with:
# browser.launch({executablePath: 'google-chrome-unstable'})
# ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD true
# Install puppeteer so it's available in the container.
RUN npm i puppeteer \
# Add user so we don't need --no-sandbox.
# same layer as npm install to keep re-chowned files from using up several hundred MBs more space
&& groupadd -r pptruser && useradd -r -g pptruser -G audio,video pptruser \
&& mkdir -p /home/pptruser/Downloads \
&& chown -R pptruser:pptruser /home/pptruser \
&& chown -R pptruser:pptruser /app/node_modules
#build the api, and move into place.... framework options are limited with the build.
RUN npm i puppeteer kiwi-server-cli && kc build -e prod
RUN rm -Rf ./environments & rm -Rf ./src && cp -R ./dist/prod/* .
# Run everything after as non-privileged user.
# USER pptruser
CMD ["google-chrome-unstable"] # I have tried adding this here as well "&&", "node", "src/server.js"
For pushing the image I'm using this command:
docker push gcr.io/<projectid>/api:latest-<version> and I have the credentials setup with cloud auth configure-docker and here's a sanitized version of the yaml manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
kompose.cmd: kompose convert -f ./docker-compose.yml
kompose.version: 1.21.0 ()
creationTimestamp: null
labels:
io.kompose.service: api
name: api
spec:
replicas: 1
selector:
matchLabels:
io.kompose.service: api
strategy:
type: Recreate
template:
metadata:
annotations:
kompose.cmd: kompose convert -f ./docker-compose.yml
kompose.version: 1.21.0 ()
creationTimestamp: null
labels:
io.kompose.service: api
spec:
containers:
- args:
- bash
- -c
- node src/server.js
env:
- name: NODE_ENV
value: production
- name: TZ
value: America/New_York
image: gcr.io/<projectId>/api:latest-0.0.9
imagePullPolicy: Always
name: api
ports:
- containerPort: 8087
resources: {}
volumeMounts:
- mountPath: /app
name: api-claim0
- mountPath: /files
name: api-claim1
restartPolicy: Always
serviceAccountName: ""
volumes:
- name: api-claim0
persistentVolumeClaim:
claimName: api-claim0
- name: api-claim1
persistentVolumeClaim:
claimName: api-claim1
status: {}
The solution comes from the original intent of the docker-compose.yml file which was converted into a kubernetes manifest via a tool called kompose. The original docker-compose file was intended for development and as such had overrides in place to push the local development environment into the running container.
This was because of this in the yml file:
services:
api:
build: ./api
volumes:
- ./api:/app
- ./api/files:/files
which translates to this on the kubernetes manifest:
volumeMounts:
- mountPath: /app
name: api-claim0
- mountPath: /files
name: api-claim1
volumes:
- name: api-claim0
persistentVolumeClaim:
claimName: api-claim0
- name: api-claim1
persistentVolumeClaim:
claimName: api-claim1
Which Kubernetes has no files to supply, and the app is essentially overwritten with an empty volume, so the file is not found.
removal of the directives in the kubernetes manifest resulted in success.
Reminder to us all to be mindful.