Search code examples
node.jsdockerdocker-composedocker-volume

Docker build does not install packages in `node_modules` volumes

What causes yarn/npm install from not installing any packages in a volume?

I have a Dockerfile that has the instruction to RUN yarn install (tested with both NPM and YARN), but the packages node_modules directory inside the container is empty. If I exec -it service_name bash and run the install command manually, it installs the packages correctly. I've noticed this after a refactor, where I had a Worker service that did the process of installation and a second that ran the development server. Decided to keep all in the same Docker-compose declaration instead but since the issue start happening, it persists. Tried a full reset etc without success (down, rm containers, prune, etc).

The Service in question declared in the Docker-compose file:

  node_dev:
    build:
      context: .
      dockerfile: ./.docker/dockerFiles/node.yml
    image: foobar/node_dev:latest
    container_name: node_dev
    working_dir: /home/node/app
    ports:
     - 8000:8000
     - 9000:9000
    environment:
      - NODE_ENV=development
      - GATSBY_WEBPACK_PUBLICPATH=/
    volumes:
      - ./foobar-blog-ui/:/home/node/app
      - ui_node_modules:/home/node/app/node_modules
      - ui_gatsbycli_node_module:/usr/local/lib/node_modules/gatsby-cli
      - ./.docker/scripts/wait-for-it.sh:/home/node/wait-for-it.sh
    command: /bin/bash -c '/home/node/wait-for-it.sh wordpress-reverse-proxy:80 -t 10 -- yarn start'
    depends_on:
      - mysql
      - wordpress
    networks:
      - foobar-wordpress-network

The related Volumes references in the service:

volumes:
    ui_node_modules:
    ui_gatsbycli_node_module:

Finally, the Dockerfile that generates the image:

FROM node:8.16.0-slim

ARG NODE_ENV=development
ENV NODE_ENV=${NODE_ENV}

WORKDIR /home/node/app

RUN apt-get update
RUN apt-get install -y rsync vim git libpng-dev libjpeg-dev libxi6 build-essential libgl1-mesa-glx

RUN yarn global add gatsby-cli
RUN yarn install

Also, tried to yarn install --force --no-lockfile and made sure that it was tested without any package or yarn lock files present in the project root, and vice-versa.

I'm finding this quite odd and definitely a typo somewhere but I haven't been able to spot yet.

The host system is macOS Mojave.

I'd like to mention that if exec -it service_name bash and execute the NPM/YARN install the node_modules is populated with the packages. Before most tests I did, I've also tried to reset by:

docker-compose stop
docker-compose rm -f
docker volume prune -f
docker network prune -f

And now tested with:

docker stop $(docker ps -a -q)
docker rm $(docker ps -a -q)
docker volume prune -f
docker volume rm $(docker volume ls -qf dangling=true)
docker network prune -f
docker system prune --all --force --volumes
rm -rf ./.docker/certs/ ./.docker/certs-data/ ./.docker/logs/nginx/ ./.docker/mysql/data

The logs for the particular image:

Building node_dev
Step 1/8 : FROM node:8.16.0-slim
8.16.0-slim: Pulling from library/node
9fc222b64b0a: Pull complete
7d73b1e8f94b: Pull complete
1059045652d5: Pull complete
08cd60b80e4e: Pull complete
b7d875c65da4: Pull complete
Digest: sha256:0ec7ac448d11fa1d162fb6fd503ec83747c80dcf74bdf937b507b189b610756a
Status: Downloaded newer image for node:8.16.0-slim
 ---> 67857c9b26e1
Step 2/8 : ARG NODE_ENV=development
 ---> Running in da99a137d733
Removing intermediate container da99a137d733
 ---> 0f9b718d3f66
Step 3/8 : ARG NPM_TOKEN=3ea44a41-9293-4569-a235-a622ae216d60
 ---> Running in e339a4939029
Removing intermediate container e339a4939029
 ---> e47b42008bc3
Step 4/8 : ENV NODE_ENV=${NODE_ENV}
 ---> Running in fdc09147e9da
Removing intermediate container fdc09147e9da
 ---> 3b28ab5539d3
Step 5/8 : WORKDIR /home/node/app
 ---> Running in 44eef1d9293d
Removing intermediate container 44eef1d9293d
 ---> 2d07ecf3de2e
Step 6/8 : RUN echo "//registry.npmjs.org/:_authToken=$NPM_TOKEN" > .npmrc
 ---> Running in a47d5e22839b
Removing intermediate container a47d5e22839b
 ---> bd9f896846b7
Step 7/8 : RUN yarn global add gatsby-cli
 ---> Running in ca3e74d12df4
yarn global v1.15.2
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Installed "[email protected]" with binaries:
      - gatsby
Done in 15.51s.
Removing intermediate container ca3e74d12df4
 ---> bc8d15985ad0
Step 8/8 : RUN yarn install --force --no-lockfile
 ---> Running in 3f0e35e5487b
yarn install v1.15.2
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Rebuilding all packages...
Done in 0.04s.
Removing intermediate container 3f0e35e5487b
 ---> 485b9e9bccba
Successfully built 485b9e9bccba
Successfully tagged foobar/node_dev:latest

Changed the Service command to sleep 300s and exect -it and ls -la the /home/node/app/node_modules to find:

.yarn-integrity

And when cat .yarn-integrity I see:

{
  "systemParams": "linux-x64-57",
  "modulesFolders": [],
  "flags": [],
  "linkedModules": [],
  "topLevelPatterns": [],
  "lockfileEntries": {},
  "files": [],
  "artifacts": {}
}
Solution

  • I found a solution by taking advantage of Docker's cache system through the COPY instruction, that I've set to COPY package.json, package-lock.json and yarn.lock; Which caches the build step and only updates if there is any difference in the files, skipping reinstalling the packages otherwise.

    To summarize, the service in the Docker compose remains the same, with the volumes, following the best practices shared in the community for nodejs projects.

    We can see in the volumes a bind-mount ./foobar-blog-ui/:/home/node/app that mounts the App source code on the host to the app directory in the container. This allows a fast development environment because the changes we make in the host are immediately populated in the container, not possible to do otherwise.

    Finally the named volume for the node_modules that I've named ui_node_modules, which is quite tricky to understand as we started by bind mount the app source code that includes the root where node_modules sits. When npm install runs the node_modules directory is created in the container, correct? But the bind-mount we've declared hides it. So, the named node_modules volume called ui_node_modules solves it by persisting the content of the /home/node/app/node_modules directory into the container and bypassing the hidden bind-mount.

      node_dev:
    build:
      context: .
      dockerfile: ./.docker/dockerFiles/node.yml
    image: foobar/node_dev:latest
    container_name: node_dev
    working_dir: /home/node/app
    ports:
     - 8000:8000
     - 9000:9000
    environment:
      - NODE_ENV=development
      - GATSBY_WEBPACK_PUBLICPATH=/
    volumes:
      - ./foobar-blog-ui/:/home/node/app
      - ui_node_modules:/home/node/app/node_modules
      - ui_gatsbycli_node_module:/usr/local/lib/node_modules/gatsby-cli
      - ./.docker/scripts/wait-for-it.sh:/home/node/wait-for-it.sh
    command: /bin/bash -c '/home/node/wait-for-it.sh wordpress-reverse-proxy:80 -t 10 -- yarn start'
    depends_on:
      - mysql
      - wordpress
    networks:
      - foobar-wordpress-network

volumes:
    ui_node_modules:
    ui_gatsbycli_node_module:

    But the Dockerfile includes the instruction to copy the package*.json files that helps determinate when the Docker cache for the instruction step or layer should be updated.

    FROM node:8.16.0-slim

ARG NODE_ENV=development

ENV NODE_ENV=${NODE_ENV}

RUN mkdir -p /home/node/app/node_modules && chown -R node:node /home/node/app

WORKDIR /home/node/app

# The package files should use Docker cache system
COPY ./foobar-blog-ui/package*.json .
COPY ./foobar-blog-ui/yarn.lock .

RUN yarn global add gatsby-cli
RUN yarn install

RUN apt-get update
RUN apt-get install -y rsync vim git libpng-dev libjpeg-dev libxi6 build-essential libgl1-mesa-glx

EXPOSE 8000

    Hope this helps someone else in the future!