Mounting Docker container files to another container

Question

I'm developing a coderunner/compiler web-service. It consists of 2 services:

1. HTTP/Web - users send code here and I run it, reporting the results (Dockerized).
2. CodeRunner container - it grabs files from /input folder and returns results.json to the /output folder. Is run with name, input folder, output folder arguments. Different images for different languages. (By Exercism)

Flow:

1. I create a "User_Solution-ID" folder with code files (put the code in Solution.cs file, configure the .csproj)
2. I run the CodeRunner container, binding the "User_Solution-ID" folder to /input and /output
3. I read the results.json contents, respond to the user and delete the folder.

While this works perfectly when HTTP/Web is run on the host, I can't make it work when running in Docker.

The script I use to run the CodeRunner container from Dockerized HTTP/Web is:

#!/bin/bash

# Check if required arguments are provided
if [ $# -ne 4 ]; then
    echo "Usage: $0 <RunnerImage> <Exercise> <InputDirectory> <OutputDirectory>"
    exit 1
fi

# Assigning parameters to variables
RunnerImage="$1"
Exercise="$2"
InputDirectory="$3"
OutputDirectory="$4"

# Constructing JSON payload
json_payload="{\"Image\": \"$RunnerImage\", \"Cmd\": [\"$Exercise\", \"/input\", \"/output\"], \"HostConfig\": {\"Binds\": [\"$InputDirectory:/input\", \"$OutputDirectory:/output\"], \"NetworkMode\": \"none\"}}"

# Sending request to create container
response=$(curl --unix-socket /var/run/docker.sock -H "Content-Type: application/json" -d "$json_payload" -X POST http://localhost/v1.42/containers/create)

# Extracting container ID from response
container_id=$(echo "$response" | jq -r '.Id')

# Starting the container
curl --unix-socket /var/run/docker.sock -X POST "http://localhost/v1.42/containers/$container_id/start"

# Waiting for the container to finish
wait_response=$(curl --unix-socket /var/run/docker.sock -X POST "http://localhost/v1.42/containers/$container_id/wait")

# Extracting StatusCode from wait response
status_code=$(echo "$wait_response" | jq -r '.StatusCode')

echo "Container exited with status code: $status_code"

As expected, Docker will treat $InputDirectory and $OutputDirectory as if they existed on the host.

How do I overcome this?

I tried:

1. For the sake of testing, creating a volume input:/input with Docker Compose for my HTTP/Web, placing my code in there (no User_Solution-ID folder now) and passing this volume to container like this: {\"Binds\": [\"input:/input\".... This didn't work, as far as I remember because it got some naming conflict (Docker Compose prefixed volume path with some Compose name)
2. Creating a volume input:/app/input with Docker Compose for my HTTP/Web, but trying to bind input/User_Solution-ID folder to it to the CodeRunner container doesn't work (bound path is app/input/User_Solution-ID which doesn't exist on host)

What are the possible solutions? I think of creating a volume for every code folder and passing it to the CodeRunner Container.

Answer 1

You cannot mount files from one container to another. As you note in the question, the Docker daemon resolves bind-mount paths, and they are directories on the host system, not in the context that's invoking Docker. (In complex environments with remote Docker daemons or a Docker-in-Docker container, the bind mount is in the context where the Docker daemon is running and not necessarily the local system.)

it grabs files from /input folder and returns results.json to the /output folder

Programs that principally read and write local files are much easier to run as local processes. A Docker container has an isolated filesystem, and you wind up having to do some potentially complex setup (including making user IDs match) to be able to work with local files.

If it's possible for you to just run this as a subprocess, this will be vastly easier. You'd need to install the runner tool in your Web application's Dockerfile, but then you'd invoke it the same way you invoked any other subprocess (like this shell script).

# without doing anything Docker-related at all
"$Exercise" /input /output

Otherwise you need some sort of shared storage between the application container and the runner container(s). A host directory or a Docker named volume will both work. I might pass this location in an environment variable, since it can very readily depend on the specific deployment environment. Since you're passing paths as command-line options to the runner program, I'd mount the entire shared storage, and pass a subpath on the command line.

A basic example using a named volume could look like:

docker volume create shared
docker run -v shared:/shared ... web_app

Your Web application would then write data to /shared/input/$SOLUTION_ID, and create an empty /shared/output/$SOLUTION_ID directory. When it launched the runner container, it would look like

#!/bin/sh
RunnerImage="$1"
SharedVolume="$2"
Exercise="$3"
SolutionID="$4"

exec docker run \
  --rm \
  -v "$SharedVolume:/shared" \
  --net none \
  "$RunnerImage" \
  "$Exercise" "/shared/input/$SolutionId" "/shared/output/$SolutionId"

(You could translate this to manual curl commands driving the Docker HTTP API, but docker run does the same thing and it's much shorter.)

The same solution would work with a bind-mounted host directory. In docker run syntax, the same -v option will work, using a an absolute /host/path directory name. I believe the API call is slightly different.