I have a question concerning Docker. I'm running a Docker:dind Container, which I'm controlling throuh the API. I'm calling the Dind Container with the following Command
docker run --network some_network --prvileged -e DOCKER_TLS_CERTDIR= -d -p 2375:2375 -p 2376:2376 -v TransformationData:/mnt docker:dind
through the named volume, my Data is getting inside the Dind Container in the /mnt Directory.
Now it gets tricky. I want to run a Container through the API which transforms the Data in the /mnt directory and saves the transformed Data in the same directory, so I can use it through the named volume. the following are the Calls for the API
<REQUEST OPERATION="POST" URL="/containers/create" CONTENT-TYPE="application/json">
<CONTENT>
{
"Image": "citygml4j/citygml-tools",
"AttachStdin": true,
"HostConfig": {"Binds": [
"/mnt:/data[[r]shared, [rw]]"
]}} ,
"Cmd":[
"to-cityjson",
"/data"
]
}
</CONTENT>
</REQUEST>
<REQUEST OPERATION="POST" CONTENT-TYPE="application/json">
<xsl:attribute name="URL">/containers/<xsl:value-of select="substring(//CONTENT,11, 7)" />/start</xsl:attribute>
<CONTENT>{}</CONTENT>
</REQUEST>
The Container gets created, runs and exits, but the Data inside the /mnt Directory stays the same. instead of the "bind" Config I also tried a "mount" config with the same result.
One possible Cause for this Problem is, that the citygml-tools Container does not have write Permission on the /mnt directory of the host. I tried to fix that with the privileged Tag, but that did not work. Also the [[r]shared, [rw]] part in the "bind" should permit write, but it does not work either.
The annoying Part about this is, that the citygml-tools Container does not give any Error Messages or Lo files, so I don't actually know what's happening while I try. It's just a Trial and error like thing right now...
Long story short: What is the best Practice for Accessing Host Data from a Container, and write Data in the Host filesystem from a Container?
Additional Information on citygml-tools:
The DinD setup you're proposing here is especially complex and I would avoid it. If possible, I'd even avoid launching additional containers here.
In the actual API request, I suspect you have too much punctuation. The Docker CLI and API documentation use square brackets for grouping and indicating optional parts of names; [r]shared means "either shared or rshared". In the API ContainerCreate operation, the "HostConfig": {"Binds: []}" option includes strings with two or three colon-separated parts, /host/path:/container/path:options; the options are a comma-separated list of options, and the :options section is optional. So you may need instead
{
"HostConfig": {
"Binds": [
"/mnt:/data:rw"
]
}
}
with no square brackets at all.
Your DinD setup is mounting a named volume from the host's Docker into the DinD container's /mnt directory, and then your API request is trying to mount that. This won't usually create directly accessible host files and you may also need to change the DinD container to use -v /host:/container bind-mount syntax.
You can use citygml-tools as a (Java) library. If your application is JVM-based, doing this avoids all of the complexity you show here. Bind-mount the host directory with the data into your application container and remove all of the Docker API setup.
If you can't do that, you can also install a JVM and the citygml-tools release into your Docker image itself. Having done this, you can run commands like citygml-tools to-cityjson as an ordinary subprocess, without involving Docker. Again, bind-mount the host directory into the application container.
RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive \
apt-get install --no-install-recommends --assume-yes \
curl \
openjdk-11-jre-headless \
unzip
RUN cd /usr/local \
&& curl -LO https://github.com/citygml4j/citygml-tools/releases/download/v1.4.2/citygml-tools-1.4.2.zip \
&& unzip citygml-tools-1.4.2.zip \
&& rm citygml-tools-1.4.2.zip
If you need to keep the Docker API path, a simpler option will be to skip the DinD container and talk directly to the host's Docker socket. This is considered insecure, but precisely because it can escape the Docker container and mount arbitrary files on the host into a new container (even the host's /etc, read-write, with the container running as root). Whichever XML-based tool you're using to make requests would need to know how to make HTTP requests to a Unix socket. (Do NOT enable TCP access to the host Docker socket.)
docker run ... \
-v /var/run/docker.sock:/var/run/docker.sock \
your-image
# without a DinD container
You may be hitting some trouble with bind propagation since you're trying to remount a mount point in another context. I don't think this applies (you're mounting the mount point directly and not trying to mount an additional filesystem as a subdirectory of a mount point). If it does, though, this DinD approach won't work at all since you can't control the propagation settings for a named-volume mount.
What is the best Practice for Accessing Host Data [...], and write Data in the Host filesystem [...]?
Run the process directly on the host. This won't require escalated privileges to run, it obeys normal host filesystem permissions, and it will usually be easier to develop and test the application. If you're thinking ahead to clustered environments like Kubernetes, these actually can't access "host data" at all (if your cluster has 100 nodes, which is "the host"?) and you'll need some other storage mechanism.