Search code examples
dockernix

nix with custom store (inside chroot) not working inside docker container - Could not resolve host: docker

made an issue https://github.com/NixOS/nix/issues/2663

How to reproduce

  1. run this in some terminal

docker run --privileged --rm --name some-docker docker:stable-dind

  1. save test file
cat > /tmp/test.nix << 'EOL'
{ pkgs ? import <nixpkgs> {} }:
with pkgs;
stdenv.mkDerivation {
  pname = "test";
  version = "0.0.1";
  DOCKER_HOST = builtins.getEnv "DOCKER_HOST";
  buildInputs = [docker curl nettools];
  phases = "installPhase";
  installPhase = ''
    (ls -al /etc || true)
    (cat /etc/nsswitch.conf || true)
    (cat /etc/hosts || true)
    (cat /etc/resolv.conf || true)

    # without --store returns
    #
    # Kernel IP routing table
    # Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
    # 0.0.0.0         172.17.0.1      0.0.0.0         UG        0 0          0 eth0
    # 172.17.0.0      0.0.0.0         255.255.0.0     U         0 0          0 eth0
    #
    # with --store returns empty
    #
    # Kernel IP routing table
    # Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
    netstat --numeric --route

    # without --store - returns without error
    # with --store - error "Could not resolve host: docker"
    curl -v http://docker:2375/v1.39/version

    # without --store - returns without error, prints server info
    # with --store - error "error during connect: Get http://docker:2375/v1.39/version: dial tcp: lookup docker on [::1]:53: read udp [::1]:39506->[::1]:53: read: connection refused"
    docker version

    # create dummy package if everything above did work fine
    mkdir -p $out
  '';
}
EOL
  1. it works without --store argument
docker run -it --rm --link some-docker:docker -v /tmp/test.nix:/tmp/test.nix nixos/nix@sha256:85299d86263a3059cf19f419f9d286cc9f06d3c13146a8ebbb21b3437f598357 sh -c 'export DOCKER_HOST=tcp://docker:2375/ && (echo "hosts: files dns" > /etc/nsswitch.conf) && nix-build /tmp/test.nix'

output - https://pastebin.com/DZmXrATR

  1. it doesnt work with --store argument
docker run -it --rm --link some-docker:docker --privileged -v /tmp/test.nix:/tmp/test.nix nixos/nix@sha256:85299d86263a3059cf19f419f9d286cc9f06d3c13146a8ebbb21b3437f598357 sh -c 'export DOCKER_HOST=tcp://docker:2375/ && (echo "hosts: files dns" > /etc/nsswitch.conf) && nix-build --store /tmp/store /tmp/test.nix'

output https://pastebin.com/Z4DxtLQr

how to make it work?

Update:

seems like it's because /etc/nsswitch.conf is not mounted when using --store

Unfortunately, nix is not allowing me to create it myself (touch /etc/nsswitch.conf throws permission denied)

Update:

I found that I can use extra-sandbox-paths to mount files from container to nix-build sandbox

mounting /etc/nsswitch.conf solved curl: (6) Could not resolve host: docker

but I cant fix * Immediate connect fail for 172.17.0.2: Network is unreachable error, I've tried mounting all network related files from /etc, but it's not working

docker run --privileged --rm --name some-docker docker:stable-dind

docker run -it --rm --link some-docker:docker --privileged -v /tmp/test.nix:/tmp/test.nix nixos/nix@sha256:85299d86263a3059cf19f419f9d286cc9f06d3c13146a8ebbb21b3437f598357 sh

nix-env -i curl nettools

# works
curl -v http://172.17.0.2:2375/v1.39/version

# works
curl -v http://docker:2375/v1.39/version

# lo and eth
ifconfig -a

# not empty
netstat -rn

export DOCKER_HOST=tcp://docker:2375/ && (echo "hosts: files dns" > /etc/nsswitch.conf)

cat > /etc/nix/nix.conf << 'EOL'
sandbox = false
extra-sandbox-paths = /etc/nsswitch.conf=/etc/nsswitch.conf /etc/resolv.conf=/etc/resolv.conf /etc/hosts=/etc/hosts /etc/protocols=/etc/protocols /etc/udhcpd.conf=/etc/udhcpd.conf /etc/modules=/etc/modules
EOL

cat > /tmp/test.nix << 'EOL'
{ pkgs ? import <nixpkgs> {} }:
with pkgs;
stdenv.mkDerivation {
  pname = "test";
  version = "0.0.1";
  DOCKER_HOST = builtins.getEnv "DOCKER_HOST";
  buildInputs = [docker curl nettools];
  phases = "installPhase";
  installPhase = ''
    # only lo
    ifconfig -a

    # empty
    netstat --numeric --route

    # fails
    curl -v http://172.17.0.2:2375/v1.39/version
    curl -v http://docker:2375/v1.39/version

    docker version
    mkdir -p $out
  '';
}
EOL

nix-build --store /tmp/store /tmp/test.nix

UPDATE

Current state of research

https://gitlab.com/gitlab-org/gitlab-ce/issues/31312#note_138576414

Solution

  • If your installPhase runs curl, you're doing it wrong. Derivations in Nix are supposed to be pure: To have their output depend only on their stated inputs, and nothing else. A derivation that connects to the network is impure by nature: Its results will depend on what is present behind the given network resource at the time when it's invoked. Thus, Nix's sandboxing intentionally (and in accordance with its documentation) disallows network access by its builders.

    Consider the following, which is still impure, but uses builtins.fetchurl instead, and so is not blocked from operation:

    { pkgs ? import <nixpkgs> {} }:
with pkgs; let
  # WARNING: This is impure; usually, downloads should include an explicit hash
  versionFile = builtins.fetchurl http://172.17.0.2:2375/v1.39/version
in stdenv.mkDerivation {
  pname = "test";
  version = "0.0.1";
  DOCKER_HOST = builtins.getEnv "DOCKER_HOST";
  buildInputs = [docker curl nettools];
  phases = "installPhase";
  installPhase = ''
    cat ${escapeShellArg versionFile}
    docker version
    mkdir -p "$out"
  '';
}

    It's strongly recommended that you use pkgs.dockerTools to build Docker-compatible images using only pure Nix code, rather than trying to run Docker inside a Nix derivation.