PGHOST for GitLab pipeline with docker:dind for postgres created by docker-compose

Question

I have a docker-compose file, that initializes postgres and service for postgres migration. And I want to run tests in gitlab pipeline against my docker-compose baked postgres service, but I can't connect to pg_db via localhost. Inside my code I use pgx package. On my local machine there is no trouble to use localhost for PGHOST env variable.

So my main question is what host to put in PGHOST variable for my tests to use for postgres connection inside gitlab pipeline.

docker-compose.yml

version: "3.3"
services:
    pg_db:
      container_name: pg_db
      image: postgres:13.2-alpine
      environment:
        - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
        - POSTGRES_USER=${POSTGRES_USER}
        - POSTGRES_DB=${POSTGRES_DB}
        - POSTGRES_SSLMODE=${POSTGRES_SSLMODE}
        - POSTGRES_HOST_AUTH_METHOD=${POSTGRES_HOST_AUTH_METHOD}
      ports:
        - ${POSTGRES_PORT}:5432
      restart: always
      deploy:
        resources:
          limits:
            cpus: '1'
            memory: 4G
      networks:
        - postgres
        - backend

    #init db
    store-init:
      image: x:latest
      container_name: store-init
      environment:
        - PGHOST=pg_db
        - PGUSER=${POSTGRES_USER}
        - PGPASSWORD=${POSTGRES_PASSWORD}
        - PGDATABASE=${POSTGRES_DB}
        - PGPORT=${POSTGRES_PORT}
      restart: on-failure
      depends_on:
        - pg_db
      networks:
        - postgres
        - backend

networks:
  backend:
  postgres:
    driver: bridge

And here is a significant part of my gitlab-ci.yml

services:
    - docker:dind

stages:
  - test

test:
  stage: test
  image: golang:1.17-alpine3.15
  variables:
    PGHOST: localhost
  before_script:
    - apk update && apk add make git openssh g++
    - apk add --no-cache docker-compose
    - git config --global user.email "$GITLAB_USER_EMAIL" && git config --global user.name "$GITLAB_USER_NAME"
    - mkdir -p ~/.ssh && echo "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa && chmod -R 600 ~/.ssh && ssh-keyscan -t rsa ssh.x >> ~/.ssh/known_hosts
  script:
    - cp .env.example .env
    - docker-compose up -d
    - sleep 30 # a temporary line to get the logs
    - cat /etc/hosts # debug line
    - docker-compose port pg_db 5432 # debug line
    - netstat -a # debug line
    - docker-compose ps # debug line
    - go test -v -timeout 30s ./... -tags=withDB

  only:
    - merge_request
    - dev
    - master

The logs I get for

variables: 
   PGHOST: localhost

$ cp .env.example .env
$ docker-compose up -d
Recreating alp-logger_pg_db_1 ... 
Recreating alp-logger_pg_db_1 ... done
store-init is up-to-date
$ sleep 30
$ cat /etc/hosts
127.0.0.1   localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.2  runner-lxzkchpx-project-304-concurrent-0
$ docker-compose port pg_db 5432
0.0.0.0:5432
$ netstat -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       
tcp        0      0 runner-lxzkchpx-project-304-concurrent-0:50294 static.124.194.21.65.clients.your-server.de:ssh TIME_WAIT   
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node Path
$ docker-compose ps
   Name                 Command              State           Ports         
---------------------------------------------------------------------------
pg_db        docker-entrypoint.sh postgres   Up      0.0.0.0:5432->5432/tcp
store-init   ./alp-store                     Up                            

and the error of connecting to postgres db:

failed to connect to `host=localhost user=test database=test`: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)

The logs for debug commands are same, so I'll skip them. The errors I get for

variables: 
    PGHOST: pg_db

and for any other named host like docker.

failed to connect to `host=pg_db user=test database=test`: hostname resolving error (lookup pg_db on 1.1.1.1:53: no such host)

The errors I get for

variables: 
    PGHOST: 127.0.0.1

failed to connect to `host=127.0.0.1 user=test database=test`: dial error (dial tcp 127.0.0.1:5432: connect: connection refused)

Answer 1

One of the important distinctions between running containers on your local machine and running them in GitLab using docker:dind is that the containers are not available on 'localhost' -- they are available on the docker:dind container.

If you want to talk to this container, in your scenario, the postgres container would be available on docker:5432 (docker being the hostname of the docker:dind container where your postgres container has its port mapping).

Illustration with simple HTTP service container

As a simplified example if you were to run the container strm/helloworld-http locally with a port mapping, the following works:

docker run -d --rm -p 80:80 strm/helloworld-http
# give it some time to startup
curl http://localhost  # this works

However, the same setup in GitLab does not:

myjob:
  variables:  # these variables are not necessarily required
    DOCKER_TLS_CERTDIR: ""
    DOCKER_HOST: "tcp://docker:2375"
  services:
    - docker:dind
  script:
    - docker run -d --rm -p 80:80 strm/helloworld-http
    - sleep 10
    - curl http://localhost  # Fails!

One fix would be to use the docker hostname instead:

script:
    - docker run -d --rm -p 80:80 strm/helloworld-http
    - sleep 10
    - curl http://docker  # works!