I am trying to run my unit tests in GitLab CI. For my test setup, I am using docker-compose to start a database cluster of three neo4j instances, then I am running the tests on the docker host via localhost.
On my machine this works very well, so I tried to do the same on GitLab CI. I initially had some problems getting docker-compose to run inside the CI container, but I managed to get that working with the following .gitlab-ci.yaml:
image: "tiangolo/docker-with-compose:latest"
before_script:
- apk add --update nodejs npm
- npm install npm@latest -g
- npm install
services:
- docker:dind
run-test:
script:
- docker-compose up -d --build
- npm test
The dependencies get installed, docker-compose starts the database successfully and npm test runs my unit tests. So far, so good. But all tests that are using a database connection in some form fail, because they are not able to reach it on localhost:7687. I get the following error message
Neo4jError: Could not perform discovery. No routing servers available. Known routing table: RoutingTable[database=default database, expirationTime=0, currentTime=1588275075260, routers=[], readers=[], writers=[]]
Keep in mind, I am using the same docker-compose.yaml on my local machine and everything works, so it shouldn't be a problem with the database configuration.
I figured that my tests are running before the database is fully started, so I added a sleep 120 like so
run-test:
script:
- docker-compose up -d --build
- sleep 120
- npm test
I even measured how long it takes for the cluster to start inside the CI container and chose the timeout appropriately, but no luck, the tests still fail.
Then I had a look at the docker networks, to see if something went wrong there
$ cat /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.2 docker 3590cde9c5df runner-k3xtxnwc-project-17881831-concurrent-0-6d9d78b64ad6df95-docker-0
172.17.0.3 runner-k3xtxnwc-project-17881831-concurrent-0
$ docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' core1
172.19.0.2
$ docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' core2
172.19.0.4
But that looks good to me. I then tried to replace localhost:7678 with docker:7678, 172.0.0.1 and even $(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' core1):7678. But none of them work.
Trying to curl localhost:7474 (the neo4j http console) returns Failed to connect to localhost port 7474: Connection refused
EDIT:
My docker-compose.yaml
version: '3'
services:
dbc1:
build: ./database
container_name: core1
hostname: core1
ports:
- 7474:7474
- 7687:7687
environment:
- NEO4J_dbms_mode=CORE
- NEO4J_causal__clustering_minimum__core__cluster__size__at__formation=3
- NEO4J_causal__clustering_minimum__core__cluster__size__at__runtime=3
- NEO4J_causal__clustering_initial__discovery__members=core1:5000,core2:5000,core3:5000
- NEO4J_ACCEPT_LICENSE_AGREEMENT=yes
- NEO4J_dbms_connector_bolt_advertised__address=localhost:7687
- NEO4J_dbms_connector_http_advertised__address=localhost:7474
volumes:
- dbdata1:/data
ulimits:
nofile:
soft: 40000
hard: 40000
dbc2:
build: ./database
container_name: core2
hostname: core2
ports:
- 8474:7474
- 8687:7687
environment:
- NEO4J_dbms_mode=CORE
- NEO4J_causal__clustering_minimum__core__cluster__size__at__formation=3
- NEO4J_causal__clustering_minimum__core__cluster__size__at__runtime=3
- NEO4J_causal__clustering_initial__discovery__members=core1:5000,core2:5000,core3:5000
- NEO4J_ACCEPT_LICENSE_AGREEMENT=yes
- NEO4J_dbms_connector_bolt_advertised__address=localhost:8687
- NEO4J_dbms_connector_http_advertised__address=localhost:8474
volumes:
- dbdata2:/data
ulimits:
nofile:
soft: 40000
hard: 40000
dbc3:
build: ./database
container_name: core3
hostname: core3
ports:
- 9474:7474
- 9687:7687
environment:
- NEO4J_dbms_mode=CORE
- NEO4J_causal__clustering_minimum__core__cluster__size__at__formation=3
- NEO4J_causal__clustering_minimum__core__cluster__size__at__runtime=3
- NEO4J_causal__clustering_initial__discovery__members=core1:5000,core2:5000,core3:5000
- NEO4J_ACCEPT_LICENSE_AGREEMENT=yes
- NEO4J_dbms_connector_bolt_advertised__address=localhost:9687
- NEO4J_dbms_connector_http_advertised__address=localhost:9474
volumes:
- dbdata3:/data
ulimits:
nofile:
soft: 40000
hard: 40000
volumes:
dbdata1:
driver: local
dbdata2:
driver: local
dbdata3:
driver: local
My database Dockerfile (located in ./database):
FROM graphfoundation/ongdb:3.6
RUN echo "* soft nofile 40000" >> /etc/security/limits.conf
RUN echo "* hard nofile 40000" >> /etc/security/limits.conf
## inject password change into startup script
# create alternative startup script
RUN echo "#!/bin/bash -eu" >> /docker-entrypoint-modified.sh
# add command to change password
RUN echo "bin/neo4j-admin set-initial-password testpassword" >> /docker-entrypoint-modified.sh
# copy the contents of docker-entrypoint.sh, but skip the first line
RUN tail -n +2 /docker-entrypoint.sh >> /docker-entrypoint-modified.sh
# replace docker-entrypoint.sh with docker-entrypoint-modify
RUN cat /docker-entrypoint-modified.sh > /docker-entrypoint.sh
EXPOSE 7474 7473 7687
Turns out I had some misconceptions about docker-in-docker, I solved the problem by "rerouting" localhost to the docker-in-docker service by changing the following lines in my .gitlab-ci.yaml:
services:
- docker:dind
to
services:
- name: docker:dind
alias: localhost