Gitlab CI extend services with includes instead of overwrite them

In our company we have central repo that everybody includes to run their pipelines.

Some of our projects require services to run. In order to also centralize settings we allow people to include settings like this:

# .gitlab-ci.yaml of a project with mariadb only
include:
  # Activate Python Pipeline
  - project: devops/gitlab
    file: /templates/python.yaml
  # Use mariadb inside the test step
  - project: devops/gitlab
    file: /templates/service_mariadb.yaml

They test step is a simple as this

# /templates/python.yaml
.test:python:
  image: custom.domain.tld/python
  script: 
    - pytest -vv
test:python:
  extends:
    - .test:python

Now the templates/service_mariadb.yaml extends the python test step:

# templates/service_mariadb.yaml
.service-mariadb:
  variables:
    MYSQL_HOST: 'mariadb'
    MYSQL_PORT: '3306'
    MYSQL_DATABASE: 'test'
    MYSQL_USER: 'root'
    MYSQL_ROOT_PASSWORD: 'some--dummy-password-very-insecure'
    DB_URL: 'mysql://$MYSQL_USER:$MYSQL_ROOT_PASSWORD@$MYSQL_HOST:$MYSQL_PORT/$MYSQL_DATABASE'
  services:
    - name: 'custom.domain.something/mariadb:latest'
      alias: 'mariadb'

test:python:
  extends:
    - .test:python
    - .service-mariadb

Now image a user also want activate kafka, they will include an additional /templates/service_mariadb.yaml like this

# .gitlab-ci.yaml of a project with mariadb and kafka
include:
  # Activate Python Pipeline
  - project: devops/gitlab
    file: /templates/python.yaml
  # Use mariadb inside the test step
  - project: devops/gitlab
    file: /templates/service_mariadb.yaml
  # Use kafka inside the test step
  - project: devops/gitlab
    file: /templates/service_kafka.yaml

# templates/service_kafka.yaml
.service-kafka:
  services:
    - name: 'custom.domain.something/kafka:latest'
      alias: 'kafka'
  variables:
    EVENTD_BROKER_URL: 'kafka:9094'
    KAFKA_ADVERTISED_LISTENERS: 'INTERNAL://:9092,EXTERNAL://kafka:9094'
test:python:
  extends:
    - .test:python
    - .service-kafka

But now the kafka step will overwrite the mariadb step.

The "full configuration" looks like this:

# Merged gitlab-ci yaml
# […]
test:python:
  image: custom.domain.tld/python
  script:
  - pytest -vv
  services:
  - name: custom.domain.something/kafka:latest
    alias: kafka
  variables:
    EVENTD_BROKER_URL: kafka:9094
    KAFKA_ADVERTISED_LISTENERS: INTERNAL://:9092,EXTERNAL://kafka:9094
  extends:
  - ".test:python"
  - ".service-kafka"
# […]

❔ How can I define templates of services that can be included and activate multiple different services❓

Background

We have centralized pipeline definitions to reduce our maintenance and support burden. Only few of our employees have CI knowledge. All employees should be able to use the pipelines without worrying about CI specific or even worse write custom .gitlab-ci.yaml files.

The optimal solution would be to provide only a generic gitlab-ci.yaml the contains one import and services can be activated through the existence of a .service/mariadb.yaml file using includes that depend on the existence of files.

Solution

Actually while writing the question a solution popped into my mind (really can recommend to write down questions here…):

The test script must be prepared to include every .service* definition but empty and the services need to be expanded by using !reference tags

So the python template looks like

# /templates/python.yaml
.service-mariadb:
  services: []

.service-kafka:
  services: []


.test:python:
  image: custom.domain.tld/python
  script: 
    - pytest -vv


test:python:
  services:
    - !reference [".service-mariadb", "services"]
    - !reference [".service-kafka", "services"]
  extends:
    - .test:python
    - .service-mariadb
    - .service-kafka

The service templates now only include the actual service definition

# templates/service_mariadb.yaml
.service-kafka:
  variables:
    MYSQL_HOST: 'mariadb'
    MYSQL_PORT: '3306'
    MYSQL_DATABASE: 'test'
    MYSQL_USER: 'root'
    MYSQL_ROOT_PASSWORD: 'some--dummy-password-very-insecure'
    DB_URL: 'mysql://$MYSQL_USER:$MYSQL_ROOT_PASSWORD@$MYSQL_HOST:$MYSQL_PORT/$MYSQL_DATABASE'
  services:
    - name: 'custom.domain.something/mariadb:latest'
      alias: 'mariadb'

# templates/service_kafka.yaml
.service-kafka:
  services:
    - name: 'custom.domain.something/kafka:latest'
      alias: 'kafka'
  variables:
    EVENTD_BROKER_URL: 'kafka:9094'
    KAFKA_ADVERTISED_LISTENERS: 'INTERNAL://:9092,EXTERNAL://kafka:9094'

And now an include containing different services:

# .gitlab-ci.yaml of a project with mariadb and kafka
include:
  # Activate Python Pipeline
  - project: devops/gitlab
    file: /templates/python.yaml
  # Use mariadb inside the test step
  - project: devops/gitlab
    file: /templates/service_mariadb.yaml
  # Use kafka inside the test step
  - project: devops/gitlab
    file: /templates/service_kafka.yaml

Will we combined correctly to:

# merged gitlab-ci-yaml
# […]
test:python:
  image: custom.domain.tld/python
  script:
  - pytest -vv
  services:
  - - name: custom.domain.something/mariadb:latest
      alias: mariadb
  - - name: custom.domain.something/kafka:latest
      alias: kafka
  variables:
    MYSQL_HOST: mariadb
    MYSQL_PORT: '3306'
    MYSQL_DATABASE: test
    MYSQL_USER: root
    MYSQL_ROOT_PASSWORD: some--dummy-password-very-insecure
    DB_URL: mysql://$MYSQL_USER:$MYSQL_ROOT_PASSWORD@$MYSQL_HOST:$MYSQL_PORT/$MYSQL_DATABASE
    EVENTD_BROKER_URL: kafka:9094
    KAFKA_ADVERTISED_LISTENERS: INTERNAL://:9092,EXTERNAL://kafka:9094
# […]