Can anyone post a working example of GitLab CI that has external submodules?

I am trying to get gitlab’s CI to work properly with an external submodule.

I have a submodule in ANOTHER repository, so no relative path.
I do NOT want to use a SSH key solution.
I want to use the CI token (CI_JOB_TOKEN).

Documentation is NOT clear because what is possible, or not, has changed and there are texts all over the place with many different approaches and, yet, nothing that fits the basic criteria.

It used to not be possible to pull submodules, with CI, if they had an absolute path, so people came up with various solutions. Then it became possible and there are a few solutions regarding authentication issues. But they all involve doing a clone which is not needed anymore because now we can set the CI to do a recursive pool.

This means that most of the online posts have become irrelevant and outdated and answers are one liners here and there but after a few hours none has worked for me.variables: GIT_SUBMODULE_STRATEGY: recursive

What has worked is to make a global admin key, giving access to everything but this is not a good solution at all.

What I have now is:

variables:
  GIT_SUBMODULE_STRATEGY: recursive

before_script:
- git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com/".insteadOf "git@gitlab.com:"
- git submodule sync && git submodule update --init

Which, according to Getting GitLab CI to clone private repositories should work.

But it fails with:

fatal: could not read Username for 'https://gitlab.com': No such device or address

Solution

I set GIT_SUBMODULE_STRATEGY: none and rewrote submodule URLs to use HTTPS with access/deploy tokens created to only allow reading of the repositories of the submodules. Something like this in .gitlab-ci.yaml:

image: docker:stable

variables:
  GIT_STRATEGY: clone
  # don't clone submodules by default
  GIT_SUBMODULE_STRATEGY: none

someJob:
  tags:
    - docker-priv
  before_script:
    - apk update
    - apk add git
    - git --version
    - git config --global url."https://${ACCESS_TOKEN_XYZ_NAME}:${ACCESS_TOKEN_XYZ}@GITLAB-HOST/GROUP/XYZ.git".insteadOf git@GITLAB-HOST/GROUP/XYZ.git
    - git config --global url."https://${ACCESS_TOKEN_XYZ_NAME}:${ACCESS_TOKEN_XYZ}@GITLAB-HOST/GROUP/XYZ.git".insteadOf https://GITLAB-HOST/GROUP/XYZ.git
    - git submodule sync --recursive
    - git submodule update --init --recursive

Please mind, that default names of deploy tokens in gitlab use a + sign and thus need to be url encoded when stored as CI variable for above usage. You may use one variable (gitlab%2Breadonly-token-name-123:randomtokenvalue) instead of two to make it a bit easier to read.