Search code examples
javamavengitlab

Problem with a GitLab project that publishes to the package registry cannot pull from another package registry

All of this code is Java and is built with Maven. I used the guidance provided here and here to make the changes to the configurations.

I have a GitLab project that provides a library of commonly used classes to other projects (let’s call this lib-1). I have put in the config modifications (in both the provider and consumers) in the pom.xml, GitLab project settings, etc. to publish the built library to its GitLab project package registry. I have confirmed that this all works, including consumer projects being able to use this library.

I have developed another, separate library (in a separate GitLab project) that provides a different set of common classes (let’s call this lib-2). The project above (lib-1) also needs to use this library. lib-2 has also been set up to push to its project package registry and I have verified that it pushes to the registry. I have also verified that an (unrelated) consumer project is able to pull and use it.

I’m running into a problem where lib-1 fails to build because it cannot pull the lib-2 library.

Here is the error:

[ERROR] Failed to execute goal on project prometheus-common: Could not resolve dependencies for project com.beast-code.devops:prometheus-common:jar:1.4.0: Failed to collect dependencies at com.beastcode.devops:rest-api-utility:jar:3.0.0: Failed to read artifact descriptor for com.beastcode.devops:rest-api-utility:jar:3.0.0: Could not transfer artifact com.beastcode.devops:rest-api-utility:pom:3.0.0 from/to gitlab-maven-rest (https://gitlab.phactory.beast-code.com/api/v4/projects/1155/packages/maven): Transfer failed for https://gitlab.phactory.beast-code.com/api/v4/projects/1155/packages/maven/com/beastcode/devops/rest-api-utility/3.0.0/rest-api-utility-3.0.0.pom: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target -> [Help 1]

prometheus-common is lib-1 and rest-api-utility is lib-2. The crux of the error seems to indicate a problem with a certificate, but I don’t know how or where that’s coming into the picture. As I mentioned earlier, I have an unrelated consumer of lib-2 that is able to pull this library without any issues.

Here are various files from both lib-1 and lib-2.

= lib-1 (AKA prometheus-common)

pom.xml (relevant sections):

<properties>
  <gitlab-api-url>https://gitlab.phactory.beast-code.com/api/v4</gitlab-api-url>
  <prometheus-common.project-id>863</prometheus-common.project-id>
  
  <rest-api-utility.project-id>1155</rest-api-utility.project-id>
  <rest-api-utility.version>3.0.0</rest-api-utility.version>
</properties>

<repositories>
  <repository>
    <id>gitlab-maven-rest-api</id>
    <url>${gitlab-api-url}/projects/${rest-api-utility.project-id}/packages/maven</url>
  </repository>

  <repository>
    <id>gitlab-maven-prometheus-common</id>
    <url>${gitlab-api-url}/projects/${prometheus-common.project-id}/packages/maven</url>
  </repository>
</repositories>

<distributionManagement>
  <repository>
    <id>gitlab-maven-prometheus-common</id>
    <url>${gitlab-api-url}/projects/${prometheus-common.project-id}/packages/maven</url>
  </repository>

  <snapshotRepository>
    <id>gitlab-maven-prometheus-common</id>
    <url>${gitlab-api-url}/projects/${prometheus-common.project-id}/packages/maven</url>
  </snapshotRepository>
</distributionManagement>

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>com.beastcode.devops</groupId>
      <artifactId>rest-api-utility</artifactId>
      <version>${rest-api-utility.version}</version>
    </dependency>
  </dependencies>
</dependencyManagement>

<dependencies>
  <dependency>
    <groupId>com.beastcode.devops</groupId>
    <artifactId>rest-api-utility</artifactId>
  </dependency>
</dependencies>

ci_settings.xml:

<settings xmlns="http://maven.apache.org/SETTINGS/1.1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd">
  <servers>
    <server>
      <id>gitlab-maven-prometheus-common</id>
      <configuration>
        <httpHeaders>
          <property>
            <name>Job-Token</name>
            <value>${CI_JOB_TOKEN}</value>
          </property>
        </httpHeaders>
      </configuration>
    </server>
  </servers>
</settings>

.gitlab-ci.yml (relevant sections):

push-to-package-registry:
  stage: push
  image: maven:3.6-jdk-11
  script:
    - 'mkdir ./certs'
    - 'export CRT_FILE=./certs/BCRCA.crt'
    - 'export CA_CERTS_PATH=$JAVA_HOME/lib/security/cacerts'
    - 'echo "$BCRCA" > $CRT_FILE'
    - 'keytool -import -noprompt -trustcacerts -alias BCRCA -file $CRT_FILE -keystore $CA_CERTS_PATH -storepass changeit'
    - 'keytool -list -keystore $CA_CERTS_PATH -alias BCRCA -noprompt -storepass changeit'
    - 'rm -rf ./certs'
    - 'mvn clean deploy -s ci_settings.xml -DskipTests'
  rules:
    - if: $CI_COMMIT_REF_PROTECTED == "true" && $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH

BCRCA contains a root CA signed x.509 certificate.

= lib-2 (AKA rest-api-utility)

pom.xml (relevant sections):

<properties>
  <gitlab-api-url>https://gitlab.phactory.beast-code.com/api/v4</gitlab-api-url>
  <gitlab-project-id>1155</gitlab-project-id>
</properties>

<repositories>
  <repository>
    <id>gitlab-maven</id>
    <url>${gitlab-api-url}/projects/${gitlab-project-id}/packages/maven</url>
  </repository>
</repositories>

<distributionManagement>
  <repository>
    <id>gitlab-maven</id>
    <url>${gitlab-api-url}/projects/${gitlab-project-id}/packages/maven</url>
  </repository>

  <snapshotRepository>
    <id>gitlab-maven</id>
    <url>${gitlab-api-url}/projects/${gitlab-project-id}/packages/maven</url>
  </snapshotRepository>
</distributionManagement>

ci_settings.xml:

<settings xmlns="http://maven.apache.org/SETTINGS/1.1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd">
  <servers>
    <server>
      <id>gitlab-maven</id>
      <configuration>
        <httpHeaders>
          <property>
            <name>Job-Token</name>
            <value>${CI_JOB_TOKEN}</value>
          </property>
        </httpHeaders>
      </configuration>
    </server>
  </servers>
</settings>

.gitlab-ci.yml (relevant sections):

push-to-package-registry:
  stage: push
  image: maven:3.6-jdk-11
  script:
    - 'mkdir ./certs'
    - 'export CRT_FILE=./certs/BCRCA.crt'
    - 'export CA_CERTS_PATH=$JAVA_HOME/lib/security/cacerts'
    - 'echo "$BCRCA" > $CRT_FILE'
    - 'keytool -import -noprompt -trustcacerts -alias BCRCA -file $CRT_FILE -keystore $CA_CERTS_PATH -storepass changeit'
    - 'keytool -list -keystore $CA_CERTS_PATH -alias BCRCA -noprompt -storepass changeit'
    - 'rm -rf ./certs'
    - 'mvn clean deploy -s ci_settings.xml -DskipTests'
  rules:
    - if: $CI_COMMIT_REF_PROTECTED == "true" && $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH

BCRCA contains a root CA signed x.509 certificate.

Has anyone had this use case and run into this before? The GitLab documentation covers quite a bit of what needs to be done, but I think there are a few areas that got glossed over and left to the developer to figure out. They do not cover a situation like what I'm doing here.

On a bit of a side topic:

All of the logic in each "push to regsitry" job before the maven deploy (to the best of my knowledge), installs a root CA signed X.509 certificate into the JVM trust store packaged into in the image. I'm not 100% sure what is going on here, but I have a memory that it has to be done, otherwise something will fail. I'm curious whether this may have anything to do with the issue above. The error message seems to indicate a certificate problem.

Solution

  • I finally figured out what I needed to do. In the lib-1 library .gitlab-ci.yml file there is a job that runs prior to the push-to-package-registry job that builds and runs tests on the built library. I needed to add the same logic that installs a trusted certificate into the Java trust store in the "push" job, into the "build" job.

    Here is the job, plus the logic that fixed the issue:

    build-and-test:
  stage: build-test
  image: maven:3.6-jdk-11
  before_script:
    - 'mkdir ./certs'
    - 'export CRT_FILE=./certs/BCRCA.crt'
    - 'export CA_CERTS_PATH=$JAVA_HOME/lib/security/cacerts'
    - 'echo "$BCRCA" > $CRT_FILE'
    - 'keytool -import -noprompt -trustcacerts -alias BCRCA -file $CRT_FILE -keystore $CA_CERTS_PATH -storepass changeit'
    - 'keytool -list -keystore $CA_CERTS_PATH -alias BCRCA -noprompt -storepass changeit'
    - 'rm -rf ./certs'
  script:
    - 'mvn clean compile test -s ci_settings.xml'
  artifacts:
    when:
      always
    paths:
      - target/surefire-reports
    reports:
      junit:
        - target/surefire-reports/*.xml

    The logic in the before_script is what was needed.