Search code examples
gitlabartifactorygitlab-cigitlab-ci-runnerjfrog-cli

Gitlab-Ci. Transfer maven m2 local repository between stages

Good day collegues. I use gitlab ci in production. I have a lot of stages. 1)Build artifact 2)Deploy to external server 3)Using jfrog cli to deploy to artifactory

I have a problem with caching maven local repository. My runners download all dependicies in first step(build) and do the same in the last step(deploy to artifactory). Also my runner delete all data from m2 folder before final stage:

Removing .m2/antlr/
Removing .m2/aopalliance/
Removing .m2/asm/
Removing .m2/avalon-framework/
Removing .m2/backport-util-concurrent/
Removing .m2/ch/
Removing .m2/classworlds/
Removing .m2/com/
Removing .m2/commons-beanutils/
Removing .m2/commons-chain/
Removing .m2/commons-cli/
Removing .m2/commons-codec/
Removing .m2/commons-collections/
Removing .m2/commons-digester/
Removing .m2/commons-io/
Removing .m2/commons-lang/
Removing .m2/commons-logging/
Removing .m2/commons-validator/
Removing .m2/dom4j/
Removing .m2/io/
Removing .m2/javax/
Removing .m2/junit/
Removing .m2/log4j/
Removing .m2/logkit/
Removing .m2/net/
Removing .m2/org/
Removing .m2/oro/
Removing .m2/sslext/
Removing .m2/xml-apis/
Removing .m2/xmlunit/
Removing jfrog
Removing target/

my gitlav-ci yaml(withou second step):

stages:
- build
- deploy-artifactory

variables:
  MAVEN_OPTS: "-Dmaven.repo.local=${CI_PROJECT_DIR}/.m2"
  MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode"

cache:
  key: "$CI_JOB_NAME"
  paths:
  - .m2/

build:
  image: maven:latest
  stage: build
  tags:
  - build
  script:
  - adduser --disabled-password --gecos '' mynonrootuser
  - chmod --recursive 777 .
  - git version
  - su --command='mvn versions:set -DgenerateBackupPoms=false -DnewVersion="$CI_COMMIT_REF_SLUG-SNAPSHOT"' mynonrootuser
  - su --command='mvn $MAVEN_CLI_OPTS clean install -B -f ./pom.xml' mynonrootuser
  artifacts:
    expire_in: 5 mins
    paths:
    - target/*.jar

  only:
  - develop

deploy-artifactory-snapshot:
  retry: 2
  image: maven:latest
  stage: deploy-artifactory
  tags:
  - release
  before_script:
  -  curl -fL https://getcli.jfrog.io | sh
  - ./jfrog rt config --url=${ARTIFACTORY_URL} --user=${ARTIFACTORY_USER} --password=${ARTIFACTORY_PASSWORD}
  - ./jfrog rt c show
  - export M2_HOME=/usr/share/maven
  - sed -i 's,MAVEN_REPO_SNAPSHOT_DEPLOYER,'"$MAVEN_REPO_SNAPSHOT_DEPLOYER"',g' configuration.yml
  - sed -i 's,MAVEN_REPO_RELEASES_DEPLOYER,'"$MAVEN_REPO_RELEASES_DEPLOYER"',g' configuration.yml
  - sed -i 's,MAVEN_REPO_SNAPSHOT_RESOLVER,'"$MAVEN_REPO_SNAPSHOT_RESOLVER"',g' configuration.yml
  - sed -i 's,MAVEN_REPO_RELEASES_RESOLVER,'"$MAVEN_REPO_RELEASES_RESOLVER"',g' configuration.yml
  script:
  - ./jfrog rt mvn "versions:set -DgenerateBackupPoms=false -Dartifactory.publish.artifacts=false -DnewVersion="$CI_COMMIT_REF_SLUG-SNAPSHOT"" configuration.yml --build-name=scdfrestrunner --build-number=$CI_JOB_ID
  - ./jfrog rt mvn "clean install" configuration.yml --build-name=scdfrestrunner --build-number=$CI_JOB_ID
  - ./jfrog rt bce scdfrestrunner $CI_JOB_ID
  - ./jfrog rt bp scdfrestrunner $CI_JOB_ID

  only:
  - develop
Solution

  • There are several ways to handle dependencies between build jobs/stages.

    caching, guide is available here.

    e.g cache section for m2

    # Cache modules in between jobs
cache:
  key: ${CI_COMMIT_REF_SLUG} # cache is for per branch
  paths:
  - ${CI_PROJECT_DIR}/.m2

    caching also can be carried out using something like minio (open source s3 like storage solution). Integration with gitlab guide is available here. But since caching requires zipping and unzipping large amounts of artifacts may introduce slowness (but obviously way speed than without cache). To avoid that problem docker volumes can be used.

    docker volume

    basically we can use a out of the box docker volume say ~/.m2 mounted from host machine location. If you using in kubernetes cluster you can leverage kubnernets volume solutions for this. You have to customize your build image bit for support mounting dependency dir, following is an example of mvn build image. This approach is better because you can't cache outside of your build dir.

    e.g. docker image

    FROM maven:3.5.4-jdk-8
# m2 dir on docker container
ENV MAVEN_OPTS "-Dmaven.repo.local=/.m2/repository"
ENV MAVEN_CLI_OPTS "-s /usr/local/.m2/settings.xml --batch-mode"

ADD .m2 /usr/local/.m2 # copy settings.xml to build image
# same as the m2 dir specified in MAVEN_OPTS
VOLUME /.m2

    Artifacts

    If you just want to pass your build dir e.g. /target, you can use gitlab artifacts solution, guide is available here.