Search code examples
dockergitlabdevopsbuild-automationgitlab-ci-runner

Automatically download binaries from a public site, then index those files and make available to a gitlab runner?


  • If there's a better place to post this, I apologize and please let me know!

I have a cron job that runs on a server daily which downloads the latest version of a particular Chrome Extension from a public site. This works great, and I end up with a folder on that server with a bunch *.crx files, which I can then select from depending on which version I want.

Currently I manually copy the version I want into a git repo, commit that then push to my gitlab server which is already set up for build automation via it's integrated CI/CD system. This integrates the version of the Chrome Extension I've chosen into my build.

I would like to further automate this process by moving the crx downloading into a gitlab scheduled pipeline, then replacing the step of manually copying the crx file in the repo.

I am certain I can migrate the cron job to a script suitable for passing to a runner in a scheduled pipeline, and I'm also sure I can modify the build automation to allow a user to select a crx version at build time.

There's one part of this that I am totally stumped with: Where/How should I store the crx library?

My gitlab and runners are all set up in Docker, otherwise I would probably just set up an NFS share (or similar) and be done with it.

I guess I'm looking for some kind of artifactory which could integrate into gitlab, rather than something hacked together?

Jfrog is pretty renowned, though I've never used it, and it seems like overkill for this task. I can't seem to identify a FOSS alternative, otherwise I'd just spin that up and just experiment.

I could make a new git repo strictly for tracking and storing these crx files, then add that as a submodule to my main repository, but that seems a bit overkill as well.

Are there other approaches I haven't considered? Are the above ideas worth pursuing?


Solution

  • I would download the crx files at build time. They are not part of your code. The version I would keep in a file in git or env var. Better a file I guess. You can upload this artifact to Jfrog. But better to download them at building time.

    Dont make a git of them. They are binaries.