Search code examples
artifactory

JFrog Artifactory - common repository - deleting many files is very slow - how to remove older files efficiently?

I am uploading screenshots from our UI integration tests from JaaS to the artifactory, so errors can be checked effectively.

I do not need to keep the screenshots of older builds, but I do want to keep the most recent ones for each run.

What I am currently trying is to send a delete request for the folder of the current PR before uploading new screenshots:

curl -sSf -u "$artifactoryUser:$artifactoryPassword" \
      -X DELETE \
      "$artifactoryScreenshotFolderUrl" &

But this takes such a long time (way over 10 minutes), that when the tests start to upload new screenshots, it is not done, and since the artifactory is still deleting files, it will also delete any file uploaded before it is completely done.

Only solution I see currently is to upload every run to a different folder and to not delete old runs, as I can never make sure a PR deletes old files anyway before it is merged.

Overwriting files is also not a solution, as when the test is failing in an earlier step, the next steps are still there from older runs, which is very confusing.

Other solution I thought of would be setting a expiry date for files and let the artifactory automatically delete them, but I could not find this functionality in the artifactory documentation, maybe it does not exist.

Solution

  • I assume you are deleting via UI or API.

    Did you try using the JFrog CLI?

    It can run with multiple threads (configurable) and is MUCH faster to complete.

    Example:

    jf rt del --url https://my-server/artifactory --user admin --password password --threads 10 my-repo/

    I use this method and it reduced my delete time by ~80% (depends on a few variables).