How to download artifacts from the last job execution when retrying a job in the same stage using Yaml pipelines?

Question

I have a single stage with 3 jobs:
Job A -> Always publishes artifact
Job B -> Always publishes artifact
Job C -> Depends on and uses the artifacts from Job A and Job B

Jobs A and B are retryable if they fail.

Ideally, jobs A and B would overwrite the artifact produced when retried but Yaml pipelines don't allow deleting artifacts and the suggested workaround is to put the Job ID in the artifact name.

How do I get the correct artifacts from Jobs A and B in Job C in the following scenario:
Pipeline is executed and Job A succeeds but Job B fails - both still publish artifacts which results in:
JobA_Artifact1
JobB_Artifact1

The failed Job B is retried and succeeds but then we end up with the following artifacts:
JobA_Artifact1
JobB_Artifact1
JobB_Artifact2

In Job C - how do I now get the latest artifact from Job A (which was from the previous execution) and the latest artifact from Job B? I'm hoping there's a better way than sticking a counter onto the end of the artifacts and then basically searching backwards for each artifact.

Answer 1

How to download artifacts from the last job execution when retrying a job in the same stage using Yaml pipelines?

Update:

After discussing with kmxp, I found that the solution I initially provided was not suitable for his situation. Let me share the final workaround.

Since the artifacts are not locally on the build agent but are published to the build pipeline artifact, so we could not clean it manually with some scripts.

To resolve this, we could use the variables for the artifact name, like:

JobA and JobB:

- powershell: |
    echo "##vso[task.setvariable variable=artifactJobId;isOutput=true]$(System.JobId)"
  Name: setArtifactJobId

JobC:

variables: 
  JobAArtifactId: $[ dependencies.JobA.outputs['setArtifactJobId.artifactJobId'] ]

But there is another situation that the variable is not accessible from Job C when the JobA or Jobb fail. We could add condition for above powershell task succeededOrFailed():

- powershell: |
    echo "##vso[task.setvariable variable=artifactJobId;isOutput=true]$(System.JobId)"
  Name: setArtifactJobId
  condition: succeededOrFailed()