How to replay Jenkins pipeline job that uses generic webhook HTTP POST content?

Question

I have a Jenkins Pipeline job that is triggered by a Bitbucket generic webhook. I.e. Jenkins has the Generic Webhook Trigger: ...and Bitbucket projects trigger this Jenkins project by adding a webhook like http://my_jenkins_server:8080/generic-webhook-trigger/invoke?token=foo

My Jenkinsfile uses the HTTP POST content -- which is JSON format -- that comes along with the invoking webhook. E.g. my Jenkinsfile has a section like this:

pipeline {
  agent any
  triggers {
    GenericTrigger (
      genericVariables: [
        [ key: "POST_actor_name", value: "\$.actor.name" ],
        [ key: "POST_actor_email", value: "\$.actor.emailAddress" ],
        [ key: "POST_ref_id", value: "\$.changes[0].refId" ],
        [ key: "POST_ref_display_id", value: "\$.changes[0].ref.displayId" ],
        [ key: "POST_commit", value: "\$.changes[0].toHash" ],
        [ key: "POST_repo_slug", value: "\$.repository.slug" ],
        [ key: "POST_project_key", value: "\$.repository.project.key" ],
        [ key: "POST_clone_url", value: "\$.repository.links.clone[1].href" ],
        [ key: "POST_pull_req_clone_url", value: "\$.pullRequest.fromRef.repository.links.clone[1].href" ],
        [ key: "POST_pull_req_id", value: "\$.pullRequest.id" ],
        [ key: "POST_pull_req_from_branch", value: "\$.pullRequest.fromRef.displayId" ],
        [ key: "POST_pull_req_to_branch", value: "\$.pullRequest.toRef.displayId" ],
        [ key: "POST_pull_req_repo_slug", value: "\$.pullRequest.toRef.repository.slug" ],
        [ key: "POST_pull_req", value: "\$.pullRequest.links.self[0].href" ],
        [ key: "POST_pull_req_url", value: "\$.pullRequest.links.self[0].href" ],
      ],

      causeString: '$committer_name pushed ref $ref to $clone_url referencing $commit',
      token: "foo", 
      printContributedVariables: true,
      printPostContent: true,
    )
  }
...

Question: how can someone replay an existing build?

If I click on the Replay button for an existing build: ...the build fails, and I get this little snippet in the build log:

[Pipeline] readJSON (hide)
[Pipeline] readJSON
[Pipeline] error

I believe this is indicating a readJSON error, because the replayed job is not triggered by a real HTTP POST, therefore there is no JSON content for that triggers.GenericTrigger.genericVariables section (posted above) to parse. Is that a correct assessment of the build error?

I imagine that generic webhooks triggering Jenkins pipeline jobs that read the HTTP POST content is common. I also imagine that the need to replay past Jenkins builds is common. Therefore I'm wondering if there's an idiom or common approach to providing a way to retrigger past Jenkins pipeline jobs that depend on HTTP POST content from the triggering generic webhook. I'm too inexperienced here to know if, for example, there's some mechanism by which the original HTTP POST content can be cached and re-sent to the replayed job. Or is there a way to retrigger the pipeline from Bitbucket without pushing a dummy change? (Git activities such as pushing new commits to the Bitbucket repo trigger the repo's webhooks).

Answer 1

Solution: Use Backing Parameters

Taking a minimal version of your example then, I believe you need something like below. I haven't tested this exact example but hope you get the idea.

The main idea is this: Create backing parameters with the exact same name as the generic variable keys you create. In fact, if you look at the generic webhook trigger from the "Configure" page of your job, you'll see a note about backing generic webhook variables with parameters:

If your job is not parameterized, then the resolved variables will just be contributed to the build. If your job is parameterized, and you resolve variables that have the same name as those parameters, then the plugin will populate the parameters when triggering job. That means you can, for example, use the parameters in combination with an SCM plugin, like GIT Plugin, to pick a branch.

Key takeaway: parameter names must match exactly the keys of your generic variables.

pipeline {
  agent any

//This is I believe the main bit you were missing
parameters{
    //Note: you can also use choice parameters instead of string here.
    string(name: 'committer_name', description: 'ENTER A DESCRIPTION FOR YOUR PARAMETER HERE'),        
    string(name: 'ref', description: 'ENTER A DESCRIPTION FOR YOUR PARAMETER HERE'),
    string(name: 'clone_url', description: 'ENTER A DESCRIPTION FOR YOUR PARAMETER HERE'),
    string(name: 'commit', description: 'ENTER A DESCRIPTION FOR YOUR PARAMETER HERE')
}

  triggers {
    GenericTrigger (
      genericVariables: [
        //Using single quotes for values because AFAIK they should just be strings corresponding to JSON path in JSON payload of POST request
        [ key: 'committer_name', value: '$.actor.name' ],        
        [ key: 'ref', value: '$.changes[0].refId' ],
        [ key: 'clone_url', value: '$.repository.links.clone[1].href' ],
        [key: 'commit', value: '$.changes[0].fromHash'] ]
      ],

      //Using single quotes to avoid interpolating the parameters here. In practice, I've observed that this sets the cause string after the parameters are read from the POST data. If you try to interpolate with double quotes, you either get a premature/default parameter value, or worse, if the parameter is unknown to Jenkins yet because you're running from SCM and the parameter isn't loaded yet, you can get a vicious circle where the job fails before the parameter even loads, and you then have to add it manually to get it to work
      causeString: '$committer_name pushed ref $ref to $clone_url referencing $commit',
      token: "foo", 
      printContributedVariables: true,
      printPostContent: true,
    )
  }
...
//You can use the parameters in the rest of your job, and they will also be captured in the build so that you can replay the build - which is what this Q. asks

My Advice: Separate Jobs into Trigger/Workhorse

This isn't a direct answer - but just my own advice. I find it a cleaner approach to separate jobs into two:

• One job is the "workhorse" let's say, that takes a list of parameters and does some work
• Another job is the "trigger" job - this one's responsibility is just to receive generic webhook requests, do any validation/cleaning of parameters, and then trigger the workhorse as a downstream job. I use the the "Pipeline: Build Step" plugin for this.

One advantage here is separation of concerns- the workhorse doesn't have to worry or get cluttered by generic webhook stuff, and the generic webhook one can focus on processing the webhook.

Another nice thing is that the trigger job can handle at least some validation of the webhook parameters, clean them, and give a human a nice detailed view of what went on with the webhook without the detail of the workhorse job.

In practice, because I do things this way, I rarely end up replaying a trigger job anyway - I usually just replay the workhorse job, which has all the parameters passed down to it. Having said that, even with my trigger jobs I can replay them if I wish, because I back up everything with parameters - as per the solution presented above.

Notes

• Not sure why you are prefixing your generic variables with POST_ in the original question - as mentioned the keys used should simply be an exact match for the parameter names.
• If you are running this from and SCM script, then the first time you run the job or if you add any new parameters or modify any parameter names, it won't have the parameters - this is standard Jenkins behaviour with parameters AFAIK and you just need to re-run the job - the second time it'll have the parameters and will work.
• This is a bit off topic for the question, but I noticed from capturing Bitbucket push events that it seems to be the fromHash that's populated. For me, I observed the toHash was always just a string of zeros. But I'm just looking at push events - maybe you're processing other events here.