I have setup a pipeline that has a health check stage. It is a webhook stage that will hit a an endpoint in a custom server. This server will check the healthiness of deployed application and return a 200 if it's ready, or a 500 if not.
This stage currently has a timeout of 10 minutes, and is configured by this value in the execution options: timeout configuration which can also be described as json with:
"overrideTimeout": true,
"stageTimeoutMs": 600000, // 10 minutes
But the stage still fails on 5 minutes (plus a 1 to 9 seconds, which I believe is the time it takes to retry).
If I lower that stageTimeoutMs to less than 300000 (less than 5 minutes) it works as expected, but increasing it does not. Is there anything else that need to be configured globally to allow more than 5 minutes for webhook stages?
BTW, the pipeline itself can continue after this (proving it's not a pipeline timeout, but this stage specifically), and manual judgement stages (and some other types) can happily go over 5 minutes, just webhooks fail.
I'm currently working with Spinnaker 1.19.13
Looks like 500 as a response to webhook endpoint will cause it to get stuck at "Create Webhook" task, which seems to really have the timeout hardcoded to 5 minutes. By changing my endpoint to return a 202 and a meaningful string on $.status (the statusJsonPath) I was able to make it get past the "Create Webhook" task and start the "Monitor Webhook" one, which can have its timeout overridden.
I think this shows lack of documentation and a not very intuitive behavior on Spinnaker side, but the solution doesn't look hacky. If folks are facing similar issue out there, hopefully this is helpful.