Search code examples
kuberneteskubernetes-helm

Pending-update issue in Helm in CI pipeline. How to deal with that?


I am experiencing a pending-update Helm issue, ie. in CI pipelines we sometimes deploy a particular application just one time after another. The first helm deployment is still in the pending-update state, while the other deployment of the same application (ran eg. 1 min later) cannot be deployed causing a well-known error: Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress.

I know it is a well-known issue and one possible solution could be Helm-related k8s secret deletion (automatically in the CI pipeline) just before the helm upgrade command but is it a good way of solving this? How do you deal with those scenarios in your CI pipelines?


Solution

  • Yes, you can solve it with a secret deletion in one of our cases it was also not working.

    So i would recommend checking the kubetl get events it will point out an error also if someone can not enable the logs of not getting debug log.

    i faced issues due to helm deploying creating the service Type: LoadBalancer and we were out of Quota for LoadBalancer in tenancy/account.

    We mostly check for the POD status but the helm might be having issues with service, secret, configmap, etc so make sure you do instead of guessing debug properly.

    Another workaround you can do is to roll back to the previous release version which will change the status from pending-update to deployed even if you are deploying the first time.

    There could be chances if anything is still pending it gets installed it will mark the release as deployed. So you upgrade or install a new one instead of getting an error.

    helm -n [NAMESPACE ] rollback <RELEASE> [REVISION]
    

    Example

    helm -n default rollback service 1
    

    this will mark your release as deployed as a workaround if stuck in state.