Search code examples
kubernetespulumi

Change Pulumi's timeout when deploying Kubernetes resources


When I deploy resources to Kubernetes with Pulumi, if I make a mistake, Pulumi will wait for the Kubernetes resources to be healthy.

     Type                                                                               Name                               Status                  Info
 +   pulumi:pulumi:Stack                                                                aws-load-balancer-controller-dev   **creating failed**     1 error
 +   ├─ jaxxstorm:aws:loadbalancercontroller                                            foo                                created
 +   ├─ kubernetes:yaml:ConfigFile                                                      foo-crd                            created
 +   │  └─ kubernetes:apiextensions.k8s.io/v1beta1:CustomResourceDefinition             targetgroupbindings.elbv2.k8s.aws  created                 1 warning
 +   ├─ kubernetes:core/v1:Namespace                                                    foo-namespace                      created
 +   ├─ kubernetes:core/v1:Service                                                      foo-webhook-service                **creating failed**     1 error
 +   ├─ kubernetes:rbac.authorization.k8s.io/v1:Role                                    foo-role                           created
 +   ├─ pulumi:providers:kubernetes                                                     k8s                                created
 +   ├─ aws:iam:Role                                                                    foo-role                           created
 +   │  └─ aws:iam:Policy                                                               foo-policy                         created
 +   ├─ kubernetes:core/v1:Secret                                                       foo-tls-secret                     created
 +   ├─ kubernetes:rbac.authorization.k8s.io/v1:ClusterRole                             foo-clusterrole                    created
 +   ├─ kubernetes:admissionregistration.k8s.io/v1beta1:ValidatingWebhookConfiguration  foo-validating-webhook             created                 1 warning
 +   ├─ kubernetes:admissionregistration.k8s.io/v1beta1:MutatingWebhookConfiguration    foo-mutating-webhook               created                 1 warning
 +   └─ kubernetes:core/v1:ServiceAccount                                               foo-serviceAccount                 **creating failed**     1 error
 C
Diagnostics:
  kubernetes:core/v1:ServiceAccount (foo-serviceAccount):
    error: resource aws-load-balancer-controller/foo-serviceaccount was not successfully created by the Kubernetes API server : ServiceAccount "foo-serviceaccount" is invalid: metadata.labels: Invalid value: "arn:aws:iam::616138583583:role/foo-role-10b9499": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')

  kubernetes:core/v1:Service (foo-webhook-service):
    error: 2 errors occurred:
        * resource aws-load-balancer-controller/foo-webhook-service-4lpopjpr was successfully created, but the Kubernetes API server reported that it failed to fully initialize or become live: Resource operation was cancelled for "foo-webhook-service-4lpopjpr"
        * Service does not target any Pods. Selected Pods may not be ready, or field '.spec.selector' may not match labels on any Pods

Is there a way to disable this so that I don't have to send a signal to Pulumi to terminate?


Solution

  • Pulumi has special await logic on Kubernetes resources. You can read more about this here

    Pulumi will wait for Kubernetes resources to be "healthy". The definition of "healthy" can change depending on the resource being created, but generally Pulumi will wait for the resource to:

    • Exist
    • Have a ready state (if the resources has one)

    You can skip this logic by adding an annotation to that resource, like so:

    pulumi.com/skipAwait: "true"
    

    You can also change the timeout, or how long Pulumi will wait, using the following example:

    pulumi.com/timeoutSeconds: 600
    

    This gets added to any Kubernetes resource you're managing with Pulumi, so for example, a service resource might look like this (with Pulumi's typescript SDK):

    const service = new k8s.core.v1.Service(`${name}-service`, {
      metadata: {
        namespace: "my-service",
      },
      annotations: {
        "pulumi.com/timeoutSeconds": "60" // Only wait 1 minute for pulumi to timeout
        "pulumi.com/skipAwait": "true" // don't use the await logic at all
    }
      spec: {
        ports: [{
          port: 443,
          targetPort: 9443,
        }],
        selector: {
          "app.kubernetes.io/name": "my-deployment",
          "app.kubernetes.io/instance": "foo",
        },
     },
    });