I am trying to use rules_k8s for Bazel to deploy to my Kubernetes cluster.
Thus I have this cloudbuild.yaml
file, which is executed by Google Cloud Build:
steps:
- name: gcr.io/cloud-builders/bazel
args: ['run', '//:kubernetes.apply']
(//:kubernetes
is just a k8s_objects
)
On my local machine running bazel run //:kubernetes.apply
works fine, but
although the Google Cloud Build succeeds, it logs those errors. So the configuration is not applied to my Kubernetes cluster:
Target //:kubernetes.apply up-to-date:
bazel-bin/kubernetes.apply
INFO: Elapsed time: 29.863s, Critical Path: 0.14s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/kubernetes.apply
INFO: Build Event Protocol files produced successfully.
INFO: Build completed successfully, 1 total action
kubectl toolchain was not properly configured so k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so projection_database_k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so projection_database_k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_service.apply cannot be executed.
I also get a warning from the bazel cache:
DEBUG: /builder/home/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/io_bazel_rules_k8s/toolchains/kubectl/kubectl_toolchain.bzl:28:9: No kubectl tool was found or built, executing run for rules_k8s targets might not work.
P.S.: I get the same errors when using //:kubernetes.create
Deployments
load("@io_bazel_rules_k8s//k8s:object.bzl", "k8s_object")
k8s_object(
name = "k8s_deployment",
kind = "deployment",
cluster = "gke_cents-ideas_europe-west3-a_cents-ideas",
template = ":ideas.deployment.yaml",
images = {
"gcr.io/cents-ideas/ideas:latest": ":image"
},
)
Services
k8s_object(
name = "k8s_service",
kind = "service",
cluster = "gke_cents-ideas_europe-west3-a_cents-ideas",
template = ":ideas.service.yaml",
)
Aggregations
load("@io_bazel_rules_k8s//k8s:objects.bzl", "k8s_objects")
k8s_objects(
name = "k8s",
objects = [
":k8s_deployment",
":k8s_service",
]
)
Final composition
k8s_objects(
name = "kubernetes",
objects = [
"//services/ideas:k8s",
# ...
]
)
I've now tried to make my own docker image with Bazel and kubectl:
FROM gcr.io/cloud-builders/bazel
RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
RUN chmod +x ./kubectl
RUN mv ./kubectl /usr/local/bin/kubectl
I pushed it to GCR and changed my cloudbuild.yaml
to:
steps:
- name: eu.gcr.io/cents-ideas/bazel-kubectl
args: ["run", "//:kubernetes.apply"]
I firstly noticed, that the step took way longer than before. However at the end it throws an error:
$ /usr/local/bin/kubectl --kubeconfig= --cluster=gke_cents-ideas_europe-west3-a_cents-ideas --context= --user= apply -f -
error: cluster "gke_cents-ideas_europe-west3-a_cents-ideas" does not exist
Here is the full log.
As for the updated question, now you need to authenticate somehow to GKE inside the container.
First thing, I recommend installing gcloud tool to your container.
Btw, as for the huge container size 1.2 GB, that's because cloud-builders/bazel
is huge :)
Have a look at our example on slim bazel container version: https://github.com/aspect-development/bazel-k8s-example/blob/master/tools/Dockerfile.dazel
And here is Dockerfile for installing gcloud and kubectl, so you can grab needed parts from both files: https://github.com/GoogleCloudPlatform/cloud-builders/blob/master/gcloud/Dockerfile
The second thing is authenticating, after gcloud is installed it should be easy. Overall cloudbuild step should look similar to this:
- name: <link to your container>
entrypoint: /bin/sh
args:
- -c
- |
gcloud container clusters get-credentials cents-ideas --zone europe-west3-a --project cents-ideas
bazel run //:kubernetes.apply