I am writing a simple test to verify that Kyverno is able to block images without attestation from being deployed in k3s cluster.
https://github.com/whoissqr/cg-test-keyless-sign
I have the following ClusterPolicy
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: check-image-keyless
spec:
validationFailureAction: Enforce
failurePolicy: Fail
webhookTimeoutSeconds: 30
rules:
- name: check-image-keyless
match:
any:
- resources:
kinds:
- Pod
verifyImages:
- imageReferences:
- "ghcr.io/whoissqr/cg-test-keyless-sign"
attestors:
- entries:
- keyless:
subject: "https://github.com/whoissqr/cg-test-keyless-sign/.github/workflows/main.yml@refs/heads/main"
issuer: "https://token.actions.githubusercontent.com"
rekor:
url: https://rekor.sigstore.dev
and the following pod yaml
apiVersion: v1
kind: Pod
metadata:
name: cg
namespace: app
spec:
containers:
- image: ghcr.io/whoissqr/cg-test-keyless-sign
name: cg-test-keyless-sign
resources: {}
And, I purposely commented out the image cosign step in Github action so that the cosign verify failed as expected, but the pod deployment to k3s is still succeeded. What am I missing here?
name: Publish and Sign Container Image
on:
schedule:
- cron: '32 11 * * *'
push:
branches: [ main ]
# Publish semver tags as releases.
tags: [ 'v*.*.*' ]
pull_request:
branches: [ main ]
jobs:
build:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
id-token: write
steps:
- name: Checkout repository
uses: actions/checkout@v2
- name: Install cosign
uses: sigstore/[email protected]
- name: Check install!
run: cosign version
- name: Setup Docker buildx
uses: docker/setup-buildx-action@v2
- name: Log into ghcr.io
uses: docker/login-action@master
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and push container image
id: push-step
uses: docker/build-push-action@master
with:
push: true
tags: ghcr.io/${{ github.repository }}:latest
- name: Sign the images with GitHub OIDC Token
env:
DIGEST: ${{ steps.push-step.outputs.digest }}
TAGS: ghcr.io/${{ github.repository }}
COSIGN_EXPERIMENTAL: "true"
run: |
echo "dont sign image"
# cosign sign --yes "${TAGS}@${DIGEST}"
- name: Verify the images
run: |
cosign verify ghcr.io/whoissqr/cg-test-keyless-sign \
--certificate-identity https://github.com/whoissqr/cg-test-keyless-sign/.github/workflows/main.yml@refs/heads/main \
--certificate-oidc-issuer https://token.actions.githubusercontent.com | jq
- name: Create k3s cluster
uses: debianmaster/actions-k3s@master
id: k3s
with:
version: 'latest'
- name: Install Kyverno chart
run: |
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm install kyverno kyverno/kyverno -n kyverno --create-namespace
- name: Apply image attestation policy
run: |
kubectl apply -f ./k3s/policy-check-image-keyless.yaml
- name: Deploy pod to k3s
run: |
set -x
# kubectl get nodes
kubectl create ns app
sleep 20
# kubectl get pods -n app
kubectl apply -f ./k3s/pod.yaml
kubectl -n app wait --for=condition=Ready pod/cg
kubectl get pods -n app
kubectl -n app describe pod cg
kubectl get polr -o wide
- name: Install Kyverno CLI
uses: kyverno/[email protected]
with:
release: 'v1.9.5'
- name: Check policy using Kyverno CLI
run: |
kyverno version
kyverno apply ./k3s/policy-check-image-keyless.yaml --cluster -v 10
in the GH action console
+ kubectl apply -f ./k3s/pod.yaml
pod/cg created
+ kubectl -n app wait --for=condition=Ready pod/cg
pod/cg condition met
+ kubectl get pods -n app
NAME READY STATUS RESTARTS AGE
cg 1/1 Running 0 12s
and the kyverno CLI output has
I0225 10:00:31.650505 6794 common.go:424] "msg"="applying policy on resource" "policy"="check-image-keyless" "resource"="app/Pod/cg"
I0225 10:00:31.652646 6794 context.go:278] "msg"="updated image info" "images"={"containers":{"cg-test-keyless-sign":{"registry":"ghcr.io","name":"cg-test-keyless-sign","path":"whoissqr/cg-test-keyless-sign","tag":"latest"}}}
I0225 10:00:31.654017 6794 utils.go:29] "msg"="applied JSON patch" "patch"=[{"op":"replace","path":"/spec/containers/0/image","value":"ghcr.io/whoissqr/cg-test-keyless-sign:latest"}]
I0225 10:00:31.659697 6794 mutation.go:39] EngineMutate "msg"="start mutate policy processing" "kind"="Pod" "name"="cg" "namespace"="app" "policy"="check-image-keyless" "startTime"="2024-02-25T10:00:31.659674165Z"
I0225 10:00:31.659737 6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"
I0225 10:00:31.659815 6794 rule.go:286] autogen "msg"="generating rule for cronJob"
I0225 10:00:31.659834 6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"
I0225 10:00:31.659940 6794 mutation.go:379] EngineMutate "msg"="finished processing policy" "kind"="Pod" "mutationRulesApplied"=0 "name"="cg" "namespace"="app" "policy"="check-image-keyless" "processingTime"="249.225µs"
I0225 10:00:31.659966 6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"
I0225 10:00:31.660040 6794 rule.go:286] autogen "msg"="generating rule for cronJob"
I0225 10:00:31.660059 6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"
I0225 10:00:31.660153 6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"
I0225 10:00:31.660218 6794 rule.go:286] autogen "msg"="generating rule for cronJob"
I0225 10:00:31.660236 6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"
I0225 10:00:31.660337 6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"
I0225 10:00:31.660402 6794 rule.go:286] autogen "msg"="generating rule for cronJob"
I0225 10:00:31.660421 6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"
I0225 10:00:31.660648 6794 discovery.go:269] dynamic-client "msg"="matched API resource to kind" "apiResource"={"name":"pods","singularName":"pod","namespaced":true,"version":"v1","kind":"Pod","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["po"],"categories":["all"]} "kind"="Pod"
I0225 10:00:31.660729 6794 imageVerify.go:121] EngineVerifyImages "msg"="processing image verification rule" "kind"="Pod" "name"="cg" "namespace"="app" "policy"="check-image-keyless" "ruleSelector"="All"
I0225 10:00:31.660889 6794 discovery.go:269] dynamic-client "msg"="matched API resource to kind" "apiResource"={"name":"daemonsets","singularName":"daemonset","namespaced":true,"group":"apps","version":"v1","kind":"DaemonSet","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["ds"],"categories":["all"]} "kind"="DaemonSet"
I0225 10:00:31.661037 6794 discovery.go:269] dynamic-client "msg"="matched API resource to kind" "apiResource"={"name":"deployments","singularName":"deployment","namespaced":true,"group":"apps","version":"v1","kind":"Deployment","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["deploy"],"categories":["all"]} "kind"="Deployment"
I0225 10:00:31.661184 6794 discovery.go:269] dynamic-client "msg"="matched API resource to kind" "apiResource"={"name":"jobs","singularName":"job","namespaced":true,"group":"batch","version":"v1","kind":"Job","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"categories":["all"]} "kind"="Job"
I0225 10:00:31.661327 6794 discovery.go:269] dynamic-client "msg"="matched API resource to kind" "apiResource"={"name":"statefulsets","singularName":"statefulset","namespaced":true,"group":"apps","version":"v1","kind":"StatefulSet","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["sts"],"categories":["all"]} "kind"="StatefulSet"
I0225 10:00:31.661465 6794 discovery.go:269] dynamic-client "msg"="matched API resource to kind" "apiResource"={"name":"replicasets","singularName":"replicaset","namespaced":true,"group":"apps","version":"v1","kind":"ReplicaSet","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["rs"],"categories":["all"]} "kind"="ReplicaSet"
I0225 10:00:31.661606 6794 discovery.go:269] dynamic-client "msg"="matched API resource to kind" "apiResource"={"name":"replicationcontrollers","singularName":"replicationcontroller","namespaced":true,"version":"v1","kind":"ReplicationController","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["rc"],"categories":["all"]} "kind"="ReplicationController"
I0225 10:00:31.661789 6794 validation.go:591] EngineVerifyImages "msg"="resource does not match rule" "kind"="Pod" "name"="cg" "namespace"="app" "policy"="check-image-keyless" "reason"="rule autogen-check-image-keyless not matched:\n 1. no resource matched"
I0225 10:00:31.661938 6794 discovery.go:269] dynamic-client "msg"="matched API resource to kind" "apiResource"={"name":"cronjobs","singularName":"cronjob","namespaced":true,"group":"batch","version":"v1","kind":"CronJob","verbs":["create","delete","deletecollection","get","list","patch","update","watch"],"shortNames":["cj"],"categories":["all"]} "kind"="CronJob"
I0225 10:00:31.662056 6794 validation.go:591] EngineVerifyImages "msg"="resource does not match rule" "kind"="Pod" "name"="cg" "namespace"="app" "policy"="check-image-keyless" "reason"="rule autogen-cronjob-check-image-keyless not matched:\n 1. no resource matched"
I0225 10:00:31.662091 6794 imageVerify.go:83] EngineVerifyImages "msg"="processed image verification rules" "applied"=0 "kind"="Pod" "name"="cg" "namespace"="app" "policy"="check-image-keyless" "successful"=true "time"="1.748335ms"
I0225 10:00:31.662113 6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"
I0225 10:00:31.662189 6794 rule.go:286] autogen "msg"="generating rule for cronJob"
I0225 10:00:31.662208 6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"
I0225 10:00:31.662302 6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"
I0225 10:00:31.662368 6794 rule.go:286] autogen "msg"="generating rule for cronJob"
I0225 10:00:31.662385 6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"
I0225 10:00:31.662481 6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"
I0225 10:00:31.662544 6794 rule.go:286] autogen "msg"="generating rule for cronJob"
I0225 10:00:31.662577 6794 rule.go:233] autogen "msg"="processing rule" "rulename"="check-image-keyless"
thanks!
==== edit 02/25/2024 8:24 PM =====
I noticed one small typo, and added tag 'latest' to image reference in the policy yaml, and now the 'kyverno apply' step failed as expected
policy check-image-keyless -> resource app/Pod/cg failed:
1. check-image-keyless: failed to verify image ghcr.io/whoissqr/cg-test-keyless-sign:latest: .attestors[0].entries[0].keyless: no matching signatures:
....
pass: 0, fail: 1, warn: 0, error: 0, skip: 98
Error: Process completed with exit code 1.
However, my 'kubectl apply -f ./k3s/pod.yaml' statement in previous step still proceed without error and pod are still created and running.
why?
==== 2nd edit =====
we need to add the following to the policy
background: false
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: check-image-keyless
spec:
validationFailureAction: Enforce
failurePolicy: Fail
background: false
webhookTimeoutSeconds: 30
rules:
- name: check-image-keyless
match:
any:
- resources:
kinds:
- Pod
verifyImages:
- imageReferences:
- "ghcr.io/whoissqr/cg-test-keyless-sign:latest"
attestors:
- entries:
- keyless:
subject: "https://github.com/whoissqr/cg-test-keyless-sign/.github/workflows/main.yml@refs/heads/main"
issuer: "https://token.actions.githubusercontent.com"
rekor:
url: https://rekor.sigstore.dev
pod yaml
apiVersion: v1
kind: Pod
metadata:
name: cg
namespace: app
spec:
containers:
- image: ghcr.io/whoissqr/cg-test-keyless-sign:latest
name: cg-test-keyless-sign
resources: {}
Github action
name: Publish and Sign Container Image
on:
schedule:
- cron: '32 11 * * *'
push:
branches: [ main ]
# Publish semver tags as releases.
tags: [ 'v*.*.*' ]
pull_request:
branches: [ main ]
jobs:
build:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
id-token: write
steps:
- name: Checkout repository
uses: actions/checkout@v2
- name: Install cosign
uses: sigstore/[email protected]
- name: Check install!
run: cosign version
- name: Setup Docker buildx
uses: docker/setup-buildx-action@v2
- name: Log into ghcr.io
uses: docker/login-action@master
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and push container image
id: push-step
uses: docker/build-push-action@master
with:
push: true
tags: ghcr.io/${{ github.repository }}:latest
- name: Sign the images with GitHub OIDC Token
env:
DIGEST: ${{ steps.push-step.outputs.digest }}
TAGS: ghcr.io/${{ github.repository }}
COSIGN_EXPERIMENTAL: "true"
run: |
echo "dont sign image"
# cosign sign --yes "${TAGS}@${DIGEST}"
- name: (optional) Verify the images
run: |
cosign verify ghcr.io/whoissqr/cg-test-keyless-sign \
--certificate-identity https://github.com/whoissqr/cg-test-keyless-sign/.github/workflows/main.yml@refs/heads/main \
--certificate-oidc-issuer https://token.actions.githubusercontent.com | jq
- name: Create k3s cluster
uses: debianmaster/actions-k3s@master
id: k3s
with:
version: 'latest'
- name: Install Kyverno chart
run: |
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm install --atomic kyverno kyverno/kyverno -n kyverno --create-namespace
sleep 10
- name: Apply image attestation policy
run: |
kubectl apply -f ./k3s/policy-check-image-keyless.yaml
- name: Deploy pod to k3s
if: always()
run: |
kubectl create ns app
kubectl apply -f ./k3s/pod.yaml
kubectl -n app wait --for=condition=Ready pod/cg
kubectl get pods -n app
- name: (optional) Install Kyverno CLI
if: always()
uses: kyverno/[email protected]
with:
release: 'v1.9.5'
- name: (optional) Dry run policy using Kyverno CLI
if: always()
run: |
kyverno version
# kyverno apply ./k3s/policy-check-image-keyless.yaml --cluster -v 10
kubectl get clusterpolicies -o yaml | kyverno apply - --resource ./k3s/pod.yaml -v 10