Search code examples
kubernetesclient-certificatesrbackubectl

How to create users/groups restricted to namespace in Kubernetes using RBAC API?

Problem

I'd like to issue certs to many different developers (different subjects) all within the dev group, and have them all have access to create and modify things within the dev namespace, but not touch anything outside it, and definitely not see secrets outside it. I suspect the roles, role bindings, etc. I'm creating in step 2 below are not correct, can anyone suggest corrections?

Attempt

  1. Deployed Kubernetes with API Server flags to support "RBAC,AlwaysAllow" authorization modes, set RBAC super user, and enable RBAC API via --runtime-config.
  2. Created a namespace, role, and role binding with the intent that (a) service accounts and system components can effectively still have "AlwaysAllow" access, and (b) any entity in group dev can access anything in namespace dev using this YAML file. NOTE: contents of this link have changed, see YAML files I got working at bottom of question.
  3. Updated Kubernetes to only allow "RBAC" authorization mode.
  4. Generated client TLS data where the certificate subject flag (for openssl) was -subj "/[email protected]/O=dev".
  5. Generated a kubeconfig file following this template.

Actual Result

I get the following errors when running: kubectl -v 8 --kubeconfig=/tmp/dev-kube-config.yml create -f /tmp/busybox.yml:

I1219 16:12:37.584657   44323 loader.go:354] Config loaded from file /tmp/dev-kube-config.yml
I1219 16:12:37.585953   44323 round_trippers.go:296] GET https://api.kubernetes.click/api
I1219 16:12:37.585968   44323 round_trippers.go:303] Request Headers:
I1219 16:12:37.585983   44323 round_trippers.go:306]     Accept: application/json, */*
I1219 16:12:37.585991   44323 round_trippers.go:306]     User-Agent: kubectl/v1.5.1+82450d0 (    darwin/amd64) kubernetes/82450d0
I1219 16:12:38.148994   44323 round_trippers.go:321] Response Status: 403 Forbidden in 562     milliseconds
I1219 16:12:38.149056   44323 round_trippers.go:324] Response Headers:
I1219 16:12:38.149070   44323 round_trippers.go:327]     Content-Type: text/plain; charset=utf-    8
I1219 16:12:38.149081   44323 round_trippers.go:327]     Content-Length: 17
I1219 16:12:38.149091   44323 round_trippers.go:327]     Date: Tue, 20 Dec 2016 00:12:38 GMT
I1219 16:12:38.149190   44323 request.go:904] Response Body: Forbidden: "/api"
I1219 16:12:38.149249   44323 request.go:995] Response Body: "Forbidden: \"/api\""
I1219 16:12:38.149567   44323 request.go:1151] body was not decodable (unable to check for     Status): Object 'Kind' is missing in 'Forbidden: "/api"'
...
I1219 16:12:38.820672   44323 round_trippers.go:296] GET https://api.kubernetes.    click/swaggerapi/api/v1
I1219 16:12:38.820702   44323 round_trippers.go:303] Request Headers:
I1219 16:12:38.820717   44323 round_trippers.go:306]     User-Agent: kubectl/v1.5.1+82450d0 (    darwin/amd64) kubernetes/82450d0
I1219 16:12:38.820731   44323 round_trippers.go:306]     Accept: application/json, */*
I1219 16:12:38.902256   44323 round_trippers.go:321] Response Status: 403 Forbidden in 81     milliseconds
I1219 16:12:38.902306   44323 round_trippers.go:324] Response Headers:
I1219 16:12:38.902327   44323 round_trippers.go:327]     Content-Type: text/plain; charset=utf-    8
I1219 16:12:38.902345   44323 round_trippers.go:327]     Content-Length: 31
I1219 16:12:38.902363   44323 round_trippers.go:327]     Date: Tue, 20 Dec 2016 00:12:38 GMT
I1219 16:12:38.902456   44323 request.go:904] Response Body: Forbidden: "/swaggerapi/api/v1"
I1219 16:12:38.902512   44323 request.go:995] Response Body: "Forbidden:     \"/swaggerapi/api/v1\""
F1219 16:12:38.903025   44323 helpers.go:116] error: error validating "/tmp/busybox.yml": error validating data: the server does not allow access to the requested resource; if you choose to ignore these errors, turn validation off with --validate=false

Expected Result

Expected to create busybox pod in dev namespace.

Additional details:

  • $ kubectl version

    Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1+82450d0", GitCommit:"82450d03cb057bab0950214ef122b67c83fb11df", GitTreeState:"not a git tree", BuildDate:"2016-12-14T04:09:31Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.6", GitCommit:"e569a27d02001e343cb68086bc06d47804f62af6", GitTreeState:"clean", BuildDate:"2016-11-12T05:16:27Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

  • GitHub issue: https://github.com/kubernetes/kubernetes/issues/38997

  • Mailing list post: https://groups.google.com/forum/#!topic/kubernetes-dev/6TBTu1AC2L8

EDIT: Working solution based on answer and comments

Based on Jordan's answer below, I upgraded to Kubernetes v1.5.1 and then got the following two YAML files to construct the namespace and all the correct RBAC resources so that everything works as desired:

system-access.yml (because the out-of-the-box cluster roles and cluster role bindings didn't seem to work):

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1alpha1
metadata:
  name: system:node--kubelet
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node
subjects:
- kind: User
  name: kubelet
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1alpha1
metadata:
  name: cluster-admin--kube-system:default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: default
  namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1alpha1
metadata:
  name: system:node-proxier--kube-proxy
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node-proxier
subjects:
- kind: User
  name: kube-proxy

dev-access.yml:

kind: Namespace
apiVersion: v1
metadata:
  name: dev
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1alpha1
metadata:
  namespace: dev
  name: dev-all
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["*"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1alpha1
metadata:
  name: dev-role-dev-all-members
  namespace: dev
subjects:
  - kind: Group
    name: dev
  - kind: Group
    name: system:serviceaccounts:dev
roleRef:
  kind: Role
  name: dev-all
  apiGroup: "rbac.authorization.k8s.io"
Solution

  • First, you need to allow access to the URLs kubectl uses for API discovery and validation (swagger, listings of API groups and resource types, etc).

    The easiest way to do that is to load the default bootstrap cluster roles and cluster role bindings:

    kubectl create -f https://raw.githubusercontent.com/kubernetes/kubernetes/master/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/testdata/cluster-roles.yaml
kubectl create -f https://raw.githubusercontent.com/kubernetes/kubernetes/master/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/testdata/cluster-role-bindings.yaml

    That will create a system:discovery ClusterRole and bind all users (authenticated and unauthenticated) to it, allowing them to access swagger and API group information.

    Second, you shouldn't include the dev service account in the all cluster role binding. That would allow that service account (and anyone with access to secrets in the dev namespace containing the dev service account credentials) cluster wide access