Trying out VMWare's Harbour registry server for first time and trying it as a deployment on new Kubernetes cluster.
Having followed the Harbor on Kubernetes guide, all Harbor resources get applied on the k8s cluster and can be seen running okay. However, I am currently unable to access the Harbor ui from a web browser (I just get "Unable to connect" back). It is my guess that security was not setup properly and something is missing or in the wrong place?
The make/harbor.cfg file is configured with:
hostname = k8s-dp-2 # This is the worker node on which Harbor is running..
ui_url_protocol = https
ssl_cert = /path/to/cert/on/host/harbor.crt
ssl_cert_key = /path/to/cert/on/host/harbor.key
secretkey_path = /data
I am assuming that the path to the certs above are the path on the host from which the python script will grab the files to then do the YAML builds?
---- UPDATE ---
After advice given in comments, I have now configured an nginx ingress controller in the k8s cluster. After adding in this ingress controller, I have updated the Harbor config to use http and no longer https since the https part should now be taken care of by the nginx ingress controller. With these config changes now in place however, I am still unable to get to the Harbor service via https but I am now able to get to the Harbor service by calling it via the kubernetes cluster's http port. See tests below
# kubectl get svc -n=nginx-ingress
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-ingress NodePort 10.103.165.23 <none> 80:31819/TCP,443:30435/TCP 20h
Test Call 1:
$ curl https://k8s-dp-2/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to k8s-dp-2 port 443: Connection refused
Test Call 2:
$ curl https://k8s-dp-2:30435/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
Test Call 3:
$ curl http://k8s-dp-2/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to k8s-dp-2 port 80: Connection refused
Test Call 4:
$ curl http://k8s-dp-2:31819/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 810 100 810 0 0 12857 0 --:--:-- --:--:-- --:--:-- 12857<!doctype html>
<html>
<head>
<meta charset="utf-8">
<title>Harbor</title>
<base href="/">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="icon" type="image/x-icon" href="favicon.ico?v=2">
</head>
<body style="overflow-y: hidden;">
...
After trying various different configurations, the YAML configurations posted below are what worked for me:
Ingress Conroller YAML:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-ingress-controller
namespace: ingress-nginx
spec:
replicas: 1
selector:
matchLabels:
app: ingress-nginx
template:
metadata:
labels:
app: ingress-nginx
annotations:
prometheus.io/port: '10254'
prometheus.io/scrape: 'true'
spec:
serviceAccountName: nginx-ingress-serviceaccount
containers:
- name: nginx-ingress-controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.11.0
args:
- /nginx-ingress-controller
- --default-backend-service=$(POD_NAMESPACE)/default-http-backend
- --default-ssl-certificate=$(POD_NAMESPACE)/default-tls-secret
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --annotations-prefix=nginx.ingress.kubernetes.io
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
Ingress YAML:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: harbor
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
tls:
- hosts:
- k8s-dp-2
rules:
- host: k8s-dp-2
http:
paths:
- path: /
backend:
serviceName: ui
servicePort: 80
- path: /v2
backend:
serviceName: registry
servicePort: repo
- path: /service
backend:
serviceName: ui
servicePort: 80
Service YAML:
apiVersion: v1
kind: Service
metadata:
name: ui
spec:
ports:
- port: 80
selector:
name: ui-apps
Getting to a working solution was not straightforward however. Had to learn a lot about ingress controllers, ingresses, etc. Also I was initially mixing configurations from two different nginx ingress controller images that work differently (The configs below work with quay.io's nginx ingress controller). Also, for a reason which I still don't properly understand, the final configuration only started working after a full reboot of the k8s nodes involved.