I'm trying to serve an application using NestJS but I'm not being able to do so.
I've already configured Traefik IngressRoutes to serve both Traefik Dashboard and also ArgoCD (and a couple more test apps), but I've been trying to deploy this new application for almost 2 days, without success.
The error is the following:
Bad Gateway
And this is the log Traefik outputs upon a request:
[traefik-c88c9f869-b8cm8] 10.0.1.122 - - [11/Dec/2020:03:13:20 +0000] "GET /graphql HTTP/2.0" 502 11 "-" "-" 764 "develop-business-app-64fa6977f85a45bb4625@kubernetescrd" "http://10.0.3.86:8080" 1ms
I don't know if there is any custom configuration I need to do in my app to use HTTP/2.0 or handle Traefik SSL (since the entry point is websecure). I've followed the docs over and over but I always get the same error (I've already tried to remove and installed Traefik again entirely)
Also, if I run kubectl port-forward I can use the application as expected.
Here are my configuration files:
This is my Traefik deployment:
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: traefik
labels:
app.kubernetes.io/name: traefik-proxy
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/component: infrastructure
app.kubernetes.io/part-of: traefik
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: traefik-proxy
template:
metadata:
labels:
app.kubernetes.io/name: traefik-proxy
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/component: infrastructure
app.kubernetes.io/part-of: traefik
spec:
serviceAccountName: traefik-ingress-controller
volumes:
- name: acme-certificates
emptyDir: {}
containers:
- name: traefik
image: traefik:v2.3
args:
- --accesslog
- --providers.kubernetescrd
- --ping
- --api.dashboard
- --entrypoints.traefik.address=:8080
- --entrypoints.web.address=:80
- --entrypoints.websecure.address=:443
- --entrypoints.web.http.redirections.entrypoint.to=websecure
- --entrypoints.websecure.http.tls.certResolver=letsencrypt
- --certificatesresolvers.letsencrypt.acme.email=accounts+letsencrypt@getbud.co
- --certificatesresolvers.letsencrypt.acme.storage=/etc/acme/letsencrypt.json
- --certificatesResolvers.letsencrypt.acme.dnsChallenge.provider=route53
- --certificatesResolvers.letsencrypt.acme.dnsChallenge.delayBeforeCheck=0
volumeMounts:
- name: acme-certificates
mountPath: /etc/acme
ports:
- containerPort: 8080
name: admin
protocol: TCP
- containerPort: 80
name: web
protocol: TCP
- containerPort: 443
name: websecure
protocol: TCP
livenessProbe:
failureThreshold: 3
httpGet:
path: /ping
port: 8080
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
readinessProbe:
failureThreshold: 1
httpGet:
path: /ping
port: 8080
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
This is my application deployment:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: business-app
labels:
app.kubernetes.io/name: business-app
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/component: business
app.kubernetes.io/part-of: application-layer
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: business-app
template:
metadata:
labels:
app.kubernetes.io/name: business-app
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/component: business
app.kubernetes.io/part-of: application-layer
spec:
containers:
- name: business-app
image: 904333181156.dkr.ecr.sa-east-1.amazonaws.com/business:$ECR_TAG <- this is updated with the latest tag using envsubst
ports:
- containerPort: 8080
name: web
protocol: TCP
This is my application service:
---
kind: Service
apiVersion: v1
metadata:
name: business-app
spec:
selector:
app.kubernetes.io/name: business-app
ports:
- name: web
port: 80
targetPort: 8080
And this is my IngressRoute:
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: business-app
labels:
app.kubernetes.io/name: business-app
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/component: business
app.kubernetes.io/part-of: application-layer
spec:
entryPoints:
- websecure
routes:
- match: Host(`api.develop.getbud.co`)
kind: Rule
services:
- name: business-app
port: 80
tls:
certResolver: letsencrypt
options: {}
Can someone give me a hint on what am I doing wrong?
Just an update, I've changed the loglevel of traefik to debug, and here is what it logs upon request:
[traefik-55888dfd67-r8b2c] time="2020-12-11T04:54:31Z" level=debug msg="Error while Peeking first byte: read tcp 10.0.3.86:80->10.0.1.122:44996: read: connection reset by peer"
[traefik-55888dfd67-r8b2c] time="2020-12-11T04:54:31Z" level=debug msg="Error while Peeking first byte: read tcp 10.0.3.86:8080->10.0.3.100:6380: read: connection reset by peer"
[traefik-55888dfd67-r8b2c] time="2020-12-11T04:54:32Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/graphql\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,pt-BR;q=0.5\"],\"Cache-Control\":[\"no-cache\"],\"Pragma\":[\"no-cache\"],\"Te\":[\"trailers\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0\"],\"X-Forwarded-Host\":[\"api.develop.getbud.co\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-55888dfd67-r8b2c\"],\"X-Real-Ip\":[\"10.0.1.122\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"api.develop.getbud.co\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.0.1.122:27473\",\"RequestURI\":\"/graphql\",\"TLS\":null}"
[traefik-55888dfd67-r8b2c] time="2020-12-11T04:54:32Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/graphql\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,pt-BR;q=0.5\"],\"Cache-Control\":[\"no-cache\"],\"Pragma\":[\"no-cache\"],\"Te\":[\"trailers\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0\"],\"X-Forwarded-Host\":[\"api.develop.getbud.co\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-55888dfd67-r8b2c\"],\"X-Real-Ip\":[\"10.0.1.122\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"api.develop.getbud.co\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.0.1.122:27473\",\"RequestURI\":\"/graphql\",\"TLS\":null}" ForwardURL="http://10.0.1.158:8080"
[traefik-55888dfd67-r8b2c] time="2020-12-11T04:54:32Z" level=debug msg="'502 Bad Gateway' caused by: dial tcp 10.0.1.158:8080: connect: connection refused"
[traefik-55888dfd67-r8b2c] time="2020-12-11T04:54:32Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/graphql\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,pt-BR;q=0.5\"],\"Cache-Control\":[\"no-cache\"],\"Pragma\":[\"no-cache\"],\"Te\":[\"trailers\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0\"],\"X-Forwarded-Host\":[\"api.develop.getbud.co\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-55888dfd67-r8b2c\"],\"X-Real-Ip\":[\"10.0.1.122\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"api.develop.getbud.co\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.0.1.122:27473\",\"RequestURI\":\"/graphql\",\"TLS\":null}"
[traefik-55888dfd67-r8b2c] 10.0.1.122 - - [11/Dec/2020:04:54:32 +0000] "GET /graphql HTTP/2.0" 502 11 "-" "-" 754 "develop-business-app-64fa6977f85a45bb4625@kubernetescrd" "http://10.0.1.158:8080" 2ms
[traefik-55888dfd67-r8b2c] time="2020-12-11T04:54:32Z" level=debug msg="Error while Peeking first byte: read tcp 10.0.3.86:443->10.0.3.75:35314: read: connection reset by peer"
[traefik-55888dfd67-r8b2c] time="2020-12-11T04:54:32Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/favicon.ico\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"image/webp,*/*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,pt-BR;q=0.5\"],\"Cache-Control\":[\"no-cache\"],\"Pragma\":[\"no-cache\"],\"Referer\":[\"https://api.develop.getbud.co/graphql\"],\"Te\":[\"trailers\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0\"],\"X-Forwarded-Host\":[\"api.develop.getbud.co\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-55888dfd67-r8b2c\"],\"X-Real-Ip\":[\"10.0.1.122\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"api.develop.getbud.co\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.0.1.122:27473\",\"RequestURI\":\"/favicon.ico\",\"TLS\":null}"
[traefik-55888dfd67-r8b2c] time="2020-12-11T04:54:32Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" ForwardURL="http://10.0.1.158:8080" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/favicon.ico\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"image/webp,*/*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,pt-BR;q=0.5\"],\"Cache-Control\":[\"no-cache\"],\"Pragma\":[\"no-cache\"],\"Referer\":[\"https://api.develop.getbud.co/graphql\"],\"Te\":[\"trailers\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0\"],\"X-Forwarded-Host\":[\"api.develop.getbud.co\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-55888dfd67-r8b2c\"],\"X-Real-Ip\":[\"10.0.1.122\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"api.develop.getbud.co\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.0.1.122:27473\",\"RequestURI\":\"/favicon.ico\",\"TLS\":null}"
[traefik-55888dfd67-r8b2c] time="2020-12-11T04:54:32Z" level=debug msg="'502 Bad Gateway' caused by: dial tcp 10.0.1.158:8080: connect: connection refused"
[traefik-55888dfd67-r8b2c] time="2020-12-11T04:54:32Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/favicon.ico\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"image/webp,*/*\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,pt-BR;q=0.5\"],\"Cache-Control\":[\"no-cache\"],\"Pragma\":[\"no-cache\"],\"Referer\":[\"https://api.develop.getbud.co/graphql\"],\"Te\":[\"trailers\"],\"User-Agent\":[\"Mozilla/5.0 (X11; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0\"],\"X-Forwarded-Host\":[\"api.develop.getbud.co\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"traefik-55888dfd67-r8b2c\"],\"X-Real-Ip\":[\"10.0.1.122\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"api.develop.getbud.co\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.0.1.122:27473\",\"RequestURI\":\"/favicon.ico\",\"TLS\":null}"
[traefik-55888dfd67-r8b2c] 10.0.1.122 - - [11/Dec/2020:04:54:32 +0000] "GET /favicon.ico HTTP/2.0" 502 11 "-" "-" 755 "develop-business-app-64fa6977f85a45bb4625@kubernetescrd" "http://10.0.1.158:8080" 1ms
So, it seems Traefik is receiving a connection refused from the pod. I've opened a shell inside Traefik's container and tried to run wget directly in the Pod IP and indeed I received the same error (connection refused).
Any other working pod whenever I ran wget it works.
Any ideas?
For those who (like me) are struggling with this issue, here is the problems:
Fastify by default listens only at 127.0.0.1, so, it automatically refuses any other host connection. To solve that you can simply add 0.0.0.0 as the second argument of your app.listen call, like the following:
previous:
await app.listen(appConfig.port)
fixed:
await app.listen(appConfig.port, '0.0.0.0')
Thanks in any case :)