nginx proxy_pass with rewrite to Jenkins

Question

I have Jenkins in private network and I use Bitbucket in Cloud. So I want to have proxy for Bitbucket Webhooks. So more or less connection flow looks like this:

Bitbucket Cloud -> AWS NLB (SSL): gate.example.com -> nginx -> AWS ALB (SSL): jenkins.internal.example.com -> Jenkins

And I have following config in nginx:

resolver 127.0.0.53 valid=5s;

upstream jen {
  server jenkins.internal.example.com:443;
  keepalive 128;
}

server {
  listen 80;
  server_name gate.example.com;

  access_log /var/log/nginx/jenkins.access.log;
  error_log /var/log/nginx/jenkins.error.log debug;

  server_tokens off;


  rewrite_log on;

  ignore_invalid_headers off;

  location /jenkins/ {
    rewrite ^/jenkins/(.*) /$1 break;
    proxy_http_version 1.1;
    proxy_set_header   Host              $host;
    proxy_set_header   X-Real-IP         $remote_addr;
    proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header   Connection "";
    proxy_set_header   X-Forwarded-Proto https;
    proxy_pass https://jen;
  }
}

And it doesn't work. Tests I can replicate:

• if I curl from my localhost (inside private network) to https://gate.example.com/jenkins/bitbucket-scmsource-hook/notify I will get 404 and no log in Jenkins. I will get also some huge debug log (because I run ginx-debug for it) in nginx (I'll paste part of it at the end)
• if I curl from my localhost (outside private network) to https://gate.example.com/jenkins/bitbucket-scmsource-hook/notify I will get 404 and no log in Jenkins.
• if I curl from my localhost (inside private network) to https://jenkins.internal.example.com/bitbucket-scmsource-hook/notify I will get 200 and some logs in Jenkins
• if I curl from gate machine (also inside private network) to https://jenkins.internal.example.com/bitbucket-scmsource-hook/notify I will get 200 and some logs in Jenkins

It looks to me that I have some problem in nginx configuration but I cannot spot it.

nginx log:

2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http header: "Host: gate.example.com"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http header: "User-Agent: curl/7.77.0"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http header: "Accept: */*"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http header: "X-Request-UUID: 8c05c48d-5db0-4e73-8c98-xxxxxxx"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http header: "X-Hook-UUID: b3551c3f-43fb-4bd3-b0a4-xxxxxxx"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http header: "X-Event-Key: repo:push"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http header: "Content-Type: application/json; charset=UTF-8"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http header: "Content-Length: 14052"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http header done
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 event timer del: 9: 2945992864
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 generic phase: 0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 rewrite phase: 1
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 test location: "/jenkins/"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 using configuration "/jenkins/"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http cl:14052 max:1048576
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 rewrite phase: 3
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http script regex: "^/jenkins/(.*)"
2022/03/24 16:39:57 [notice] 2178625#2178625: *3061 "^/jenkins/(.*)" matches "/jenkins/bitbucket-scmsource-hook/notify", client: MY.IP.MY.IP, server: gate.example.com, request: "POST /jenkins/bitbucket-scmsource-hook/notify HTTP/1.1", host: "gate.example.com"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http script copy: "/"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http script capture: "bitbucket-scmsource-hook/notify"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http script regex end
2022/03/24 16:39:57 [notice] 2178625#2178625: *3061 rewritten data: "/bitbucket-scmsource-hook/notify", args: "", client: MY.IP.MY.IP, server: gate.example.com, request: "POST /jenkins/bitbucket-scmsource-hook/notify HTTP/1.1", host: "gate.example.com"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 post rewrite phase: 4
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 generic phase: 5
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 generic phase: 6
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 generic phase: 7
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 access phase: 8
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 access phase: 9
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 access phase: 10
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 post access phase: 11
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 generic phase: 12
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 generic phase: 13
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http client request body preread 692
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http request body content length filter
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http body new buf t:1 f:0 000055689A5B32FC, pos 000055689A5B32FC, size: 692 file: 0, size: 0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 malloc: 000055689A5D2E40:8192
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http read client request body
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 recv: eof:0, avail:9216
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 recv: fd:9 8192 of 8192
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 recv: avail:1024
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http client request body recv 8192
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http body new buf t:1 f:0 000055689A5D2E40, pos 000055689A5D2E40, size: 8192 file: 0, size: 0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http write client request body, bufs 000055689A68A3B8
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 add cleanup: 000055689A68A5D8
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 hashed path: /var/cache/nginx/client_temp/0000000002
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 temp fd:10
2022/03/24 16:39:57 [warn] 2178625#2178625: *3061 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000002, client: 89.64.38.110, server: gate.mkyc.test.symmetrical.ai, request: "POST /jenkins/bitbucket-scmsource-hook/notify HTTP/1.1", host: "gate.mkyc.test.symmetrical.ai"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 writev: 10, 8884, 0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http write client request body, bufs 0000000000000000
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 recv: eof:0, avail:1024
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 recv: fd:9 5168 of 5168
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 recv: avail:0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http client request body recv 5168
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http body new buf t:1 f:0 000055689A5D2E40, pos 000055689A5D2E40, size: 5168 file: 0, size: 0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http write client request body, bufs 000055689A68A488
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 write: 10, 000055689A5D2E40, 5168, 8884
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http client request body rest 0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http init upstream, client timer: 0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 epoll add event: fd:9 op:3 ev:80002005
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http script copy: "Host"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http script var: "gate.mkyc.test.symmetrical.ai"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http script copy: "X-Real-IP"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http script var: "MY.IP.MY.IP"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http script copy: "X-Forwarded-For"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http script var: "MY.IP.MY.IP"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http script copy: "X-Forwarded-Proto"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http script copy: "https"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http script copy: "Content-Length"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http script var: "14052"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http script copy: ""
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http proxy header: "User-Agent: curl/7.77.0"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http proxy header: "Accept: */*"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http proxy header: "X-Request-UUID: 8c05c48d-5db0-4e73-8c98-xxxxxxx"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http proxy header: "X-Hook-UUID: b3211c3f-43fb-4bd3-b0f4-xxxxxxx"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http proxy header: "X-Event-Key: repo:push"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http proxy header: "Content-Type: application/json; charset=UTF-8"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http proxy header:
"POST /bitbucket-scmsource-hook/notify HTTP/1.1
Host: gate.example.com
X-Real-IP: MY.IP.MY.IP
X-Forwarded-For: MY.IP.MY.IP
X-Forwarded-Proto: https
Content-Length: 14052
User-Agent: curl/7.77.0
Accept: */*
X-Request-UUID: 8c05c48d-5db0-4e73-8c98-xxxxxxx
X-Hook-UUID: b3211c3f-43fb-4bd3-b0f4-xxxxxxx
X-Event-Key: repo:push
Content-Type: application/json; charset=UTF-8

"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http cleanup add: 000055689A68A8C8
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 init keepalive peer
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 get keepalive peer
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 get rr peer, try: 3
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 get rr peer, current: 000055689A5F9FF0 -1
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 stream socket 15
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 epoll add connection: fd:15 ev:80002005
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 connect to 10.10.3.187:443, fd:15 #3062
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http upstream connect: -2
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 posix_memalign: 000055689A5DB270:128 @16
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 event timer add: 15: 60000:2945992940
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http finalize request: -4, "/bitbucket-scmsource-hook/notify?" a:1, c:2
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http request count:2 blk:0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http run request: "/bitbucket-scmsource-hook/notify?"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http upstream check client, write event:1, "/bitbucket-scmsource-hook/notify"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http upstream request: "/bitbucket-scmsource-hook/notify?"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http upstream send request handler
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 malloc: 000055689A5CEAA0:96
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 set session: 0000000000000000
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 tcp_nodelay
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 SSL_do_handshake: -1
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 SSL_get_error: 2
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 SSL handshake handler: 0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 SSL_do_handshake: -1
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 SSL_get_error: 2
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 SSL handshake handler: 1
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 SSL_do_handshake: -1
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 SSL_get_error: 2
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 SSL handshake handler: 0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 save session: 000055689A686D60
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 SSL_do_handshake: 1
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 SSL: TLSv1.2, cipher: "ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http upstream ssl handshake: "/bitbucket-scmsource-hook/notify?"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http upstream send request
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http upstream send request body
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 read: 10, 000055689A5D2E40, 8192, 0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 chain writer buf fl:0 s:406
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 chain writer buf fl:0 s:8192
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 chain writer in: 000055689A68A970
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 malloc: 000055689A68EFA0:80
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 malloc: 000055689A67AA30:16384
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 SSL buf copy: 406
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 SSL buf copy: 8192
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 chain writer out: 0000000000000000
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 read: 10, 000055689A5D2E40, 5860, 8192
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 chain writer buf fl:1 s:5860
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 chain writer in: 000055689A68A950
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 SSL buf copy: 5860
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 SSL to write: 14458
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 SSL_write: 14458
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 chain writer out: 0000000000000000
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 event timer del: 15: 2945992940
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 event timer add: 15: 60000:2945992952
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http upstream process header
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 malloc: 000055689A68A9E0:4096
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 posix_memalign: 000055689A5D4E50:4096 @16
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 SSL_read: -1
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 SSL_get_error: 2
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http upstream request: "/bitbucket-scmsource-hook/notify?"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http upstream dummy handler
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http upstream request: "/bitbucket-scmsource-hook/notify?"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http upstream process header
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 SSL_read: 167
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 SSL_read: -1
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 SSL_get_error: 2
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http proxy status 404 "404 Not Found"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http proxy header: "Server: awselb/2.0"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http proxy header: "Date: Thu, 24 Mar 2022 16:39:57 GMT"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http proxy header: "Content-Type: text/plain; charset=utf-8"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http proxy header: "Content-Length: 0"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http proxy header: "Connection: keep-alive"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http proxy header done
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 HTTP/1.1 404 Not Found
Server: nginx
Date: Thu, 24 Mar 2022 16:39:57 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 0
Connection: keep-alive

2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 write new buf t:1 f:0 000055689A5D5140, pos 000055689A5D5140, size: 162 file: 0, size: 0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http write filter: l:0 f:0 s:162
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 file cleanup: fd:10
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http cacheable: 0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http proxy filter init s:404 h:0 c:0 l:0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http upstream process upstream
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 pipe read upstream: 0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 pipe preread: 0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 pipe buf free s:0 t:1 f:0 000055689A68A9E0, pos 000055689A68AA87, size: 0 file: 0, size: 0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 pipe length: 0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 pipe write downstream: 1
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 pipe write downstream done
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 event timer: 15, old: 2945992952, new: 2945992952
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http upstream exit: 0000000000000000
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 finalize http upstream request: 0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 finalize http proxy request
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 free keepalive peer
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 free keepalive peer: saving connection 000055689A60EBE0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 event timer: 15, old: 2945992952, new: 2945992952
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 free rr peer 3 0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http upstream temp fd: -1
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http output filter "/bitbucket-scmsource-hook/notify?"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http copy filter: "/bitbucket-scmsource-hook/notify?"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http postpone filter "/bitbucket-scmsource-hook/notify?" 00007FFFBB415E10
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 write old buf t:1 f:0 000055689A5D5140, pos 000055689A5D5140, size: 162 file: 0, size: 0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 write new buf t:0 f:0 0000000000000000, pos 0000000000000000, size: 0 file: 0, size: 0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http write filter: l:1 f:0 s:162
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http write filter limit 0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 writev: 162 of 162
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http write filter 0000000000000000
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http copy filter: 0 "/bitbucket-scmsource-hook/notify?"
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http finalize request: 0, "/bitbucket-scmsource-hook/notify?" a:1, c:1
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 set http keepalive handler
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http close request
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http log handler
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 free: 000055689A68A9E0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 free: 000055689A5D2E40
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 free: 000055689A5DCB10, unused: 8
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 free: 000055689A6899D0, unused: 11
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 free: 000055689A5D4E50, unused: 2501
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 free: 000055689A5B31B0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 hc free: 0000000000000000
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 hc busy: 0000000000000000 0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 tcp_nodelay
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 reusable connection: 1
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 event timer add: 9: 65000:2945997952
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 http keepalive handler
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 malloc: 000055689A5B31B0:1024
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 recv: eof:1, avail:-1
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 recv: fd:9 0 of 1024
2022/03/24 16:39:57 [info] 2178625#2178625: *3061 client 89.64.38.110 closed keepalive connection
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 close http connection: 9
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 event timer del: 9: 2945997952
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 reusable connection: 0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 free: 000055689A5B31B0
2022/03/24 16:39:57 [debug] 2178625#2178625: *3061 free: 000055689A5CD0D0, unused: 136

Update 1

I also tested it in following ways:

• if I curl from gate machine: curl -vvv -X POST -H "X-Request-UUID: 8c05c48d-5db0-4e73-8c98-xxxxxx" -H "X-Hook-UUID: b3211c3f-43fb-4bd3-b0f4-xxxxxxxx" -H "X-Event-Key: repo:push" -H "Content-Type: application/json; charset=UTF-8" https://jenkins.internal.example.com/bitbucket-scmsource-hook/notify -d @push.json I will get 200
• if I curl from gate machine: curl -vvv -X POST -H "Host: gate.example.com" -H "X-Request-UUID: 8c05c48d-5db0-4e73-8c98-xxxxxxx" -H "X-Hook-UUID: b3211c3f-43fb-4bd3-b0f4-xxxxxxxx" -H "X-Event-Key: repo:push" -H "Content-Type: application/json; charset=UTF-8" http://127.0.0.1/jenkins/bitbucket-scmsource-hook/notify -d @push.json I will get 404

That would narrow down problems directly to nginx if I'm not mistaken.

Answer 1

Reason of a problem was proxy_set_header Host $host;. Due to it request performed by nginx to internal ALB was having this header with original value. Because internal ALB doesn't have default backend but is balancing for multiple services it wasn't able to route request correctly to correct service.