SSL works from a VPN client but fails over the GCP-PaloAlto VPN tunnel.
The server is an on-prem Keycloak (version 13.0.0)
Attempt from VPN Client (Successful)
# openssl s_client -connect fqdn:443 -servername fqdn -tls1_2 --prexit
CONNECTED(00000005)
[...]
depth=0 CN = <fqdn>
verify return:1
---
Certificate chain
0 s:CN = <fqdn>
[...]
---
Server certificate
[...]
subject=CN = <fqdn>
issuer=C = US, O = Let's Encrypt, CN = R3
---
[...]
---
SSL handshake has read 4672 bytes and written 311 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
[...]
---
closed
---
Certificate chain
0 s:CN = <fqdn>
[...]
---
Server certificate
[...]
subject=CN = <fqdn>
issuer=C = US, O = Let's Encrypt, CN = R3
[...]
tcpdump from the working client
19:32:09.582389 IP CLIENT.51013 > SERVER.https: Flags [S], seq 3647907685, win 65535, options [mss 1360,nop,wscale 6,nop,nop,TS val 350978343 ecr 0,sackOK,eol], length 0
19:32:09.582551 IP SERVER.https > CLIENT.51013: Flags [S.], seq 3495009322, ack 3647907686, win 65160, options [mss 1460,sackOK,TS val 2099619205 ecr 350978343,nop,wscale 7], length 0
19:32:09.596385 IP CLIENT.51013 > SERVER.https: Flags [.], ack 1, win 2064, options [nop,nop,TS val 350978356 ecr 2099619205], length 0
19:32:09.596385 IP CLIENT.51013 > SERVER.https: Flags [P.], seq 1:219, ack 1, win 2064, options [nop,nop,TS val 350978356 ecr 2099619205], length 218
19:32:09.596502 IP SERVER.https > CLIENT.51013: Flags [.], ack 219, win 508, options [nop,nop,TS val 2099619219 ecr 350978356], length 0
19:32:09.596976 IP SERVER.https > CLIENT.51013: Flags [P.], seq 1:2697, ack 219, win 508, options [nop,nop,TS val 2099619219 ecr 350978356], length 2696
19:32:09.597009 IP SERVER.https > CLIENT.51013: Flags [P.], seq 2697:4097, ack 219, win 508, options [nop,nop,TS val 2099619219 ecr 350978356], length 1400
19:32:09.599161 IP SERVER.https > CLIENT.51013: Flags [P.], seq 4097:4415, ack 219, win 508, options [nop,nop,TS val 2099619221 ecr 350978356], length 318
19:32:09.612262 IP CLIENT.51013 > SERVER.https: Flags [.], ack 1349, win 2043, options [nop,nop,TS val 350978368 ecr 2099619219], length 0
19:32:09.612262 IP CLIENT.51013 > SERVER.https: Flags [.], ack 2697, win 2022, options [nop,nop,TS val 350978368 ecr 2099619219], length 0
19:32:09.612262 IP CLIENT.51013 > SERVER.https: Flags [.], ack 4045, win 2000, options [nop,nop,TS val 350978368 ecr 2099619219], length 0
19:32:09.612262 IP CLIENT.51013 > SERVER.https: Flags [.], ack 4097, win 2000, options [nop,nop,TS val 350978368 ecr 2099619219], length 0
19:32:09.612262 IP CLIENT.51013 > SERVER.https: Flags [.], ack 4097, win 2048, options [nop,nop,TS val 350978368 ecr 2099619219], length 0
19:32:09.614194 IP CLIENT.51013 > SERVER.https: Flags [.], ack 4415, win 2043, options [nop,nop,TS val 350978369 ecr 2099619221], length 0
19:32:09.614194 IP CLIENT.51013 > SERVER.https: Flags [P.], seq 219:312, ack 4415, win 2048, options [nop,nop,TS val 350978370 ecr 2099619221], length 93
19:32:09.614248 IP SERVER.https > CLIENT.51013: Flags [.], ack 312, win 508, options [nop,nop,TS val 2099619236 ecr 350978370], length 0
19:32:09.614711 IP SERVER.https > CLIENT.51013: Flags [P.], seq 4415:4673, ack 312, win 508, options [nop,nop,TS val 2099619237 ecr 350978370], length 258
19:32:09.626178 IP CLIENT.51013 > SERVER.https: Flags [.], ack 4673, win 2043, options [nop,nop,TS val 350978384 ecr 2099619237], length 0
19:33:09.651599 IP SERVER.https > CLIENT.51013: Flags [P.], seq 4673:4704, ack 312, win 508, options [nop,nop,TS val 2099679274 ecr 350978384], length 31
19:33:09.651690 IP SERVER.https > CLIENT.51013: Flags [F.], seq 4704, ack 312, win 508, options [nop,nop,TS val 2099679274 ecr 350978384], length 0
19:33:09.678658 IP CLIENT.51013 > SERVER.https: Flags [.], ack 4704, win 2047, options [nop,nop,TS val 351038150 ecr 2099679274], length 0
19:33:09.678806 IP CLIENT.51013 > SERVER.https: Flags [.], ack 4705, win 2047, options [nop,nop,TS val 351038150 ecr 2099679274], length 0
19:33:09.680749 IP CLIENT.51013 > SERVER.https: Flags [P.], seq 312:343, ack 4705, win 2048, options [nop,nop,TS val 351038150 ecr 2099679274], length 31
19:33:09.680749 IP CLIENT.51013 > SERVER.https: Flags [F.], seq 343, ack 4705, win 2048, options [nop,nop,TS val 351038150 ecr 2099679274], length 0
19:33:09.680824 IP SERVER.https > CLIENT.51013: Flags [R], seq 3495014027, win 0, length 0
19:33:09.680888 IP SERVER.https > CLIENT.51013: Flags [R], seq 3495014027, win 0, length 0
Attempt from GCP Instance (Unsuccessful)
# openssl s_client -connect fqdn:443 -servername fqdn -tls1_2 --prexit
CONNECTED(00000003)
write:errno=0
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 212 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
[...]
SSL-Session:
Protocol : TLSv1.2
[...]
---
tcpdump from the client that doesn't work
19:36:01.818130 IP CLIENT.55282 > SERVER.https: Flags [S], seq 3880590205, win 65320, options [mss 1350,sackOK,TS val 3904950778 ecr 0,nop,wscale 7], length 0
19:36:01.818241 IP SERVER.https > CLIENT.55282: Flags [S.], seq 1198318204, ack 3880590206, win 65160, options [mss 1460,sackOK,TS val 2676835275 ecr 3904950778,nop,wscale 7], length 0
19:36:01.829890 IP CLIENT.55282 > SERVER.https: Flags [.], ack 1, win 511, options [nop,nop,TS val 3904950791 ecr 2676835275], length 0
19:37:01.890295 IP SERVER.https > CLIENT.55282: Flags [F.], seq 1, ack 1, win 510, options [nop,nop,TS val 2676895347 ecr 3904950791], length 0
19:37:02.105125 IP SERVER.https > CLIENT.55282: Flags [F.], seq 1, ack 1, win 510, options [nop,nop,TS val 2676895562 ecr 3904950791], length 0
19:37:02.321181 IP SERVER.https > CLIENT.55282: Flags [F.], seq 1, ack 1, win 510, options [nop,nop,TS val 2676895778 ecr 3904950791], length 0
19:37:02.753156 IP SERVER.https > CLIENT.55282: Flags [F.], seq 1, ack 1, win 510, options [nop,nop,TS val 2676896210 ecr 3904950791], length 0
19:37:03.617211 IP SERVER.https > CLIENT.55282: Flags [F.], seq 1, ack 1, win 510, options [nop,nop,TS val 2676897074 ecr 3904950791], length 0
19:37:05.345180 IP SERVER.https > CLIENT.55282: Flags [F.], seq 1, ack 1, win 510, options [nop,nop,TS val 2676898802 ecr 3904950791], length 0
19:37:08.769186 IP SERVER.https > CLIENT.55282: Flags [F.], seq 1, ack 1, win 510, options [nop,nop,TS val 2676902226 ecr 3904950791], length 0
19:37:15.681150 IP SERVER.https > CLIENT.55282: Flags [F.], seq 1, ack 1, win 510, options [nop,nop,TS val 2676909138 ecr 3904950791], length 0
Not sure if it matters, but a difference I have observed is the MTU of the networks.
VPN Client can ping the server with up to size 1372
GCP node can ping the server with up to size 1362
Turns out, it was a case of asynchronous routing. Two on-prem sites were advertising the same route but only one of them had the network.
In summary:
SYN: GCP -> SITE A
ACK, SYN: SITE A -> GCP
ACK: GCP -(through SITE B)-> SITE A
FIN: SITE A doesn't like that the ACK came though a different tunnel then the one used to send the SYN.
Terminates the connection and sends a FIN.