Nginx Proxy-Bind UDP datagram not received (bad udp checksum) on upstream

Question

I'm setting up a UDP Load Balancer.

I'm using Nginx Plus r14 on Ubuntu 16.04.3 LTS. The hosting is done as a barebone server for the upstream, and a a VPS instance for the load balancer.

I wish I could setup a transparent proxy with a direct response back to the client (this is the Origin-NAT use case described here https://www.nginx.com/blog/ip-transparency-direct-server-return-nginx-plus-transparent-proxy/).

The Nginx configuration looks correct :

user  root;
worker_processes  auto;
worker_rlimit_nofile 65535;

error_log  /var/log/nginx/error.log debug;
pid        /var/run/nginx.pid;

events {
    worker_connections  20000;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

stream {
  upstream my_udp_upstreams {
    hash $remote_addr;
    server preprods.mycorp.com:5684;
  }

  server {
    listen 5684 udp;
    proxy_bind $remote_addr:$remote_port transparent;
    proxy_pass my_udp_upstreams;
    proxy_responses 0;
  }
}

A tcpdump executed on the Nginx load-balancer shows the following, which lookes nice :

09:29:55.112096 IP 54.38.X.Y.57064 > 217.182.A.B.5656: UDP, length 95 
(client -> LB)
09:29:55.112447 IP 54.38.X.Y.57064 > 149.202.C.D.5656: UDP, [bad udp cksum 0xb338 -> 0x8b86!] length 95 
(IP is spoofed, client->upstream)

Port 5656 is the destination port of the serviec on the upstream server, 54.38.X.Y is the client address, 217.182.A.B is the Load Balancer address and 149.202.C.D is the upstream server address.

My issue is that no data arrives on the upstream on the 5656 port. A tcpdump performed on the upstream server shows nothing :

sudo tcpdump -i eth0 -n port 5684

Looks like spoofed packets either do not get out of the LB or are not received by the upstream. I disabled checksum offloading with ethtool, but the issue remains :

sudo  ethtool --offload  ens3 rx off  tx off
sudo ethtool -K ens3 gso off

Am I doing something wrong or is there filtering taking place somewhere ?

Answer 1

The setup was correct, but the hosting company (OVH) doesn't allow for IP spoofing so that datagrams are dropped and never get through the upstreams.