Search code examples
asp.netnginxswagger-uikestrel-http-serverbad-gateway

502 bad gateway nginx for dotnet WebApplication

I am trying to run a test WebApplication on a Digital Ocean droplet running Centos9 Stream and Nginx The WebApplication is the default project created from the visual studio template. It uses net8.0. Microsoft.AspNetCore.OpenApi 7.0.14 and Swashbuckle.AspNetCore 6.5.0

I also installed a certificate via LetsEncrypt

After starting nginx on the droplet I also start the application using

DotNet WebApplication.dll

and see that it is listening on port 5000

listening on port 5000

However if I try

curl -I https://example.com

I get

   HTTP/1.1 502 Bad Gateway
   Server: nginx/1.22.1

And I see the WebApplication console outputs a warning

Microsoft.AspNetCore.HttpsPolicy.HttpsRedirectionMiddleware[3]
Failed to determine the https port for redirect

I have the following in /etc/nginx/nginx.conf

# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log notice;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    keepalive_timeout   65;
    types_hash_max_size 4096;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    server {
        server_name example.com;
        location / {
            proxy_pass         http://localhost:5000; # Your .NET app's URL
            proxy_http_version 1.1;
            proxy_set_header   Upgrade $http_upgrade;
            proxy_set_header   Connection keep-alive;
            proxy_set_header   Host $host;
            proxy_cache_bypass $http_upgrade;
            proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header   X-Forwarded-Proto $scheme;
        }


        root         /usr/share/nginx/html;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        error_page 404 /404.html;
        location = /404.html {
        }

        error_page 500 502 503 504 /50x.html;
        location = /50x.html {
        }
    
    listen [::]:443 ssl ipv6only=on; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/examplecom/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

# Settings for a TLS enabled server.
#
#    server {
#        listen       443 ssl http2;
#        listen       [::]:443 ssl http2;
#        server_name  _;
#        root         /usr/share/nginx/html;
#
#        ssl_certificate "/etc/pki/nginx/server.crt";
#        ssl_certificate_key "/etc/pki/nginx/private/server.key";
#        ssl_session_cache shared:SSL:1m;
#        ssl_session_timeout  10m;
#        ssl_ciphers PROFILE=SYSTEM;
#        ssl_prefer_server_ciphers on;
#
#        # Load configuration files for the default server block.
#        include /etc/nginx/default.d/*.conf;
#
#        error_page 404 /404.html;
#        location = /404.html {
#        }
#
#        error_page 500 502 503 504 /50x.html;
#        location = /50x.html {
#        }
#    }



    server {
    if ($host = example.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


        listen       80;
        listen       [::]:80;
        server_name example.com;
    return 404; # managed by Certbot


}}

[Update]

cat /var/log/nginx/error.log

reports

Permission denied while connecting to upstream

[Update]

After looking at this ticket I tried

setsebool -P httpd_can_network_connect 1

And restarted.

Now when I run

dotnet WebApplication.dll

I get a warning

Microsoft.AspNetCore.HttpsPolicy.HttpsRedirectionMiddleware[3]
failed to deterimine the https port for redirect.
Microsoft.Hosting.Lifetime[0]
Application is shutting down.

And I still get the 502 Bad Gateway error on

curl -I https://example.com

I no longer see the Permission Denied error in the log. It is ( with IPAddress obfuscated )

2024/01/06 04:39:52 [notice] 1323#1323: signal 3 (SIGQUIT) received from 1, shutting down
2024/01/06 04:39:52 [notice] 1323#1323: signal 17 (SIGCHLD) received from 1324
2024/01/06 04:39:52 [notice] 1323#1323: worker process 1324 exited with code 0
2024/01/06 04:39:52 [notice] 1323#1323: exit
2024/01/06 04:43:25 [notice] 1339#1339: using the "epoll" event method
2024/01/06 04:43:25 [notice] 1339#1339: nginx/1.22.1
2024/01/06 04:43:25 [notice] 1339#1339: built by gcc 11.3.1 20221121 (Red Hat 11.3.1-4) (GCC) 
2024/01/06 04:43:25 [notice] 1339#1339: OS: Linux 5.14.0-391.el9.x86_64
2024/01/06 04:43:25 [notice] 1339#1339: getrlimit(RLIMIT_NOFILE): 1024:524288
2024/01/06 04:43:25 [notice] 1340#1340: start worker processes
2024/01/06 04:43:25 [notice] 1340#1340: start worker process 1341
2024/01/06 05:00:14 [error] 1341#1341: *5 connect() failed (111: Connection refused) while connecting to upstream, client: MyIPAddress, server: example.com, request: "HEAD / HTTP/1.1", upstream: "http://127.0.0.1:5000/", host: "example.com"
2024/01/06 05:00:14 [warn] 1341#1341: *5 upstream server temporarily disabled while connecting to upstream, client: MyIPAddress, server: example.com, request: "HEAD / HTTP/1.1", upstream: "http://127.0.0.1:5000/", host: "example.com"
2024/01/06 05:00:14 [error] 1341#1341: *5 connect() failed (111: Connection refused) while connecting to upstream, client: MyIPAddress, server: example.com, request: "HEAD / HTTP/1.1", upstream: "http://[::1]:5000/", host: "example.com"
2024/01/06 05:00:14 [warn] 1341#1341: *5 upstream server temporarily disabled while connecting to upstream, client: MyIPAddress, server: example.com, request: "HEAD / HTTP/1.1", upstream: "http://[::1]:5000/", host: "example.com"

[Update]

I see that https is not mentioned in the published appsettings.json (whereas it is in the development launchsettings.json)

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*"
}

[Update] From here I learned that

If no ports are specified, Kestrel binds to http://localhost:5000.

Next step adding

  "Kestrel": {
    "Endpoints": {
      "Https": {
        "Url": "https://localhost:5000"
      }
    }
  },

Now I get

The ASP.NET developer certificate is not trusted. See this

Also the Bad Gateway is still not solved.

[Update]

I have solved the certificate issue by making both nginx.conf and appsettings.json use http://localhost:5000

I still get the 502 Bad Gateway

https://example.com/swagger/index.html gives

nginx error. The page you are looking for is temporarily unavailable.

[Update]

I enabled IPV6 on the droplet (after noticing it was On in nginx.conf)

Now

curl -I https://example.com

reports 404 Not Found

Also the WebApplication console warns

Microsoft.AspNetCore.HttpsPolicy.HttpsRedirectionMiddleware3 Failed to determine the https port for redirect.

I can see using

cat /var/log/nginx/error.log

that

SSL_do_handshake() failed (SSL:error0A00006C:SSL routines::bad key share) while SSL Handshaking

[Update]

After reading ozkanpakdil's comment, I changed program.cs to remove app.UseHttpsRedirection();

Now

curl -I example.com

gives

301 Moved Permanently

[Update]

I am wondering about the disabled words in the status nginx output.

nginx status

If I end the WebApplication and go to example.com in the browser I get an error page

error page

If I start the WebApplication and go to example.com I see

Failed to load resource: the server responded with a status of 404.

On my development machine in Visual Studio the app opens https://localhost:7223/swagger/index.html

[Update]

To get proper resolution on example.com I needed to change the demo code to use the swagger ui

UseSwaggerUI

Solution

    • Somewhere at AspNetCore side code is trying to redirect to user to https because nginx sending the request via http, I suggest to disable AspNetCore https redirection because you already have nginx at front and it is doing the SSL stuff, you do not need to do anything related to it at backend.

    • about systemctl please check most of the things are disabled in linux at first install, you need to enable them for future restarts. if they are enabled they will start at boot time/startup.