Search code examples
nginxamazon-s3proxyreverse-proxyminio

nginx proxy in front of Minio-like S3 server bad request


I have a Minio-like S3 server on my lab local network that is exposed locally on LAN accessible thru http://s3.corph.mouradcloud.com This Minio is exposing the S3 API for object storage. For security and performance purposes, I added the nginx proxy and I want to expose this to internet as http://s3.mouradcloud.com

Local access is working fine on http://s3.corph.mouradcloud.com, however, I keep having the bad request message and a strange message telling the request was sent to https.. while every thing is Http only for the moment.

I used the documentation of minio here and alos tried the stackoverflow thread here

Both solutions are not working.

Here is my Nginx config :

    server {
     listen 80;
     server_name s3.mouradcloud.com;
     access_log /var/log/nginx/s3.mouradcloud.access.log ;
     error_log /var/log/nginx/s3.mouradcloud.errot.log debug;
     ignore_invalid_headers off;
     client_max_body_size 0;
     proxy_buffering off;

     location / {
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_set_header Host $http_host;
       proxy_connect_timeout 300;
       proxy_http_version 1.1;
       proxy_set_header Connection "";
       chunked_transfer_encoding off;
       proxy_pass http://s3.corph.mouradcloud.com;
     }

    }

and here is the error log received for simple browser get request :

        2020/03/25 16:47:50 [debug] 29649#29649: *11110 http header: "Host: s3.mouradcloud.com"
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 http header: "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0"
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 http header: "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 http header: "Accept-Language: en-US,en;q=0.5"
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 http header: "Accept-Encoding: gzip, deflate"
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 http header: "Connection: keep-alive"
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 http header: "Upgrade-Insecure-Requests: 1"
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 http header done
        2020/03/25 16:47:50 [info] 29649#29649: *11110 client sent plain HTTP request to HTTPS port while reading client request headers, client: 176.187.156.60, server: s3.mouradcloud.com, request: "GET / HTTP/1.1", host: "s3.mouradcloud.com"
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 http finalize request: 497, "/?" a:1, c:1
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 event timer del: 7: 353020654
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 http special response: 497, "/?"
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 http set discard body
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 xslt filter header
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 HTTP/1.1 400 Bad Request
        Server: nginx/1.14.0 (Ubuntu)
        Date: Wed, 25 Mar 2020 16:47:50 GMT
        Content-Type: text/html
        Content-Length: 280
        Connection: close

        2020/03/25 16:47:50 [debug] 29649#29649: *11110 write new buf t:1 f:0 0000556138469CB0, pos 0000556138469CB0, size: 161 file: 0, size: 0
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 http write filter: l:0 f:0 s:161
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 http output filter "/?"
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 http copy filter: "/?"
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 image filter
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 xslt filter body
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 http postpone filter "/?" 00005561384ECE10
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 write old buf t:1 f:0 0000556138469CB0, pos 0000556138469CB0, size: 161 file: 0, size: 0
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 write new buf t:0 f:0 0000000000000000, pos 0000556137684D60, size: 218 file: 0, size: 0
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 write new buf t:0 f:0 0000000000000000, pos 00005561376860A0, size: 62 file: 0, size: 0
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 http write filter: l:1 f:0 s:441
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 http write filter limit 0
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 writev: 441 of 441
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 http write filter 0000000000000000
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 http copy filter: 0 "/?"
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 http finalize request: 0, "/?" a:1, c:1
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 http request count:1 blk:0
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 http close request
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 http log handler
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 free: 0000556138468E90, unused: 0
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 free: 00005561384ECA30, unused: 2848
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 close http connection: 7
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 reusable connection: 0
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 free: 00005561384BAB80
        2020/03/25 16:47:50 [debug] 29649#29649: *11110 free: 00005561384437C0, unused: 136
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 http header: "Host: s3.mouradcloud.com"
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 http header: "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0"
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 http header: "Accept: image/webp,*/*"
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 http header: "Accept-Language: en-US,en;q=0.5"
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 http header: "Accept-Encoding: gzip, deflate"
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 http header: "Connection: keep-alive"
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 http header done
        2020/03/25 16:47:50 [info] 29649#29649: *11111 client sent plain HTTP request to HTTPS port while reading client request headers, client: 176.187.156.60, server: s3.mouradcloud.com, request: "GET /favicon.ico HTTP/1.1", host: "s3.mouradcloud.com"
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 http finalize request: 497, "/favicon.ico?" a:1, c:1
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 event timer del: 15: 353020666
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 http special response: 497, "/favicon.ico?"
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 http set discard body
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 xslt filter header
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 HTTP/1.1 400 Bad Request
        Server: nginx/1.14.0 (Ubuntu)
        Date: Wed, 25 Mar 2020 16:47:50 GMT
        Content-Type: text/html
        Content-Length: 280
        Connection: close

        2020/03/25 16:47:50 [debug] 29649#29649: *11111 write new buf t:1 f:0 0000556138469C98, pos 0000556138469C98, size: 161 file: 0, size: 0
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 http write filter: l:0 f:0 s:161
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 http output filter "/favicon.ico?"
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 http copy filter: "/favicon.ico?"
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 image filter
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 xslt filter body
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 http postpone filter "/favicon.ico?" 0000556138469E78
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 write old buf t:1 f:0 0000556138469C98, pos 0000556138469C98, size: 161 file: 0, size: 0
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 write new buf t:0 f:0 0000000000000000, pos 0000556137684D60, size: 218 file: 0, size: 0
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 write new buf t:0 f:0 0000000000000000, pos 00005561376860A0, size: 62 file: 0, size: 0
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 http write filter: l:1 f:0 s:441
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 http write filter limit 0
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 writev: 441 of 441
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 http write filter 0000000000000000
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 http copy filter: 0 "/favicon.ico?"
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 http finalize request: 0, "/favicon.ico?" a:1, c:1
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 http request count:1 blk:0
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 http close request
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 http log handler
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 free: 0000556138468E90, unused: 8
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 free: 00005561384ECA30, unused: 2853
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 close http connection: 15
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 reusable connection: 0
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 free: 00005561384BAB80
        2020/03/25 16:47:50 [debug] 29649#29649: *11111 free: 00005561384F3D50, unused: 136

Solution

  • Well, I got the answer. I used the solution from scality, the cloud server and actually this is related to the backend storage. Actually you have to add :

    • SSL on Nginx ( easy with certbot)
    • In the backend storage of scality, you have to :

      docker exec -it f8b1045ec132 bash
      cd conf/
      cat config.json
      rm config.json 
      json='{"port": 8000,"log": {"logLevel": "info","dumpLevel": "error"},"restEndpoints": {"localhost": "true","127.0            .0.1": "true","s3.mouradcloud.com": "true","s3.corph.mouradcloud.com": "true"  }}'
      echo $json > config.json
      exit # fron the container context 
      #stop and start the container again 
      docker stop f8b1045ec132 
      docker start f8b1045ec132