Search code examples
sslamazon-s3tls1.3seaweedfsweed-fs

How do I enable TLS for Seaweedfs S3 API?

My local development consists of a SPA, an ASP.NET Core backend with Seaweedfs S3 API to store some images uploaded from the client. At first I was using S3 with HTTP and it was temporarily okay, everything works perfectly without the need of TLS. But I was wondering if enabling TLS on my localhost was possible. And how would that work in a production environment.

I have tried 2 ways:

  1. Created self signed CA certificate and signed a certificate for S3, and configured it like this inside my docker-compose:
version: '3.9'

services:
  master:
    image: chrislusf/seaweedfs:dev # use a remote dev image
    ports:
      - 9333:9333
      - 19333:19333
    command: "master -ip=master"
  volume:
    image: chrislusf/seaweedfs:dev # use a remote dev image
    ports:
      - 8080:8080
      - 18080:18080
    command: 'volume -mserver="master:9333" -port=8080 -ip=volume'
    depends_on:
      - master
  filer:
    image: chrislusf/seaweedfs:dev # use a remote dev image
    ports:
      - 8888:8888
      - 18888:18888
    command: 'filer -master="master:9333" -ip.bind=0.0.0.0'
    depends_on:
      - master
      - volume
  s3:
    image: chrislusf/seaweedfs:dev # use a remote dev image
    ports:
      - 8333:8333
    command: 's3 -filer="filer:8888" -ip.bind=0.0.0.0 -key.file=/etc/seaweed/server-key.pem -cert.file=/etc/seaweed/server-cert.pem'
    depends_on:
      - master
      - volume
      - filer
    volumes:
      - ./server-key.pem:/etc/seaweed/server-key.pem
      - ./server-cert.pem:/etc/seaweed/server-cert.pem
      - ./ca.pem:/etc/seaweed/ca.pem

but the logs when I try to access the port through HTTPS:

2023-12-23 00:05:12 I1222 17:05:12.391994 s3.go:203 wait to connect to filer filer:8888 grpc address filer:18888
2023-12-23 00:05:13 I1222 17:05:13.392769 s3.go:203 wait to connect to filer filer:8888 grpc address filer:18888
2023-12-23 00:05:14 I1222 17:05:14.394313 s3.go:203 wait to connect to filer filer:8888 grpc address filer:18888
2023-12-23 00:05:15 I1222 17:05:15.395713 s3.go:203 wait to connect to filer filer:8888 grpc address filer:18888
2023-12-23 00:05:16 I1222 17:05:16.397732 s3.go:203 wait to connect to filer filer:8888 grpc address filer:18888
2023-12-23 00:05:17 I1222 17:05:17.400436 s3.go:203 wait to connect to filer filer:8888 grpc address filer:18888
2023-12-23 00:05:18 I1222 17:05:18.402642 s3.go:203 wait to connect to filer filer:8888 grpc address filer:18888
2023-12-23 00:05:19 I1222 17:05:19.404917 s3.go:203 wait to connect to filer filer:8888 grpc address filer:18888
2023-12-23 00:05:20 I1222 17:05:20.406364 s3.go:203 wait to connect to filer filer:8888 grpc address filer:18888
2023-12-23 00:05:21 I1222 17:05:21.407280 s3.go:203 wait to connect to filer filer:8888 grpc address filer:18888
2023-12-23 00:05:22 I1222 17:05:22.408284 s3.go:203 wait to connect to filer filer:8888 grpc address filer:18888
2023-12-23 00:05:23 I1222 17:05:23.409097 s3.go:203 wait to connect to filer filer:8888 grpc address filer:18888
2023-12-23 00:05:24 I1222 17:05:24.411966 s3.go:203 wait to connect to filer filer:8888 grpc address filer:18888
2023-12-23 00:05:25 I1222 17:05:25.413006 s3.go:203 wait to connect to filer filer:8888 grpc address filer:18888
2023-12-23 00:05:26 I1222 17:05:26.414096 s3.go:203 wait to connect to filer filer:8888 grpc address filer:18888
2023-12-23 00:05:27 I1222 17:05:27.416081 s3.go:203 wait to connect to filer filer:8888 grpc address filer:18888
2023-12-23 00:05:28 I1222 17:05:28.418798 s3.go:203 wait to connect to filer filer:8888 grpc address filer:18888
2023-12-23 00:05:29 I1222 17:05:29.421864 s3.go:203 wait to connect to filer filer:8888 grpc address filer:18888
2023-12-23 00:05:30 I1222 17:05:30.424521 s3.go:203 wait to connect to filer filer:8888 grpc address filer:18888
2023-12-23 00:05:31 I1222 17:05:31.427257 s3.go:203 wait to connect to filer filer:8888 grpc address filer:18888
2023-12-23 00:05:32 I1222 17:05:32.429834 s3.go:203 wait to connect to filer filer:8888 grpc address filer:18888
2023-12-23 00:05:33 I1222 17:05:33.433957 s3.go:199 S3 read filer buckets dir: /buckets
2023-12-23 00:05:33 I1222 17:05:33.433999 s3.go:206 connected to filer filer:8888 grpc address filer:18888
2023-12-23 00:05:33 W1222 17:05:33.434679 auth_credentials.go:128 fail to load config: read S3 config: filer: no entry is found in filer store
2023-12-23 00:05:33 I1222 17:05:33.435244 s3api_circuit_breaker.go:35 s3 circuit breaker not configured
2023-12-23 00:05:33 I1222 17:05:33.439810 s3.go:320 Start Seaweed S3 API Server 30GB 3.59 453826541 at https port 8333
2023-12-23 00:05:57 2023/12/22 17:05:57 http: TLS handshake error from 172.19.0.1:43416: remote error: tls: unknown certificate
2023-12-23 00:05:57 2023/12/22 17:05:57 http: TLS handshake error from 172.19.0.1:43430: remote error: tls: unknown certificate
2023-12-23 00:06:01 2023/12/22 17:06:01 http: TLS handshake error from 172.19.0.1:43402: read tcp 172.19.0.5:8333->172.19.0.1:43402: i/o timeout
2023-12-23 00:06:46 2023/12/22 17:06:46 http: TLS handshake error from 172.19.0.1:50130: remote error: tls: unknown certificate
2023-12-23 00:06:46 2023/12/22 17:06:46 http: TLS handshake error from 172.19.0.1:50142: remote error: tls: unknown certificate
2023-12-23 00:06:53 2023/12/22 17:06:53 http: TLS handshake error from 172.19.0.1:40220: remote error: tls: unknown certificate
2023-12-23 00:06:53 2023/12/22 17:06:53 http: TLS handshake error from 172.19.0.1:40224: remote error: tls: unknown certificate
2023-12-23 00:06:53 2023/12/22 17:06:53 http: TLS handshake error from 172.19.0.1:40226: remote error: tls: unknown certificate
2023-12-23 00:06:56 2023/12/22 17:06:56 http: TLS handshake error from 172.19.0.1:40230: remote error: tls: unknown certificate
2023-12-23 00:06:56 2023/12/22 17:06:56 http: TLS handshake error from 172.19.0.1:40234: remote error: tls: unknown certificate
2023-12-23 01:31:08 2023/12/22 18:31:08 http2: server: error reading preface from client 172.19.0.1:49596: remote error: tls: unknown certificate authority
2023-12-23 01:31:11 2023/12/22 18:31:11 http2: server: error reading preface from client 172.19.0.1:46566: remote error: tls: unknown certificate authority
2023-12-23 01:31:11 2023/12/22 18:31:11 http2: server: error reading preface from client 172.19.0.1:46574: remote error: tls: unknown certificate authority
2023-12-23 01:44:56 2023/12/22 18:44:56 http: TLS handshake error from 172.19.0.1:57452: remote error: tls: unknown certificate
2023-12-23 01:44:57 2023/12/22 18:44:57 http: TLS handshake error from 172.19.0.1:57460: remote error: tls: unknown certificate
2023-12-23 01:44:57 2023/12/22 18:44:57 http: TLS handshake error from 172.19.0.1:57470: remote error: tls: unknown certificate
2023-12-23 01:47:14 2023/12/22 18:47:14 http: TLS handshake error from 172.19.0.1:50270: remote error: tls: unknown certificate
2023-12-23 01:47:14 2023/12/22 18:47:14 http: TLS handshake error from 172.19.0.1:50276: remote error: tls: unknown certificate
2023-12-23 01:47:28 2023/12/22 18:47:28 http: TLS handshake error from 172.19.0.1:47388: remote error: tls: unknown certificate
2023-12-23 01:47:28 2023/12/22 18:47:28 http: TLS handshake error from 172.19.0.1:47396: remote error: tls: unknown certificate
2023-12-23 01:48:13 2023/12/22 18:48:13 http: TLS handshake error from 172.19.0.1:34004: remote error: tls: unknown certificate
2023-12-23 01:48:13 2023/12/22 18:48:13 http: TLS handshake error from 172.19.0.1:34016: remote error: tls: unknown certificate
2023-12-23 01:48:14 2023/12/22 18:48:14 http: TLS handshake error from 172.19.0.1:34020: remote error: tls: unknown certificate
2023-12-23 01:48:14 2023/12/22 18:48:14 http: TLS handshake error from 172.19.0.1:34024: remote error: tls: unknown certificate
2023-12-23 01:48:14 2023/12/22 18:48:14 http: TLS handshake error from 172.19.0.1:34036: remote error: tls: unknown certificate
2023-12-23 01:48:14 2023/12/22 18:48:14 http: TLS handshake error from 172.19.0.1:34052: remote error: tls: unknown certificate
2023-12-23 01:48:14 2023/12/22 18:48:14 http: TLS handshake error from 172.19.0.1:34072: remote error: tls: unknown certificate
2023-12-23 01:48:14 2023/12/22 18:48:14 http: TLS handshake error from 172.19.0.1:34064: remote error: tls: unknown certificate
2023-12-23 01:48:15 2023/12/22 18:48:15 http: TLS handshake error from 172.19.0.1:34084: remote error: tls: unknown certificate
2023-12-23 01:48:15 2023/12/22 18:48:15 http: TLS handshake error from 172.19.0.1:34100: remote error: tls: unknown certificate
  1. put a reverse proxy (YARP in this case) before the S3 to handle TLS termination, the GET request works fine, but if I call method AmazonS3Util.DoesS3BucketExistV2Async in .NET it returns a 403 saying the signature is wrong, meanwhile if I turn off HTTPS it works. I think that this is more of an S3/Seaweed problem than TLS.

any idea how do you guys do it in the real world? I may need to be enlightened on TLS/SSL so any help would be appreciated.

Solution

  • Per https://github.com/seaweedfs/seaweedfs/discussions/5345#discussioncomment-8606445

    there 3 options are required:

    -s3.port.https=8334 -s3.cert.file=/etc/seaweedfs/certs/s3.crt -s3.key.file=/etc/seaweedfs/certs/s3.key