Search code examples
mongodbdockersslreplicaset

Running a MongoDB-Replica-Set inside docker with x509 authentication: Container Startup fails with tlsMode not enabled,

I am trying to set up a docker cluster/ replica-set with clusterAuthentication set to TLS. We use mongodb:7 (latest) docker container from the official docker-hub.

Currently we are stuck at startup (after a painful learning process of how to configure a openssl-selfsigned cert and csr)

Starting the first replica-set node throws immediately two errors:

{"t":{"$date":"2023-12-13T14:39:18.314Z"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"main","msg":"User assertion","attr":{"error":"BadValue: need to enable TLS via the sslMode/tlsMode flag when using TLS configuration parameters","file":"src/mongo/util/net/ssl_options_server.cpp","line":228}}

{"t":{"$date":"2023-12-13T14:39:18.314Z"},"s":"F",  "c":"CONTROL",  "id":20574,   "ctx":"main","msg":"Error during global initialization","attr":{"error":{"code":2,"codeName":"BadValue","errmsg":"need to enable TLS via the sslMode/tlsMode flag when using TLS configuration parameters"}}}

Which are more or less the same. I can't figure out, why this error is thrown. The tls-section is set in the mongod.conf, also it doesn't matter which value I use. (requireTLS, allowTLS or preferTLS)

We use this configuration:

file mongod.conf (yaml)

storage:
  dbPath: /var/lib/mongodb
systemLog:
  verbosity: 3
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log
net:
  tls:
    clusterAuthX509:
      attributes: O=TestOrganisation, OU=TestDepartment, CN=MongoDbCluster
    mode: requireTLS
    allowInvalidCertificates: true
    certificateKeyFile: /etc/certs/server.pem
    CAFile: /etc/certs/server.crt
    clusterFile: /etc/certs/server.pem
  bindIp: 0.0.0.0,mongodb-cluster
  port: 27017
processManagement:
  timeZoneInfo: /usr/share/zoneinfo
replication:
  replSetName: rs0
security:
  authorization: enabled
  clusterAuthMode: x509

We successfully generated the certs with following commands:

openssl genrsa -out server.key 4096
openssl req -x509 -new -nodes -sha256 -days 1825 -config ca_req.conf -newkey rsa:4096 -keyout serverROOTCA.key -out server.crt

openssl req -new -out server.csr -key server.key -config req_ext.conf -extensions v3_req

openssl x509 -req -in server.csr -CA server.crt -CAkey serverROOTKey.key -CAcreateserial -out server.crt -days 730 -sha256 -extfile req_ext.conf -extensions v3_req

We used this file for both, csr and actual cert as req_ext.conf, the req_conf for rootCA looks different.

[CA_default]
copy_extensions = copy
[req]
distinguished_name = client_ca
req_extensions = v3_req
prompt = no
[alt_names]
DNS.1 = mongodb-cluster
DNS.2 = replace_me_1
DNS.3 = replace_me_2
[client_ca]
C = SC
ST = SC
L = SampleCity
O = TestOrganisation
OU = TestDepartment
CN = MongoDbCluster
[v3_req]
subjectAltName = @alt_names
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = clientAuth, serverAuth

Also the server.pem contains both, key and cert with the corresponding prefixex BEGIN/END key or certificate. The "replace_me" values are actual alternative dns-names of the other servers. Each server has its own docker-network and exposes the mongodb-instance on the same port later on

For docker we rely on run-scripts and pass some env-values and files:

 docker run -d --restart always --name mongodb-cluster \
    --network traefik \
    -e MONGO_INITDB_ROOT_USERNAME=someuser \
    -e MONGO_INITDB_ROOT_PASSWORD=somepassword \
    -v $PWD/data:/var/lib/mongodb \
    -v $PWD/log/mongod.log:/var/log/mongodb/mongod.log \
    -v $PWD/conf/mongod.conf:/etc/mongod.conf \
    -v $PWD/certs/server.crt:/etc/certs/server.crt \
    -v $PWD/certs/server.pem:/etc/certs/server.pem \
    -p SOMEPORT:27017 \
    mongo:7 mongod -f /etc/mongod.conf

We followed this documentation for using a tls-protected x509 instance: https://www.mongodb.com/docs/manual/tutorial/configure-ssl/

We also tried to specify --tls or --tlsMode requireTLS as a start-parameter for the docker-run command, but this also results in the same error.

What could be the problem that this configuration does not work?

Edit: Additionaly the purposes for the root_ca when running the command:

openssl x509 -in server.crt -noout -text -purpose

Certificate purposes:
SSL client : Yes
SSL client CA : Yes (WARNING code=3)
SSL server : Yes
SSL server CA : Yes (WARNING code=3)
Netscape SSL server : Yes
Netscape SSL server CA : Yes (WARNING code=3)
S/MIME signing : Yes
S/MIME signing CA : Yes (WARNING code=3)
S/MIME encryption : Yes
S/MIME encryption CA : Yes (WARNING code=3)
CRL signing : Yes
CRL signing CA : Yes (WARNING code=3)
Any Purpose : Yes
Any Purpose CA : Yes
OCSP helper : Yes
OCSP helper CA : Yes (WARNING code=3)
Time Stamp signing : No
Time Stamp signing CA : Yes (WARNING code=3)

EDIT 2: The command

openssl -in server.pem -noout -ext keyUsage,extendedKeyUsage,basicConstraints

results in this output:

X509v3 Key Usage: 
   Key Encipherment, Data Encipherment
X509v3 Extended Key Usage: 
   TLS Web Client Authentication, TLS Web Server Authentication

for the root ca i get this output

openssl x509 -in server.crt -noout -ext keyUsage,extendedKeyUsage,basicConstraints
No extensions in certificate

Additional following ca_req.conf is used for the root-ca:

[req]
distinguished_name = RootCa
req_extensions = v3_req
prompt = no
[RootCa]
C = DE
ST = SC
L = SampleCity
O = TestOrganisation
OU = TestDepartment
CN = RootCa

[v3_req]
basicConstraints = CA:true, pathlen:0
keyUsage = keyCertSign
extendedKeyUsage = serverAuth
Solution

  • The CA output should be similar to this:

    X509v3 Key Usage: critical
    Certificate Sign, CRL Sign
X509v3 Basic Constraints: critical
    CA:TRUE

    I created my certificates like this:

    CA-Certificate config-file ca.conf:

    [req]
distinguished_name = req_distinguished_name
prompt = no
[req_distinguished_name]
C = CH
O = Company
OU = OSS
CN = Root CA
[v3_ca]
keyUsage = critical, keyCertSign, cRLSign
basicConstraints = critical, CA:true
subjectKeyIdentifier = hash

    Certificate config-file mongo.conf:

    [req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no
[req_distinguished_name]
C = CH
O = Company
OU = OSS
CN = MongoDB
[v3_req]
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = localhost

    And then use these commands:

    # Create private key for CA:
openssl genrsa -out ca.key 4096

# Create CA certificate:
openssl req -x509 -new -noenc -extensions v3_ca -config ca.conf -key ca.key -days 7305 -sha256 -out ca.cer


# Create certificate request with explicit private key:
openssl genrsa -out mongo.key 2048
openssl req -new -noenc -key mongo.key -config mongo.conf -out mongo.csr

# Alternative: Create certificate request with automatically generated private key
openssl req -new -noenc -newkey rsa:2048 -keyout mongo.key -config mongo.conf -out mongo.csr


# Create certificate, i.e. sign the certificate request
openssl x509 -req -in mongo.csr -CA ca.cer -CAkey ca.key -CAcreateserial -days 365 -sha512 -copy_extensions copyall -out mongo.cer

    Output:

    openssl x509 -in ca.cer -noout -subject -issuer -ext keyUsage,extendedKeyUsage,basicConstraints -purpose

subject=C = CH, O = Company, OU = OSS, CN = Root CA
issuer=C = CH, O = Company, OU = OSS, CN = Root CA

X509v3 Key Usage: critical
    Certificate Sign, CRL Sign
X509v3 Basic Constraints: critical
    CA:TRUE

Certificate purposes:
SSL client : No
SSL client CA : Yes
SSL server : No
SSL server CA : Yes
Netscape SSL server : No
Netscape SSL server CA : Yes
S/MIME signing : No
S/MIME signing CA : Yes
S/MIME encryption : No
S/MIME encryption CA : Yes
CRL signing : Yes
CRL signing CA : Yes
Any Purpose : Yes
Any Purpose CA : Yes
OCSP helper : Yes
OCSP helper CA : Yes
Time Stamp signing : No
Time Stamp signing CA : Yes


openssl x509 -in mongo.cer -noout -purpose -subject -issuer -ext keyUsage,extendedKeyUsage,basicConstraints,subjectAltName -purpose

subject=C = CH, O = Company, OU = OSS, CN = MongoDB
issuer=C = CH, O = Company, OU = OSS, CN = Root CA

X509v3 Key Usage: critical
    Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
    TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Subject Alternative Name:
    DNS:localhost

Certificate purposes:
SSL client : Yes
SSL client CA : No
SSL server : Yes
SSL server CA : No
Netscape SSL server : Yes
Netscape SSL server CA : No
S/MIME signing : No
S/MIME signing CA : No
S/MIME encryption : No
S/MIME encryption CA : No
CRL signing : No
CRL signing CA : No
Any Purpose : Yes
Any Purpose CA : Yes
OCSP helper : Yes
OCSP helper CA : No
Time Stamp signing : No
Time Stamp signing CA : No

    Once you managed to make it working, I would suggest to separate client and server certificate. That means

    [v3_req]
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = localhost

    is split to

    [v3_req]
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = localhost

    and

    [v3_req]
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth
# On client certificates, subjectAltName (SAN) is not used

    Using openssl-ca is a bit different. You can create certificates directly, i.e. without creating a certificate request, so a bit simpler to use. On the other hand openssl-ca uses a kind of mini-database for used/generated certificates, which makes it more complex again.

    Just a note, some time ago I discovered X Certificate and Key Management (or https://hohnstaedt.de/xca/index.php/download) - much easier to use than openssl from command line.