I have a Node.js server which uses MongoDb and Neo4j in a AKS Cluster all deployed via Helm Charts. I have A Certificate issued by Let's Encrypt which is used both for the server and for Neo4j bolt connection. I deployed the cluster in February and it all worked fine, but now when writing to Neo4j it throws the Failed to connect to server error with the Socket responded with: CERT_HAS_EXPIRED reason and Browser does not connect to the db.
Neo4jError: Failed to connect to server. Please ensure that your database is listening on the correct host and port and that you have compatible encryption settings both on Neo4j server and driver. Note that the default encryption setting has changed in Neo4j 4.0. Caused by: Server certificate is not trusted. If you trust the database you are connecting to, use TRUST_CUSTOM_CA_SIGNED_CERTIFICATES and add the signing certificate, or the server certificate, to the list of certificates trusted by this driver using `neo4j.driver(.., { trustedCertificates:['path/to/certificate.crt']}). This is a security measure to protect against man-in-the-middle attacks. If you are just trying Neo4j out and are not concerned about encryption, simply disable it using `encrypted="ENCRYPTION_OFF"` in the driver options. Socket responded with: CERT_HAS_EXPIRED
0|server | at new Neo4jError (/usr/app/node_modules/neo4j-driver-core/lib/error.js:77:16)
0|server | at newError (/usr/app/node_modules/neo4j-driver-core/lib/error.js:113:12)
0|server | at NodeChannel._handleConnectionError (/usr/app/node_modules/neo4j-driver-bolt-connection/lib/channel/node/node-channel.js:227:56)
0|server | at TLSSocket.<anonymous> (/usr/app/node_modules/neo4j-driver-bolt-connection/lib/channel/node/node-channel.js:69:17)
0|server | at Object.onceWrapper (node:events:641:28)
0|server | at TLSSocket.emit (node:events:527:28)
0|server | at TLSSocket.onConnectSecure (node:_tls_wrap:1564:10)
0|server | at TLSSocket.emit (node:events:527:28)
0|server | at TLSSocket._finishInit (node:_tls_wrap:945:8)
0|server | at ssl.onhandshakedone (node:_tls_wrap:726:12) {
0|server | constructor: [Function: Neo4jError] { isRetriable: [Function (anonymous)] },
0|server | code: 'ServiceUnavailable',
0|server | retriable: true
0|server | }
The certificate has been renewed in April and my guess is that Neo4j just stuck with the first certificate, is it possible?
this is the Certificate
Name: tls-certificate
Namespace: default
Labels: app.kubernetes.io/managed-by=Helm
Annotations: meta.helm.sh/release-name: cluster
meta.helm.sh/release-namespace: default
API Version: cert-manager.io/v1
Kind: Certificate
Metadata:
Creation Timestamp: 2023-02-15T15:25:52Z
Generation: 1
Managed Fields:
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:meta.helm.sh/release-name:
f:meta.helm.sh/release-namespace:
f:labels:
.:
f:app.kubernetes.io/managed-by:
f:spec:
.:
f:dnsNames:
f:issuerRef:
.:
f:kind:
f:name:
f:secretName:
Manager: helm
Operation: Update
Time: 2023-02-15T15:25:52Z
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:revision:
Manager: cert-manager-certificates-issuing
Operation: Update
Subresource: status
Time: 2023-04-16T14:27:13Z
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:conditions:
.:
k:{"type":"Ready"}:
.:
f:lastTransitionTime:
f:message:
f:observedGeneration:
f:reason:
f:status:
f:type:
f:notAfter:
f:notBefore:
f:renewalTime:
Manager: cert-manager-certificates-readiness
Operation: Update
Subresource: status
Time: 2023-04-16T14:27:13Z
Resource Version: 20023818
UID: 9edc761c-9382-4597-8048-ec5e85d0871d
Spec:
Dns Names:
xxx.westeurope.cloudapp.azure.com
Issuer Ref:
Kind: ClusterIssuer
Name: letsencrypt-issuer
Secret Name: tls-secret
Status:
Conditions:
Last Transition Time: 2023-02-15T15:26:48Z
Message: Certificate is up to date and has not expired
Observed Generation: 1
Reason: Ready
Status: True
Type: Ready
Not After: 2023-07-15T13:27:11Z
Not Before: 2023-04-16T13:27:12Z
Renewal Time: 2023-06-15T13:27:11Z
Revision: 2
Events: <none>
Now looking at the message
use TRUST_CUSTOM_CA_SIGNED_CERTIFICATES and add the signing certificate, or the server certificate, to the list of certificates trusted by this driver using `neo4j.driver(.., { trustedCertificates:['path/to/certificate.crt']})
I see that I haven't placed the certificate in the array of trusted certificates but only used the secret that the Certificate creates.
Neo4 ssl settings in the Neo4j Chart's Values :
ssl:
# setting per "connector" matching neo4j config
bolt:
privateKey:
secretName: tls-secret
subPath: tls.key
publicCertificate:
secretName: tls-secret
subPath: tls.crt
trustedCerts:
sources: []
revokedCerts:
sources: []
After scaling down Neo4j cluster to 0 replicas and up to 1 replicas it all start working again as expected, but how do I set it up in order to make this automatic?
I don't see any TRUST_CUSTOM_CA_SIGNED_CERTIFICATES parameter in the Neo4j Chart's Values. Also what would the path/to/certificate.crt be? Should I just add the name of the Certificate in the trustedCerts.sources array like sources: [tls-certificate]?
Many thanks.
Assuming you are using the helm charts https://github.com/neo4j/helm-charts you will need to refresh the secrets you created with the certs, and then either roll your cluster, or in your case, scale to 0 and then back up to pick up the secret. At this time (Neo4j 5.8.0) Certs are only picked up at startup. See https://neo4j.com/docs/operations-manual/current/kubernetes/configuration/#configure-ssl