So, I am trying to create a fabric network with a root CA, and several intermediate CAs, one for each org, everything with TLS enabled and self signed certs (no cryptogen). After running the root CA, I enroll the admin for it, and create affiliations and register identities for the intermediate CAs using the flag hf.IntermediateCA=true. Then I copy the tls-cert.pem from the root CA to the volume to be used by the intermediate CA, and then I try to start the intermediate CA issuing:
fabric-ca-server start -b admin:adminpw -u https://<registered-identity>:<pw>@ca-root:7054 --tls.certfile /path/to/copied/tls-cert.pem
And I get the following answer:
[lots of things...]
Post https://ca-root:7054/enroll: dial tcp 10.111.83.239:7054: connect: connection refused
Without any clue about why the connection was refused. Checking the logs of the root CA looks like if nothing happened. Also I know it is not a connectivity problem (I'm on minikube and as can be seen in the logs, the DNS is being resolved correctly - double checked). I tried using env variables instead of the CLI flag, also tried using the flag --intermediate.tls.certfiles with the same certificate, and also tried the ca-root-cert.pem instead of the tls-cert.pem (which I think should also work), all with the same result.
Important to say that I've been able to start intermediate CAs without problem if I disable TLS everywhere, but I really need to enable this.
I am using fabric 1.4.2 images.
What am I missing? Thanks!
Edit: To answer @kekomal questions: I used both flags for TLS cert and enebled debug mode. Here's the complete Intermediate CA's log:
2019/10/25 08:22:56 [DEBUG] Home directory: /etc/hyperledger/fabric-ca-server
2019/10/25 08:22:56 [INFO] Configuration file location: /etc/hyperledger/fabric-ca-server/fabric-ca-server-config.yaml
2019/10/25 08:22:56 [INFO] Starting server in home directory: /etc/hyperledger/fabric-ca-server
2019/10/25 08:22:56 [DEBUG] Set log level:
2019/10/25 08:22:56 [INFO] Server Version: 1.4.2
2019/10/25 08:22:56 [INFO] Server Levels: &{Identity:2 Affiliation:1 Certificate:1 Credential:1 RAInfo:1 Nonce:1}
2019/10/25 08:22:56 [DEBUG] Making server filenames absolute
2019/10/25 08:22:56 [DEBUG] Initializing default CA in directory /etc/hyperledger/fabric-ca-server
2019/10/25 08:22:56 [DEBUG] Init CA with home /etc/hyperledger/fabric-ca-server and config {Version:1.4.2 Cfg:{Identities:{PasswordAttempts:10 AllowRemove:false} Affiliations:{AllowRemove:false}} CA:{Name:ca-corp Keyfile: Certfile:ca-cert.pem Chainfile:ca-chain.pem} Signing:0xc000220870 CSR:{CN: Names:[{C:US ST:North Carolina L: O:Hyperledger OU:Fabric SerialNumber:}] Hosts:[0.0.0.0 ca-corp ca-corp.blockchain-alm-ns.svc localhost] KeyRequest:0xc00022cfe0 CA:0xc00022d060 SerialNumber:} Registry:{MaxEnrollments:-1 Identities:[{ Name:**** Pass:**** Type:client Affiliation: MaxEnrollments:0 Attrs:map[hf.Registrar.Roles:* hf.Registrar.DelegateRoles:* hf.Revoker:1 hf.IntermediateCA:1 hf.GenCRL:1 hf.Registrar.Attributes:* hf.AffiliationMgr:1] }]} Affiliations:map[org1:[department1 department2] org2:[department1]] LDAP:{ Enabled:false URL:ldap://****:****@<host>:<port>/<base> UserFilter:(uid=%s) GroupFilter:(memberUid=%s) Attribute:{[uid member] [{ }] map[groups:[{ }]]} TLS:{false [] { }} } DB:{ Type:sqlite3 Datasource:fabric-ca-server.db TLS:{false [] { }} } CSP:0xc00022d100 Client:<nil> Intermediate:{ParentServer:{ URL:https://****:****@ca-root:7054 CAName: } TLS:{Enabled:false CertFiles:[/etc/hyperledger/fabric-ca-server/tls-cert.pem] Client:{KeyFile: CertFile:}} Enrollment:{ Name: Secret:**** CAName: AttrReqs:[] Profile: Label: CSR:<nil> Type:x509 }} CRL:{Expiry:24h0m0s} Idemix:{IssuerPublicKeyfile: IssuerSecretKeyfile: RevocationPublicKeyfile: RevocationPrivateKeyfile: RHPoolSize:1000 NonceExpiration:15s NonceSweepInterval:15m}}
2019/10/25 08:22:56 [DEBUG] CA Home Directory: /etc/hyperledger/fabric-ca-server
2019/10/25 08:22:56 [DEBUG] Checking configuration file version '1.4.2' against server version: '1.4.2'
2019/10/25 08:22:56 [DEBUG] Initializing BCCSP: &{ProviderName:SW SwOpts:0xc0001d0500 PluginOpts:<nil>}
2019/10/25 08:22:56 [DEBUG] Initializing BCCSP with software options &{SecLevel:256 HashFamily:SHA2 Ephemeral:false FileKeystore:0xc00022f4b0 DummyKeystore:<nil> InmemKeystore:<nil>}
2019/10/25 08:22:56 [DEBUG] Initialize key material
2019/10/25 08:22:56 [DEBUG] Making CA filenames absolute
2019/10/25 08:22:56 [WARNING] &{69 The specified CA certificate file /etc/hyperledger/fabric-ca-server/ca-cert.pem does not exist}
2019/10/25 08:22:56 [DEBUG] Getting CA cert; parent server URL is https://****:****@ca-root:7054
2019/10/25 08:22:56 [DEBUG] Intermediate enrollment request: { Name: Secret:**** CAName: AttrReqs:[] Profile:ca Label: CSR:&{ [] [] <nil> 0xc00022d5a0 } Type:x509 }, CSR: &{CN: Names:[] Hosts:[] KeyRequest:<nil> CA:0xc00022d5a0 SerialNumber:}, CA: &{PathLength:0 PathLenZero:true Expiry:}
2019/10/25 08:22:56 [DEBUG] Enrolling { Name:ca-corp Secret:**** CAName: AttrReqs:[] Profile:ca Label: CSR:&{ [{US North Carolina Hyperledger Fabric }] [0.0.0.0 ca-corp ca-corp.blockchain-alm-ns.svc localhost] 0xc00022cfe0 0xc00022d060 } Type:x509 }
2019/10/25 08:22:56 [DEBUG] Initializing client with config: &{URL:https://ca-root:7054 MSPDir: TLS:{Enabled:true CertFiles:[/etc/hyperledger/fabric-ca-server/tls-cert.pem] Client:{KeyFile: CertFile:}} Enrollment:{ Name:ca-corp Secret:**** CAName: AttrReqs:[] Profile:ca Label: CSR:&{ [{US North Carolina Hyperledger Fabric }] [0.0.0.0 ca-corp ca-corp.blockchain-alm-ns.svc localhost] 0xc00022cfe0 0xc00022d060 } Type:x509 } CSR:{CN: Names:[{C:US ST:North Carolina L: O:Hyperledger OU:Fabric SerialNumber:}] Hosts:[0.0.0.0 ca-corp ca-corp.blockchain-alm-ns.svc localhost] KeyRequest:0xc00022cfe0 CA:0xc00022d060 SerialNumber:} ID:{Name: Type: Secret: MaxEnrollments:0 Affiliation: Attributes:[] CAName:} Revoke:{Name: Serial: AKI: Reason: CAName: GenCRL:false} CAInfo:{CAName:} CAName: CSP:0xc00022d100 Debug:false LogLevel:}
2019/10/25 08:22:56 [DEBUG] Initializing BCCSP: &{ProviderName:SW SwOpts:0xc0001d0500 PluginOpts:<nil>}
2019/10/25 08:22:56 [DEBUG] Initializing BCCSP with software options &{SecLevel:256 HashFamily:SHA2 Ephemeral:false FileKeystore:0xc00022f4b0 DummyKeystore:<nil> InmemKeystore:<nil>}
2019/10/25 08:22:56 [INFO] TLS Enabled
2019/10/25 08:22:56 [DEBUG] CA Files: [/etc/hyperledger/fabric-ca-server/tls-cert.pem]
2019/10/25 08:22:56 [DEBUG] Client Cert File:
2019/10/25 08:22:56 [DEBUG] Client Key File:
2019/10/25 08:22:56 [DEBUG] Client TLS certificate and/or key file not provided
2019/10/25 08:22:56 [DEBUG] GenCSR &{CN: Names:[{C:US ST:North Carolina L: O:Hyperledger OU:Fabric SerialNumber:}] Hosts:[0.0.0.0 ca-corp ca-corp.blockchain-alm-ns.svc localhost] KeyRequest:0xc00022cfe0 CA:0xc00022d060 SerialNumber:}
2019/10/25 08:22:56 [INFO] generating key: &{A:ecdsa S:256}
2019/10/25 08:22:56 [DEBUG] generate key from request: algo=ecdsa, size=256
2019/10/25 08:22:56 [INFO] encoded CSR
2019/10/25 08:22:56 [DEBUG] Sending request
POST https://ca-root:7054/enroll
{"hosts":["0.0.0.0","ca-corp","ca-corp.blockchain-alm-ns.svc","localhost"],"certificate_request":"-----BEGIN CERTIFICATE REQUEST-----\nMIIBgzCCASoCAQAwXzELMAkGA1UEBhMCVVMxFzAVBgNVBAgTDk5vcnRoIENhcm9s\naW5hMRQwEgYDVQQKEwtIeXBlcmxlZGdlcjEPMA0GA1UECxMGRmFicmljMRAwDgYD\nVQQDEwdjYS1jb3JwMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEws3abjcupN/X\nVa53vjv9Cdv1oVT/fDF4wNLGrNs6gMazFKN+0FCBQIywQxbdqszAxwl0Wh1yYzm5\np4E17FPPnqBpMGcGCSqGSIb3DQEJDjFaMFgwQgYDVR0RBDswOYIHY2EtY29ycIId\nY2EtY29ycC5ibG9ja2NoYWluLWFsbS1ucy5zdmOCCWxvY2FsaG9zdIcEAAAAADAS\nBgNVHRMBAf8ECDAGAQH/AgEAMAoGCCqGSM49BAMCA0cAMEQCIHvXuMelMU9gVgWu\nb0tURDxJ/W5yvwikVEiMjAFma9tpAiBl3YQ1pbcNH53QAhn/4TjLaQLKeVDrK4a9\nX3/4HvG8iw==\n-----END CERTIFICATE REQUEST-----\n","profile":"ca","crl_override":"","label":"","NotBefore":"0001-01-01T00:00:00Z","NotAfter":"0001-01-01T00:00:00Z","CAName":""}
2019/10/25 08:22:57 [DEBUG] Closing server DBs
Error: POST failure of request: POST https://ca-root:7054/enroll
{"hosts":["0.0.0.0","ca-corp","ca-corp.blockchain-alm-ns.svc","localhost"],"certificate_request":"-----BEGIN CERTIFICATE REQUEST-----\nMIIBgzCCASoCAQAwXzELMAkGA1UEBhMCVVMxFzAVBgNVBAgTDk5vcnRoIENhcm9s\naW5hMRQwEgYDVQQKEwtIeXBlcmxlZGdlcjEPMA0GA1UECxMGRmFicmljMRAwDgYD\nVQQDEwdjYS1jb3JwMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEws3abjcupN/X\nVa53vjv9Cdv1oVT/fDF4wNLGrNs6gMazFKN+0FCBQIywQxbdqszAxwl0Wh1yYzm5\np4E17FPPnqBpMGcGCSqGSIb3DQEJDjFaMFgwQgYDVR0RBDswOYIHY2EtY29ycIId\nY2EtY29ycC5ibG9ja2NoYWluLWFsbS1ucy5zdmOCCWxvY2FsaG9zdIcEAAAAADAS\nBgNVHRMBAf8ECDAGAQH/AgEAMAoGCCqGSM49BAMCA0cAMEQCIHvXuMelMU9gVgWu\nb0tURDxJ/W5yvwikVEiMjAFma9tpAiBl3YQ1pbcNH53QAhn/4TjLaQLKeVDrK4a9\nX3/4HvG8iw==\n-----END CERTIFICATE REQUEST-----\n","profile":"ca","crl_override":"","label":"","NotBefore":"0001-01-01T00:00:00Z","NotAfter":"0001-01-01T00:00:00Z","CAName":""}: Post https://ca-root:7054/enroll: dial tcp 10.110.28.98:7054: connect: connection refused
And the logs from the CA root:
2019/10/25 08:18:21 [DEBUG] DB: Getting identity admin
2019/10/25 08:18:21 [DEBUG] Successful token authentication of 'admin'
2019/10/25 08:18:21 [DEBUG] Received registration request from admin: { Name:ca-corp Type:client Secret:**** MaxEnrollments:0 Affiliation: Attributes:[{hf.IntermediateCA true false}] CAName: }
2019/10/25 08:18:21 [DEBUG] No affiliation provided in registration request, will default to using registrar's affiliation of ''
2019/10/25 08:18:21 [DEBUG] canRegister - Check to see if user 'admin' can register
2019/10/25 08:18:21 [DEBUG] Checking to see if caller 'admin' can act on type 'client'
2019/10/25 08:18:21 [DEBUG] Checking to see if caller 'admin' is a registrar
2019/10/25 08:18:21 [DEBUG] Validating affiliation:
2019/10/25 08:18:21 [DEBUG] Checking to see if affiliation '' contains caller's affiliation ''
2019/10/25 08:18:21 [DEBUG] Caller has root affiliation
2019/10/25 08:18:21 [DEBUG] Checking to see if registrar can register the requested attributes: [{Name:hf.IntermediateCA Value:true ECert:false}]
2019/10/25 08:18:21 [DEBUG] Validating that registrar with the following values for hf.Registrar.Attributes '*' is authorized to register the requested attribute '&{Name:hf.IntermediateCA Value:true ECert:false}'
2019/10/25 08:18:21 [DEBUG] Checking if registrar can register attribute: hf.IntermediateCA
2019/10/25 08:18:21 [DEBUG] Performing authorization check...
2019/10/25 08:18:21 [DEBUG] Checking if caller is authorized to register attribute 'hf.IntermediateCA' with the requested value of 'true'
2019/10/25 08:18:21 [DEBUG] Requested attribute type is boolean
2019/10/25 08:18:21 [DEBUG] Registering user id: ca-corp
2019/10/25 08:18:21 [DEBUG] Max enrollment value verification - User specified max enrollment: 0, CA max enrollment: -1
2019/10/25 08:18:21 [DEBUG] DB: Getting identity ca-corp
2019/10/25 08:18:21 [DEBUG] DB: Add identity ca-corp
2019/10/25 08:18:21 [DEBUG] Successfully added identity ca-corp to the database
2019/10/25 08:18:21 [INFO] 127.0.0.1:34862 POST /register 201 0 "OK"
2019/10/25 08:30:30 [DEBUG] Cleaning up expired nonces for CA 'ca-root'
Where you can see the previous process of registering the identity that should be enrolled when starting the intermediate CA. But does not show anything for the enrolling process (the refused connection).
Regarding your comment: you say that I should also send --tls.keyfile while starting intermediate CA. I don't understand this... private key should always be kept by the owner (in this case the root CA), should I also copy it to intermediate CA's volume? why? does that make any sense?
Edit 2: Finally @kekomal's answer gave me the clue: adding information about the CN of the parent CA let me go further. The working command is:
fabric-ca-server start -b admin:adminpw -u https://<registered-identity>:<pw>@ca-root:7054 --intermediate.parentserver.caname ca-root --tls.certfile /path/to/copied/tls-cert.pem
Where the new part is --intermediate.parentserver.caname ca-root. BUT... Issuing this command gave me a new error in the intermediate CA's log:
Error: Could not load TLS certificate with BCCSP: Could not find matching private key for SKI: Failed getting key for SKI [[167 199 194 85 105 166 191 99 62 19 149 57 140 61 6 182 134 104 105 223 132 142 13 221 99 68 170 0 45 246 28 17]]: Key with SKI a7c7c25569a6bf633e1395398c3d06b6866869df848e0ddd6344aa002df61c11 not found in /etc/hyperledger/fabric-ca-server/msp/keystore
As the error suggested, I copied the private key from the root CA to the intermediate CA inside /etc/hyperledger/fabric-ca-server/msp/keystore, and just after that, the intermediate CA finally started.
But why is it necessary to copy this PK from root to intermediate CA? I'm really confused as we are working with asymmetric cryptography. Is it a bug? or am I missing something important?
Edit 3: Ok, one more update on this: As suggested by @kekomal, I tried issuing the command without enabling tls in the intermediate CA. So I stopped the network, deleted persistence for intermediate CA, and issued the command as suggested. To my surprise, the result was again a connection refused message. Then I tried again all the variations of fabric-ca-server start with no luck (same connection refused message). Finally I realized that the last time it worked, I issued first a fabric-ca-server start without params (to start it as a stand alone server), stopped it, and then sent the command with params. So, what I have done this time is to send first a fabric-ca-server init -b admin:adminpw, and then send the fabric-ca-server start -b admin:adminpw -u https://ca-corp:NdckmlUAhTUP@ca-root:7054 --tls.enabled --intermediate.parentserver.caname ca-root --intermediate.tls.certfiles /etc/hyperledger/fabric-ca-server/tls-cert.pem (yes, with --tls.enabled, because I want the entire CAs network to work with tls enabled, not just the root CA), and yes!, intermediate CA is now working, and yes!, there was no need to copy the root CA's PK. My conclussion is that even when fabric documentation states that starting without running init before will implicitly run init, something is not working fine there, and init must be called before start for things to work fine, at least when TLS is enabled.
Mmm... The parent root CA TLS certificate (the trusted root certificates) should be specified with --intermediate.tls.certfiles. --tls.certfile and --tls.keyfile define the intermediate CA TLS certificate and key respect to its children/clients.
Set --loglevel debug to get more information (and share it).
Edit: To answer to your edition (I have no reputation to comment your post), when you talk about --tls.keyfile. I don't mean that you must use it. I mean that, if you use --tls.certfile, you must also use --tls.keyfile, as they are intended to serve your intermediate CA services via TLS, not to trust your parent root CA's TLS certificate. For your purpose, you have --intermediate.tls.certfiles.
Your command should look like:
fabric-ca-server start -b admin:adminpw -u https://<registered-identity>:<pw>@ca-root:7054 --intermediate.parentserver.caname ca-root --intermediate.tls.certfiles /path/to/copied/tls-cert.pem
Possible errors I can think of:
ca-root does not point to your root CA.ca-root.If you say that it worked when the root CA did not use TLS, and you have not broken any other thing during your changes, it can be one of 4-6...