I've been trying to set up a Hyperledger Fabric network for days. I'm using version 2.0.0 of Hyperledger and this is all running on one maching using Docker. After difficult progress, this is the part that stumps me now. I thought I had most of it firgured out. This is a variant of "Your first network" from the docs, with just a few names changed. There are two peers and five orderers. When I look at the logs for the first orderer with docker logs <container name>, I get this, which is not good:
(See below for an example.)
For information, the start of my docker-compose-cli.yaml looks like this, as adding the orders explicitly seemed to remove some errors:
version: '2'
volumes:
orderer.diro.umontreal.ca:
orderer2.diro.umontreal.ca:
orderer3.diro.umontreal.ca:
orderer4.diro.umontreal.ca:
orderer5.diro.umontreal.ca:
peer0.med1.diro.umontreal.ca:
peer1.med1.diro.umontreal.ca:
peer0.med2.diro.umontreal.ca:
peer1.med2.diro.umontreal.ca:
networks:
byfn:
services:
orderer.diro.umontreal.ca:
extends:
file: base/docker-compose-base.yaml
service: orderer.diro.umontreal.ca
container_name: orderer.diro.umontreal.ca
networks:
- byfn
orderer2.diro.umontreal.ca:
extends:
file: base/docker-compose-base.yaml
service: orderer.diro.umontreal.ca
container_name: orderer2.diro.umontreal.ca
networks:
- byfn
orderer3.diro.umontreal.ca:
extends:
file: base/docker-compose-base.yaml
service: orderer.diro.umontreal.ca
container_name: orderer3.diro.umontreal.ca
networks:
- byfn
orderer4.diro.umontreal.ca:
extends:
file: base/docker-compose-base.yaml
service: orderer.diro.umontreal.ca
container_name: orderer4.diro.umontreal.ca
networks:
- byfn
orderer5.diro.umontreal.ca:
extends:
file: base/docker-compose-base.yaml
service: orderer.diro.umontreal.ca
container_name: orderer5.diro.umontreal.ca
networks:
- byfn
The start of my crypto-config.yaml looks like this:
OrdererOrgs:
# ---------------------------------------------------------------------------
# Orderer
# ---------------------------------------------------------------------------
- Name: Orderer
Domain: diro.umontreal.ca
EnableNodeOUs: true
# ---------------------------------------------------------------------------
# "Specs" - See PeerOrgs below for complete description
# ---------------------------------------------------------------------------
Specs:
- Hostname: orderer
- Name: Orderer2
Domain: diro.umontreal.ca
EnableNodeOUs: true
# ---------------------------------------------------------------------------
# "Specs" - See PeerOrgs below for complete description
# ---------------------------------------------------------------------------
Specs:
- Hostname: orderer2
- Name: Orderer3
Domain: diro.umontreal.ca
EnableNodeOUs: true
# ---------------------------------------------------------------------------
# "Specs" - See PeerOrgs below for complete description
# ---------------------------------------------------------------------------
Specs:
- Hostname: orderer3
- Name: Orderer4
Domain: diro.umontreal.ca
EnableNodeOUs: true
# ---------------------------------------------------------------------------
# "Specs" - See PeerOrgs below for complete description
# ---------------------------------------------------------------------------
Specs:
- Hostname: orderer4
- Name: Orderer5
Domain: diro.umontreal.ca
EnableNodeOUs: true
# ---------------------------------------------------------------------------
# "Specs" - See PeerOrgs below for complete description
# ---------------------------------------------------------------------------
Specs:
- Hostname: orderer5
This is part of my configx.yaml:
SampleMultiNodeEtcdRaft:
<<: *ChannelDefaults
Capabilities:
<<: *ChannelCapabilities
Orderer:
<<: *OrdererDefaults
OrdererType: etcdraft
EtcdRaft:
Consenters:
- Host: orderer.diro.umontreal.ca
Port: 7050
ClientTLSCert: crypto-config/ordererOrganizations/diro.umontreal.ca/orderers/orderer.diro.umontreal.ca/msp/tlscacerts/tlsca.diro.umontreal.ca-cert.pem
ServerTLSCert: crypto-config/ordererOrganizations/diro.umontreal.ca/orderers/orderer.diro.umontreal.ca/tls/server.crt
- Host: orderer2.diro.umontreal.ca
Port: 8050
ClientTLSCert: crypto-config/ordererOrganizations/diro.umontreal.ca/orderers/orderer2.diro.umontreal.ca/msp/tlscacerts/tlsca.diro.umontreal.ca-cert.pem
ServerTLSCert: crypto-config/ordererOrganizations/diro.umontreal.ca/orderers/orderer2.diro.umontreal.ca/tls/server.crt
- Host: orderer3.diro.umontreal.ca
Port: 9050
ClientTLSCert: crypto-config/ordererOrganizations/diro.umontreal.ca/orderers/orderer3.diro.umontreal.ca/msp/tlscacerts/tlsca.diro.umontreal.ca-cert.pem
ServerTLSCert: crypto-config/ordererOrganizations/diro.umontreal.ca/orderers/orderer3.diro.umontreal.ca/tls/server.crt
- Host: orderer4.diro.umontreal.ca
Port: 10050
ClientTLSCert: crypto-config/ordererOrganizations/diro.umontreal.ca/orderers/orderer4.diro.umontreal.ca/msp/tlscacerts/tlsca.diro.umontreal.ca-cert.pem
ServerTLSCert: crypto-config/ordererOrganizations/diro.umontreal.ca/orderers/orderer4.diro.umontreal.ca/tls/server.crt
- Host: orderer5.diro.umontreal.ca
Port: 11050
ClientTLSCert: crypto-config/ordererOrganizations/diro.umontreal.ca/orderers/orderer5.diro.umontreal.ca/msp/tlscacerts/tlsca.diro.umontreal.ca-cert.pem
ServerTLSCert: crypto-config/ordererOrganizations/diro.umontreal.ca/orderers/orderer5.diro.umontreal.ca/tls/server.crt
Addresses:
- orderer.diro.umontreal.ca:7050
- orderer2.diro.umontreal.ca:8050
- orderer3.diro.umontreal.ca:9050
- orderer4.diro.umontreal.ca:10050
- orderer5.diro.umontreal.ca:11050
Originally, all the ClientTLSCert and ServerTLSCert had the same values, but I changed the ClientTLSCert values, thiniking it didn't make sense that their argument would be a file that was called server.crt.
But I still have all the errors in the my orderer logs. Maybe that's normal, but it doesn't look normal. What's wrong?
EDIT:
Based on the comment below, I changed configtx.yaml to start with:
Organizations:
# SampleOrg defines an MSP using the sampleconfig. It should never be used
# in production but may be used as a template for other definitions
- &OrdererOrg
# DefaultOrg defines the organization which is used in the sampleconfig
# of the fabric.git development environment
Name: OrdererOrg
# ID to load the MSP definition as
ID: OrdererMSP
# MSPDir is the filesystem path which contains the MSP configuration
MSPDir: crypto-config/ordererOrganizations/diro.umontreal.ca/orderers/orderer.diro.umontreal.ca/msp/
MSPDir: crypto-config/ordererOrganizations/diro.umontreal.ca/orderers/orderer2.diro.umontreal.ca/msp/
MSPDir: crypto-config/ordererOrganizations/diro.umontreal.ca/orderers/orderer3.diro.umontreal.ca/msp/
MSPDir: crypto-config/ordererOrganizations/diro.umontreal.ca/orderers/orderer4.diro.umontreal.ca/msp/
MSPDir: crypto-config/ordererOrganizations/diro.umontreal.ca/orderers/orderer5.diro.umontreal.ca/msp/
And I made docker-compose-base.yaml start with:
version: '2'
services:
orderer.diro.umontreal.ca:
container_name: orderer.diro.umontreal.ca
extends:
file: peer-base.yaml
service: orderer-base
volumes:
- ../channel-artifacts/genesis.block:/var/hyperledger/orderer/orderer.genesis.block
- ../crypto-config/ordererOrganizations/diro.umontreal.ca/orderers/orderer.diro.umontreal.ca/msp:/var/hyperledger/orderer/msp
- ../crypto-config/ordererOrganizations/diro.umontreal.ca/orderers/orderer.diro.umontreal.ca/tls/:/var/hyperledger/orderer/tls
- ../crypto-config/ordererOrganizations/diro.umontreal.ca/tlsca:/var/hyperledger/orderer/tlsca
- orderer.diro.umontreal.ca:/var/hyperledger/production/orderer
ports:
- 7050:7050
orderer2.diro.umontreal.ca:
container_name: orderer2.diro.umontreal.ca
extends:
file: peer-base.yaml
service: orderer-base
volumes:
- ../channel-artifacts/genesis.block:/var/hyperledger/orderer/orderer.genesis.block
- ../crypto-config/ordererOrganizations/diro.umontreal.ca/orderers/orderer2.diro.umontreal.ca/msp:/var/hyperledger/orderer/msp
- ../crypto-config/ordererOrganizations/diro.umontreal.ca/orderers/orderer2.diro.umontreal.ca/tls/:/var/hyperledger/orderer/tls
- ../crypto-config/ordererOrganizations/diro.umontreal.ca/tlsca:/var/hyperledger/orderer/tlsca
- orderer.diro.umontreal.ca:/var/hyperledger/production/orderer
ports:
- 8050:8050
And continue like that with a different port and a different path for each orderer. And these are the orderer called and extended in docker-compose-cli.yamlh But still, when I check the log of the first orderer, I get:
Version: 2.0.0
Commit SHA: 0432c3e
Go version: go1.13.4
OS/Arch: linux/amd64
2020-03-11 19:11:39.757 UTC [orderer.common.server] Main -> INFO 020 Beginning to serve requests
2020-03-11 19:11:39.759 UTC [core.comm] ServerHandshake -> ERRO 021 TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=172.22.0.5:57256
2020-03-11 19:11:39.759 UTC [core.comm] ServerHandshake -> ERRO 022 TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=172.22.0.4:33190
2020-03-11 19:11:39.759 UTC [core.comm] ServerHandshake -> ERRO 023 TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=172.22.0.2:54038
2020-03-11 19:11:39.966 UTC [orderer.consensus.etcdraft] apply -> INFO 024 Applied config change to add node 1, current nodes in channel: [1 2 3 4 5] channel=byfn-sys-channel node=1
2020-03-11 19:11:39.967 UTC [orderer.consensus.etcdraft] apply -> INFO 025 Applied config change to add node 2, current nodes in channel: [1 2 3 4 5] channel=byfn-sys-channel node=1
2020-03-11 19:11:39.967 UTC [orderer.consensus.etcdraft] apply -> INFO 026 Applied config change to add node 3, current nodes in channel: [1 2 3 4 5] channel=byfn-sys-channel node=1
2020-03-11 19:11:39.968 UTC [orderer.consensus.etcdraft] apply -> INFO 027 Applied config change to add node 4, current nodes in channel: [1 2 3 4 5] channel=byfn-sys-channel node=1
2020-03-11 19:11:39.968 UTC [orderer.consensus.etcdraft] apply -> INFO 028 Applied config change to add node 5, current nodes in channel: [1 2 3 4 5] channel=byfn-sys-channel node=1
2020-03-11 19:11:40.765 UTC [core.comm] ServerHandshake -> ERRO 029 TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=172.22.0.2:54068
2020-03-11 19:11:41.115 UTC [core.comm] ServerHandshake -> ERRO 02a TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=172.22.0.3:55468
2020-03-11 19:11:42.278 UTC [core.comm] ServerHandshake -> ERRO 02b TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=172.22.0.2:54086
2020-03-11 19:11:44.408 UTC [core.comm] ServerHandshake -> ERRO 02c TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=172.22.0.5:57326
2020-03-11 19:11:44.630 UTC [core.comm] ServerHandshake -> ERRO 02d TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=172.22.0.2:54104
2020-03-11 19:11:47.384 UTC [core.comm] ServerHandshake -> ERRO 02e TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=172.22.0.4:33268
2020-03-11 19:11:47.758 UTC [orderer.consensus.etcdraft] Step -> INFO 02f 1 is starting a new election at term 1 channel=byfn-sys-channel node=1
2020-03-11 19:11:47.758 UTC [orderer.consensus.etcdraft] becomePreCandidate -> INFO 030 1 became pre-candidate at term 1 channel=byfn-sys-channel node=1
2020-03-11 19:11:47.758 UTC [orderer.consensus.etcdraft] poll -> INFO 031 1 received MsgPreVoteResp from 1 at term 1 channel=byfn-sys-channel node=1
2020-03-11 19:11:47.759 UTC [orderer.consensus.etcdraft] campaign -> INFO 032 1 [logterm: 1, index: 5] sent MsgPreVote request to 2 at term 1 channel=byfn-sys-channel node=1
2020-03-11 19:11:47.759 UTC [orderer.consensus.etcdraft] campaign -> INFO 033 1 [logterm: 1, index: 5] sent MsgPreVote request to 3 at term 1 channel=byfn-sys-channel node=1
2020-03-11 19:11:47.759 UTC [orderer.consensus.etcdraft] campaign -> INFO 034 1 [logterm: 1, index: 5] sent MsgPreVote request to 4 at term 1 channel=byfn-sys-channel node=1
2020-03-11 19:11:47.759 UTC [orderer.consensus.etcdraft] campaign -> INFO 035 1 [logterm: 1, index: 5] sent MsgPreVote request to 5 at term 1 channel=byfn-sys-channel node=1
2020-03-11 19:11:47.759 UTC [orderer.consensus.etcdraft] logSendFailure -> ERRO 036 Failed to send StepRequest to 2, because: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"tlsca.diro.umontreal.ca\")" channel=byfn-sys-channel node=1
2020-03-11 19:11:47.760 UTC [orderer.consensus.etcdraft] logSendFailure -> ERRO 037 Failed to send StepRequest to 3, because: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"tlsca.diro.umontreal.ca\")" channel=byfn-sys-channel node=1
2020-03-11 19:11:47.760 UTC [orderer.consensus.etcdraft] logSendFailure -> ERRO 038 Failed to send StepRequest to 4, because: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"tlsca.diro.umontreal.ca\")" channel=byfn-sys-channel node=1
2020-03-11 19:11:49.344 UTC [core.comm] ServerHandshake -> ERRO 039 TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=172.22.0.2:54126
2020-03-11 19:11:50.057 UTC [core.comm] ServerHandshake -> ERRO 03a TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=172.22.0.5:57358
2020-03-11 19:11:51.337 UTC [core.comm] ServerHandshake -> ERRO 03b TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=172.22.0.3:55534
2020-03-11 19:11:55.758 UTC [orderer.consensus.etcdraft] Step -> INFO 03c 1 is starting a new election at term 1 channel=byfn-sys-channel node=1
2020-03-11 19:11:55.758 UTC [orderer.consensus.etcdraft] becomePreCandidate -> INFO 03d 1 became pre-candidate at term 1 channel=byfn-sys-channel node=1
2020-03-11 19:11:55.758 UTC [orderer.consensus.etcdraft] poll -> INFO 03e 1 received MsgPreVoteResp from 1 at term 1 channel=byfn-sys-channel node=1
2020-03-11 19:11:55.758 UTC [orderer.consensus.etcdraft] campaign -> INFO 03f 1 [logterm: 1, index: 5] sent MsgPreVote request to 2 at term 1 channel=byfn-sys-channel node=1
2020-03-11 19:11:55.758 UTC [orderer.consensus.etcdraft] campaign -> INFO 040 1 [logterm: 1, index: 5] sent MsgPreVote request to 3 at term 1 channel=byfn-sys-channel node=1
2020-03-11 19:11:55.758 UTC [orderer.consensus.etcdraft] campaign -> INFO 041 1 [logterm: 1, index: 5] sent MsgPreVote request to 4 at term 1 channel=byfn-sys-channel node=1
2020-03-11 19:11:55.759 UTC [orderer.consensus.etcdraft] campaign -> INFO 042 1 [logterm: 1, index: 5] sent MsgPreVote request to 5 at term 1 channel=byfn-sys-channel node=1
2020-03-11 19:11:56.829 UTC [core.comm] ServerHandshake -> ERRO 043 TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=172.22.0.2:54158
2020-03-11 19:11:57.314 UTC [core.comm] ServerHandshake -> ERRO 044 TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=172.22.0.4:33314
2020-03-11 19:12:01.805 UTC [core.comm] ServerHandshake -> ERRO 045 TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=172.22.0.5:57392
2020-03-11 19:12:03.758 UTC [orderer.consensus.etcdraft] Step -> INFO 046 1 is starting a new election at term 1 channel=byfn-sys-channel node=1
2020-03-11 19:12:03.758 UTC [orderer.consensus.etcdraft] becomePreCandidate -> INFO 047 1 became pre-candidate at term 1 channel=byfn-sys-channel node=1
2020-03-11 19:12:03.758 UTC [orderer.consensus.etcdraft] poll -> INFO 048 1 received MsgPreVoteResp from 1 at term 1 channel=byfn-sys-channel node=1
2020-03-11 19:12:03.758 UTC [orderer.consensus.etcdraft] campaign -> INFO 049 1 [logterm: 1, index: 5] sent MsgPreVote request to 3 at term 1 channel=byfn-sys-channel node=1
2020-03-11 19:12:03.758 UTC [orderer.consensus.etcdraft] campaign -> INFO 04a 1 [logterm: 1, index: 5] sent MsgPreVote request to 4 at term 1 channel=byfn-sys-channel node=1
2020-03-11 19:12:03.758 UTC [orderer.consensus.etcdraft] campaign -> INFO 04b 1 [logterm: 1, index: 5] sent MsgPreVote request to 5 at term 1 channel=byfn-sys-channel node=1
2020-03-11 19:12:03.758 UTC [orderer.consensus.etcdraft] campaign -> INFO 04c 1 [logterm: 1, index: 5] sent MsgPreVote request to 2 at term 1 channel=byfn-sys-channel node=1
2020-03-11 19:12:03.759 UTC [orderer.consensus.etcdraft] logSendFailure -> ERRO 04d Failed to send StepRequest to 5, because: aborted channel=byfn-sys-channel node=1
And yes, I did prune the docker container and volumes.
So this was solved in a very surprising way. It seems that the issue was with docker. I had run docker container prune and docker volume prune regularly, but running docker rmi -f $(docker images -q) made everything work. I suppose it's an equivalent to docker images prunne or something like that. I've not tried it yet. Anyway, after trying that, everything just worked and continued to work even after a reboot.