Search code examples
hyperledger-fabrichyperledgerraft

How to fix "Failed creating puller config from bootstrap block: unable to decode TLS certificate PEM" in HLF with Raft Cluster

I am trying to set up a Hyperledger Fabric network with the orderer as Raft (3 nodes in the cluster). I am using Kubernetes with Helm in Cloud.

Everything works fine when it is a single node Raft set up. However, for a multi-node setup, I am getting the following error.

[orderer.common.cluster] createReplicator -> PANI 3f4 Failed creating puller config from bootstrap block: unable to decode TLS certificate PEM: 
panic: Failed creating puller config from bootstrap block: unable to decode TLS certificate PEM: 

goroutine 15 [running]:
github.com/hyperledger/fabric/vendor/go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0000f1ce0, 0x0, 0x0, 0x0)
    /opt/gopath/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/zapcore/entry.go:229 +0x515
github.com/hyperledger/fabric/vendor/go.uber.org/zap.(*SugaredLogger).log(0xc000140158, 0x4, 0x1047e13, 0x36, 0xc000289c50, 0x1, 0x1, 0x0, 0x0, 0x0)
    /opt/gopath/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:234 +0xf6
github.com/hyperledger/fabric/vendor/go.uber.org/zap.(*SugaredLogger).Panicf(0xc000140158, 0x1047e13, 0x36, 0xc000289c50, 0x1, 0x1)
    /opt/gopath/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:159 +0x79
github.com/hyperledger/fabric/common/flogging.(*FabricLogger).Panicf(0xc000140160, 0x1047e13, 0x36, 0xc000289c50, 0x1, 0x1)
    /opt/gopath/src/github.com/hyperledger/fabric/common/flogging/zap.go:74 +0x60
github.com/hyperledger/fabric/orderer/common/server.(*replicationInitiator).createReplicator(0xc0001c8ba0, 0xc0003ce340, 0xc00031a9a0, 0xb)
    /opt/gopath/src/github.com/hyperledger/fabric/orderer/common/server/onboarding.go:60 +0x30c
github.com/hyperledger/fabric/orderer/common/server.(*replicationInitiator).ReplicateChains(0xc0001c8ba0, 0xc0003ce340, 0xc00031a000, 0x1, 0x1, 0x0, 0x0, 0x0)
    /opt/gopath/src/github.com/hyperledger/fabric/orderer/common/server/onboarding.go:117 +0x1ca
github.com/hyperledger/fabric/orderer/common/server.(*inactiveChainReplicator).replicateDisabledChains(0xc0003701e0)
    /opt/gopath/src/github.com/hyperledger/fabric/orderer/common/server/onboarding.go:224 +0x1f5
github.com/hyperledger/fabric/orderer/common/server.(*inactiveChainReplicator).run(0xc0003701e0)
    /opt/gopath/src/github.com/hyperledger/fabric/orderer/common/server/onboarding.go:202 +0x42
created by github.com/hyperledger/fabric/orderer/common/server.initializeEtcdraftConsenter
    /opt/gopath/src/github.com/hyperledger/fabric/orderer/common/server/main.go:637 +0x3fc

Other configs as follows

configtx.yaml

MultiNodeEtcdRaft:
        <<: *ChannelDefaults
        Capabilities:
            <<: *ChannelCapabilities
        Orderer:
            <<: *OrdererDefaults
            OrdererType: etcdraft
            EtcdRaft:
                Consenters:
                - Host: orderer.example.com
                  Port: 7050
                  ClientTLSCert: crypto-config/ordererOrganizations/example.com/orderers/orderer.example.com/tls/server.crt
                  ServerTLSCert: crypto-config/ordererOrganizations/example.com/orderers/orderer.example.com/tls/server.crt
                Consenters:
                - Host: orderer2.example.com
                  Port: 7050
                  ClientTLSCert: crypto-config/ordererOrganizations/example.com/orderers/orderer2.example.com/tls/server.crt
                  ServerTLSCert: crypto-config/ordererOrganizations/example.com/orderers/orderer2.example.com/tls/server.crt
                Consenters:
                - Host: orderer3.example.com
                  Port: 7050
                  ClientTLSCert: crypto-config/ordererOrganizations/example.com/orderers/orderer3.example.com/tls/server.crt
                  ServerTLSCert: crypto-config/ordererOrganizations/example.com/orderers/orderer3.example.com/tls/server.crt
            Addresses:
                - myorg-orderer-hlf-ord:7050
                - myorg-orderer2-hlf-ord:7050
                - myorg-orderer3-hlf-ord:7050
            Organizations:
            - *OrdererOrg
            Capabilities:
                <<: *OrdererCapabilities
        Application:
            <<: *ApplicationDefaults
            Organizations:
            - <<: *OrdererOrg
        Consortiums:
            SampleConsortium:
                Organizations:
                - *myorg
Solution

  • I found the solution to this problem.

    I had to fix two issues inorder to get this working.

    1. I was missing a few environment variables for TLS in my helm setup for orderer.
    ORDERER_GENERAL_CLUSTER_CLIENTCERTIFICATE: "/var/hyperledger/tls/server/pair/tls.crt"
ORDERER_GENERAL_CLUSTER_CLIENTPRIVATEKEY: "/var/hyperledger/tls/server/pair/tls.key"
ORDERER_GENERAL_CLUSTER_ROOTCAS: "/var/hyperledger/tls/server/cert/cacert.pem"
    1. Syntax error in configtx.yaml YAML config. I was repeating Consenters key multiple times.