I have a network with 3 ordering nodes running on a Kubernetes cluster. I am using NFS for persistence storage. The Kubernetes cluster is running on bare metal machines. I onboarded a new organization into the network, added that organization into the constortium. As soon as I execute d the command peer channel create, the orderer started throwing the below mentioned error:
2021-01-31 11:09:04.412 UTC [orderer.consensus.etcdraft] createOrReadWAL -> INFO 089 No WAL data found, creating new WAL at path '/var/hyperledger/production/orderer/etcdraft/wal/mvp1x-channel' channel=mvp1x-channel node=1
2021-01-31 11:09:04.549 UTC [orderer.consensus.etcdraft] createOrReadWAL -> WARN 08a failed to create a temporary WAL directory channel=mvp1x-channel node=1 tmp-dir-path=/var/hyperledger/production/orderer/etcdraft/wal/mvp1x-channel.tmp dir-path=/var/hyperledger/production/orderer/etcdraft/wal/mvp1x-channel error="expected \"/var/hyperledger/production/orderer/etcdraft/wal/mvp1x-channel.tmp\" to be empty, got [\".nfs000000000982c7c700005443\"]"
2021-01-31 11:09:04.549 UTC [orderer.commmon.multichannel] newChainSupport -> PANI 08b [channel: mvp1x-channel] Error creating consenter: failed to restore persisted raft data: failed to create or read WAL: failed to initialize WAL:expected "/var/hyperledger/production/orderer/etcdraft/wal/mvp1x-channel.tmp" to be empty, got [".nfs000000000982c7c700005443"]
panic: [channel: mvp1x-channel] Error creating consenter: failed to restore persisted raft data: failed to create or read WAL: failed to initialize WAL:expected "/var/hyperledger/production/orderer/etcdraft/wal/mvp1x-channel.tmp" to be empty, got [".nfs000000000982c7c700005443"]
After the automatic restart, it started throwing an error:
2021-02-01 04:55:31.392 UTC [orderer.commmon.multichannel] newChain -> PANI 1519 Error creating chain support: error creating consenter forchannel: mvp1x-channel: failed to restore persisted raft data: failed to create or read WAL: failed to open WAL: fileutil: file already locked
panic: Error creating chain support: error creating consenter for channel: mvp1x-channel: failed to restore persisted raft data: failed to create or read WAL: failed to open WAL: fileutil: file already locked
goroutine 74 [running]:
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc00014fe40, 0x0, 0x0, 0x0)
/go/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/zapcore/entry.go:230 +0x545
go.uber.org/zap.(*SugaredLogger).log(0xc0000101b0, 0xc000557004, 0x101b08c, 0x20, 0xc001ca7588, 0x1, 0x1, 0x0, 0x0, 0x0)
/go/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:234 +0x100
go.uber.org/zap.(*SugaredLogger).Panicf(...)
/go/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/sugar.go:159
github.com/hyperledger/fabric/common/flogging.(*FabricLogger).Panicf(...)
/go/src/github.com/hyperledger/fabric/common/flogging/zap.go:74
github.com/hyperledger/fabric/orderer/common/multichannel.(*Registrar).newChain(0xc000146a00, 0xc000452b40)
/go/src/github.com/hyperledger/fabric/orderer/common/multichannel/registrar.go:369 +0x251
github.com/hyperledger/fabric/orderer/common/multichannel.(*BlockWriter).WriteConfigBlock(0xc00056f540, 0xc00021cb40, 0xc000573ca0, 0x9, 0x9)
/go/src/github.com/hyperledger/fabric/orderer/common/multichannel/blockwriter.go:118 +0x357
github.com/hyperledger/fabric/orderer/consensus/etcdraft.(*Chain).writeConfigBlock(0xc001d7a000, 0xc00021cb40, 0x12)
/go/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain.go:1226 +0x28f
github.com/hyperledger/fabric/orderer/consensus/etcdraft.(*Chain).writeBlock(0xc001d7a000, 0xc00021cb40, 0x12)
/go/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain.go:805 +0x1a9
github.com/hyperledger/fabric/orderer/consensus/etcdraft.(*Chain).apply(0xc001d7a000, 0xc00059db10, 0x1, 0x1)
/go/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain.go:990 +0x24a
github.com/hyperledger/fabric/orderer/consensus/etcdraft.(*Chain).run(0xc001d7a000)
/go/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain.go:713 +0x95c
created by github.com/hyperledger/fabric/orderer/consensus/etcdraft.(*Chain).Start
/go/src/github.com/hyperledger/fabric/orderer/consensus/etcdraft/chain.go:360 +0x230
I believe the first error is thrown because the NFS client creates a file with pattern .nfsxxxxxxx. But I don't understand the second error and I am looking for a way to resolve it.
I am also looking for the other best storage mechanism alternative to the NFS server for a bare-metal cluster.
Any suggestion/feedback would be appreciated
Update: The other 2 orderers are running fine.
This bug is fixed in https://github.com/hyperledger/fabric/pull/2934, you can look at the issue https://github.com/hyperledger/fabric/issues/2931