The running instances were fine today in the afternoon, but suddenly this error started to pop out. It presumes to call the function and write to the ledger successfully but rejects with this:
Too many requests for /protos.Deliver, exceeding concurrency limit (2500)
peer0.org1.xxx.com-hold-em_1-c068ec292dc2ab380801d4f31ca83c6b86104d5adf00d7ee78b940a7a7381c02] func2 -> INFO 1402c 2020-06-16T05:43:20.627Z info [c-api:lib/handler.js] [xxx-dc987af0] Calling chaincode Invoke() succeeded. Sending COMPLETED message back to peer
2020-06-16 05:43:20.628 UTC [peer.chaincode.dev-peer0.org1.xxx.com-xxx_1-171cd178bf0f55cbedf8e78207e38a4f25f12a26f8ae99732d2b7a0a64ffb656] func2 -> INFO 1402d 2020-06-16T05:43:20.628Z info [c-api:lib/handler.js] [xxx-dc987af0] Calling chaincode Invoke() succeeded. Sending COMPLETED message back to peer
2020-06-16 05:43:20.628 UTC [endorser] callChaincode -> INFO 1402e finished chaincode: xxxduration: 39ms channel=xxx txID=dc987af0
2020-06-16 05:43:20.629 UTC [comm.grpc.server] 1 -> INFO 1402f unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=10.23.0.254:44788 grpc.peer_subject="CN=fabric-common" grpc.code=OK grpc.call_duration=41.702366ms
2020-06-16 05:43:20.647 UTC [nodeCmd] func1 -> ERRO 14030 Too many requests for /protos.Deliver, exceeding concurrency limit (2500)
2020-06-16 05:43:20.647 UTC [comm.grpc.server] 1 -> INFO 14031 streaming call completed grpc.service=protos.Deliver grpc.method=DeliverFiltered grpc.peer_address=10.23.0.254:44792 grpc.peer_subject="CN=fabric-common" error="too many requests for /protos.Deliver, exceeding concurrency limit (2500)" grpc.code=Unknown grpc.call_duration=124.448µs
2020-06-16 05:43:22.676 UTC [gossip.privdata] StoreBlock -> INFO 14032 [xxx] Received block [2657] from buffer
2020-06-16 05:43:22.681 UTC [committer.txvalidator] Validate -> INFO 14033 [xxx] Validated block [2657] in 4ms
2020-06-16 05:43:22.681 UTC [gossip.privdata] prepareBlockPvtdata -> INFO 14034 Successfully fetched all eligible collection private write sets for block [2657] channel=xxx
I don't know what could be happening. If there is a fix for this on a newer version I would like to know how can I upgrade Fabric to the latest version.
The number of concurrent requests to the peer services has been capped by default in Fabric v2.1.0+ to prevent poorly programmed or malicious clients from DoS-ing the peers. You can always remove this restriction, or increase it by modifying these values in core.yaml. To remove the restriction, set the limit to 0, or increase it to a value that makes sense for your environment.
Generally speaking, if these messages are occurring, it is because the client application is misbehaving and not appropriately releasing resources. Once a client is done with a particular call, it's important to close and cleanup the associated network resources. A common source of leaks for connections to the peer is requesting event streams, reading one event, and then opening a new one without closing the previous.