Hyperledger Fabric - Unable to Start Network with Four Kafka and Three Zookeeper Ensemble

Question

I'm trying to setup a network of 2 organizations each having two peers. A 3rd organisation having 2 orderer nodes with kakfa-zookeeper ensemble with 4 kafka and 3 zookeeper nodes.

Below is the relevant part of my crypto-config.yaml file:

OrdererOrgs:
  - Name: Orderer
    Domain: ordererOrg.example.com
    Template:
      Count: 2

Below is the relevant part of my configtx.yaml file:

- &OrdererOrg

    Name: OrdererOrg
    ID: OrdererMSP
    MSPDir: crypto-config/ordererOrganizations/ordererOrg.example.com/msp
    Policies:
        Readers:
            Type: Signature
            Rule: "OR('OrdererMSP.member')"
        Writers:
            Type: Signature
            Rule: "OR('OrdererMSP.member')"
        Admins:
            Type: Signature
            Rule: "OR('OrdererMSP.admin')"
      .................
Orderer: &OrdererDefaults

OrdererType: kafka
Addresses:
    - orderer0.ordererOrg.example.com:7050
    - orderer1.ordererOrg.example.com:7040
BatchTimeout: 2s
BatchSize:
    MaxMessageCount: 10
    AbsoluteMaxBytes: 99 MB
    PreferredMaxBytes: 512 KB

Kafka:
    Brokers:
        - kafka0.ordererOrg.example.com:9092
        - kafka1.ordererOrg.example.com:9092
        - kafka2.ordererOrg.example.com:9092
        - kafka3.ordererOrg.example.com:9092
...............

Below is the relevant part of my Docker base file:

zookeeper:
    image: hyperledger/fabric-zookeeper
    environment:
      - ZOO_SERVERS=server.1=zookeeper0.ordererOrg.example.com:2888:3888 server.2=zookeeper1.ordererOrg.example.com:2888:3888 server.3=zookeeper2.ordererOrg.example.com:2888:3888
    restart: always

kafka:
    image: hyperledger/fabric-kafka
    restart: always
    environment:
      - KAFKA_MESSAGE_MAX_BYTES=103809024 # 99 * 1024 * 1024 B
      - KAFKA_REPLICA_FETCH_MAX_BYTES=103809024 # 99 * 1024 * 1024 B
      - KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE=false
      - KAFKA_MIN_INSYNC_REPLICAS=2
      - KAFKA_DEFAULT_REPLICATION_FACTOR=3
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper0.ordererOrg.example.com:2181,zookeeper1.ordererOrg.example.com:2181,zookeeper2.ordererOrg.example.com:2181

Below is the relevant part of my Docker Compose file:

zookeeper0.ordererOrg. example.com:
    container_name: zookeeper0.ordererOrg.example.com
    extends:
      file: base/kafka-base.yaml
      service: zookeeper
    environment:
      - ZOO_MY_ID=1
    ports:
      - '2181:2181'
      - '2888:2888'
      - '3888:3888'
    networks:
      - byfn

  kafka0.ordererOrg.example.com:
    container_name: kafka0.ordererOrgvodworks.example.com
    extends:
      file: base/kafka-base.yaml
      service: kafka
    depends_on:
      - zookeeper0.ordererOrg.example.com
      - zookeeper1.ordererOrg.example.com
      - zookeeper2.ordererOrg.example.com
    environment:
      - KAFKA_BROKER_ID=0
    ports:
      - '9092:9092'
      - '9093:9093'
    networks:
      - byfn
----------------------- 
Note: The same structure is being followed for:
    - zookeeper1.ordererOrg. example.com
    - zookeeper2.ordererOrg. example.com
And
    - kafka1.ordererOrg.example.com
    - kafka2.ordererOrg.example.com
    - kafka3.ordererOrg.example.com

When I run the network start command I get the following error messages:

✖ Starting business network definition. This may take a minute... Error: Error trying to start business network. Error: No valid responses from any peers. Response from attempted peer comms was an error: Error: REQUEST_TIMEOUT

And when I run the same network start command again, I get the following:

✖ Starting business network definition. This may take a minute... Error: Error trying to start business network. Error: No valid responses from any peers. Response from attempted peer comms was an error: Error: chaincode registration failed: timeout expired while starting chaincode tt_poc:0.0.1 for transaction

And images files are also not being created against the chaincode (BNA file) as you can see the ccenv containers and orderer logs in the image below:

And I get the following logs as well on console after peer channel create command, though channel gets created successfully:

2019-03-25 15:20:34.567 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and rderer connections initialized 
2019-03-25 15:20:34.956 UTC [cli.common] readBlock -> INFO 002 Got status: &{SERVICE_UNAVAILABLE}

I tried to provide maximum information but still please let me know if you require logs of any other container as well. Thanks for your time.

Answer 1

I finally able to resolve this issue. There was nothing wrong with these YAML configurations. The issue was with the docker configurations that It was lacking in resources and the strange thing is that I didn't get any resources related error in any container logs file. So, I just increased CPUs and Memory settings in the docker advanced configurations like below:

And after these configurational changes, my network started successfully and working properly.

Thanks to my colleague @Rafiq who help me in sorting out this issue.