Search code examples
javasslssl-certificatehyperledger-fabrichyperledger-chaincode

PKIX path building failed: unable to find valid certification path to requested target on chaincode commit on Hyperledger Fabric production network


During java chaincode package commit i got this error on one of two peers:

Shutting down the chaincode stream.UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
Channel Pipeline:  
Caused by: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
  at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.handshakeException(ReferenceCountedOpenSslEngine.java:1927)
  at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.wrap(ReferenceCountedOpenSslEngine.java:848)
  at java.base/javax.net.ssl.SSLEngine.wrap(SSLEngine.java:564)
  at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:1131)

Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target  

I'm trying to setup a production network using cryptogen tool for certs and folder stuctures using crypto-config.yaml below

OrdererOrgs:
  - Name: OrdererOrg
    Domain: crpt.com
    EnableNodeOUs: true
    Specs:
      - Hostname: orderer
        SANS:
          - orderer.crpt.com
          - <serverip>
      - Hostname: orderer1
        SANS:
          - orderer1.crpt.com
          - <serverip>
      - Hostname: orderer2
        SANS:
          - orderer2.crpt.com
          - <serverip>
PeerOrgs:
  - Name: CrpOrg
    Domain: crpt.com
    EnableNodeOUs: true    
    Specs:
      - Hostname: peer
        SANS:
          - peer.crpt.com
          - <serverip>
    Template:
      Count: 0
    Users:
      Count: 1

  - Name: CrpetOrg
    Domain: Crpet.it
    EnableNodeOUs: true    
    Specs:
      - Hostname: peer
        SANS:
          - peer.crpet.it
          - <serverip>
    Template:
      Count: 0
    Users:
      Count: 1

This is my docker compose file

networks:
  prod:
    name: prod-fabric-network
services: 
  peer.crpt.com:
    image: hyperledger/fabric-peer:latest
    container_name: peer.crpt.com
    environment:
      - FABRIC_LOGGING_SPEC=INFO
      - CORE_GENERAL_LOGLEVEL=INFO
      - CORE_LOGGING_LEVEL=DEBUG
      - CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock
      - CORE_PEER_TLS_ENABLED=true
      - CORE_PEER_TLS_CERT_FILE=/etc/hyperledger/fabric/tls/server.crt
      - CORE_PEER_TLS_KEY_FILE=/etc/hyperledger/fabric/tls/server.key
      - CORE_PEER_TLS_ROOTCERT_FILE=/etc/hyperledger/fabric/tls/ca.crt
      - CORE_PEER_TLS_CLIENTAUTHREQUIRED=true
      - CORE_PEER_ID=peer.crpt.com
      - CORE_PEER_ADDRESS=peer.crpt.com:7051
      - CORE_PEER_LISTENADDRESS=0.0.0.0:7051
      - CORE_PEER_CHAINCODEADDRESS=peer.crpt.com:7052
      - CORE_PEER_CHAINCODELISTENADDRESS=0.0.0.0:7052
      - CORE_PEER_GOSSIP_BOOTSTRAP=peer.crpt.com:7051
      - CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer.crpt.com:7051
      - CORE_PEER_LOCALMSPID=CrpOrgMSP
      - CORE_PEER_PROFILE_ENABLED=true
    volumes:
      - /root/network/organizations/peerOrganizations/crpt.com/peers/peer.crpt.com/msp:/etc/hyperledger/fabric/msp
      - /root/network/organizations/peerOrganizations/crpt.com/peers/peer.crpt.com/tls:/etc/hyperledger/fabric/tls
      - /root/network/organizations/peerOrganizations/crpt.com/peers/peer.crpt.com:/etc/hyperledger/production
      - /var/run/docker.sock:/host/var/run/docker.sock
    ports:
      - "7051:7051"
      - "7052:7052"
    networks:
      - prod 
    depends_on:
      - couchdb.peer.crpt.com

  peer.crpet.it:
    image: hyperledger/fabric-peer:latest
    container_name: peer.crpet.it
    environment:
      - FABRIC_LOGGING_SPEC=INFO
      - CORE_GENERAL_LOGLEVEL=INFO
      - CORE_LOGGING_LEVEL=DEBUG
      - CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock
      - CORE_PEER_TLS_ENABLED=true
      - CORE_PEER_TLS_CERT_FILE=/etc/hyperledger/fabric/tls/server.crt
      - CORE_PEER_TLS_KEY_FILE=/etc/hyperledger/fabric/tls/server.key
      - CORE_PEER_TLS_ROOTCERT_FILE=/etc/hyperledger/fabric/tls/ca.crt
      - CORE_PEER_TLS_CLIENTAUTHREQUIRED=true
      - CORE_PEER_ID=peer.crpet.it
      - CORE_PEER_ADDRESS=peer.crpet.it:7051
      - CORE_PEER_LISTENADDRESS=0.0.0.0:7051
      - CORE_PEER_CHAINCODEADDRESS=peer.crpet.it:7052
      - CORE_PEER_CHAINCODELISTENADDRESS=0.0.0.0:7052
      - CORE_PEER_GOSSIP_BOOTSTRAP=peer.crpet.it:7051
      - CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer.crpet.it:7054
      - CORE_PEER_LOCALMSPID=CrpetOrgMSP
      - CORE_PEER_PROFILE_ENABLED=true
    volumes:
      - /root/network/organizations/peerOrganizations/Crpet.it/peers/peer.crpet.it/msp:/etc/hyperledger/fabric/msp
      - /root/network/organizations/peerOrganizations/Crpet.it/peers/peer.crpet.it/tls:/etc/hyperledger/fabric/tls
      - /root/network/organizations/peerOrganizations/Crpet.it/peers/peer.crpet.it:/etc/hyperledger/production
      - /var/run/docker.sock:/host/var/run/docker.sock
    ports:
      - "7054:7051"
      - "7055:7052"
    networks:
      - prod
    depends_on:
      - couchdb.peer.crpet.it
 
  orderer.crpt.com:
    image: hyperledger/fabric-orderer:3.0
    container_name: orderer.crpt.com
    environment:
      - FABRIC_LOGGING_SPEC=INFO 
      - ORDERER_GENERAL_LOGLEVEL=INFO
    volumes:
      - /root/network/organizations/ordererOrganizations/crpt.com/orderers/orderer.crpt.com:/etc/hyperledger/fabric
    ports:
      - "7050:7050"
      - "8443:8443"
      - "7053:7053"
    networks:
      - prod  

  orderer1.crpt.com:
    image: hyperledger/fabric-orderer:3.0
    container_name: orderer1.crpt.com
    environment:
      - FABRIC_LOGGING_SPEC=INFO
      - ORDERER_GENERAL_LOGLEVEL=INFO
    volumes:
      - /root/network/organizations/ordererOrganizations/crpt.com/orderers/orderer1.crpt.com:/etc/hyperledger/fabric 
    ports:
      - "8050:7050"
      - "9443:8443"
      - "8053:7053"
    networks:
      - prod

  orderer2.crpt.com:
    image: hyperledger/fabric-orderer:3.0
    container_name: orderer2.crpt.com
    environment:
      - FABRIC_LOGGING_SPEC=INFO
      - ORDERER_GENERAL_LOGLEVEL=INFO
    volumes:
      - /root/network/organizations/ordererOrganizations/crpt.com/orderers/orderer2.crpt.com:/etc/hyperledger/fabric
    ports:
      - "8051:7050"
      - "10443:8443"
      - "9053:7053"
    networks:
      - prod

Then, after run compose start, i followed these steps:

  1. generate genesis channel block file
  2. add orderer nodes to channel
  3. add peer nodes to channel
  4. deploy java chaincode to peers

The channel is generated by this configtx.yaml below

Organizations:
    - &OrdererOrg
        Name: OrdererOrg
        SkipAsForeign: false
        ID: OrdererOrgMSP
        MSPDir: /root/network/organizations/ordererOrganizations/crpt.com/msp
        Policies: &OrdererOrgPolicies
            Readers:
                Type: Signature
                Rule: "OR('OrdererOrgMSP.member')"
            Writers:
                Type: Signature
                Rule: "OR('OrdererOrgMSP.member')"
            Admins:
                Type: Signature
                Rule: "OR('OrdererOrgMSP.admin')"
        OrdererEndpoints:
           - orderer.crpt.com:7050
           - orderer1.crpt.com:8050
           - orderer2.crpt.com:8051
    - &CrpOrg
        Name: CrpOrg
        SkipAsForeign: false
        ID: CrpOrgMSP
        MSPDir: /root/network/organizations/peerOrganizations/crpt.com/msp
        Policies: &CrpOrgPolicies
            Readers:
                Type: Signature
                Rule: "OR('CrpOrgMSP.admin', 'CrpOrgMSP.member', 'CrpOrgMSP.peer', 'CrpOrgMSP.client')"
            Writers:
                Type: Signature
                Rule: "OR('CrpOrgMSP.admin', 'CrpOrgMSP.client')"
            Admins:
                Type: Signature
                Rule: "OR('CrpOrgMSP.admin')"
            Endorsement:
                Type: Signature
                Rule: "OR('CrpOrgMSP.peer')"
        AnchorPeers:
          - Host: peer.crpt.com
            Port: 7051
    - &CrpetOrg
        Name: CrpetOrg
        SkipAsForeign: false
        ID: CrpetOrgMSP
        MSPDir: /root/network/organizations/peerOrganizations/Crpet.it/msp
        Policies: &CrpetOrgPolicies
            Readers:
                Type: Signature
                Rule: "OR('CrpetOrgMSP.admin', 'CrpetOrgMSP.member', 'CrpetOrgMSP.peer', 'CrpetOrgMSP.client')"
            Writers:
                Type: Signature
                Rule: "OR('CrpetOrgMSP.admin', 'CrpetOrgMSP.client')"
            Admins:
                Type: Signature
                Rule: "OR('CrpetOrgMSP.admin')"
            Endorsement:
                Type: Signature
                Rule: "OR('CrpetOrgMSP.peer')"
        AnchorPeers:
          - Host: peer.crpet.it
            Port: 7051 

Orderer: &OrdererDefaults
    OrdererType: etcdraft
    Addresses:
         - orderer.crpt.com:7050
         - orderer1.crpt.com:8050
         - orderer2.crpt.com:8051
    MaxChannels: 0
    EtcdRaft:
        Consenters:
            - Host: orderer.crpt.com
              Port: 7050
              ClientTLSCert: /root/network/organizations/ordererOrganizations/crpt.com/orderers/orderer.crpt.com/tls/server.crt
              ServerTLSCert: /root/network/organizations/ordererOrganizations/crpt.com/orderers/orderer.crpt.com/tls/server.crt
            - Host: orderer1.crpt.com
              Port: 8050
              ClientTLSCert: /root/network/organizations/ordererOrganizations/crpt.com/orderers/orderer1.crpt.com/tls/server.crt
              ServerTLSCert: /root/network/organizations/ordererOrganizations/crpt.com/orderers/orderer1.crpt.com/tls/server.crt
            - Host: orderer2.crpt.com
              Port: 8051
              ClientTLSCert: /root/network/organizations/ordererOrganizations/crpt.com/orderers/orderer2.crpt.com/tls/server.crt
              ServerTLSCert: /root/network/organizations/ordererOrganizations/crpt.com/orderers/orderer2.crpt.com/tls/server.crt
        Options:
            TickInterval: 500ms
            ElectionTick: 10
            HeartbeatTick: 1
            MaxInflightBlocks: 5
            SnapshotIntervalSize: 16 MB
Profiles:
    CrpChannel:
        <<: *ChannelDefaults
        Orderer:
            <<: *OrdererDefaults
            Organizations:
              - <<: *OrdererOrg
            Capabilities: *OrdererCapabilities
        Application:
            <<: *ApplicationDefaults
            Organizations:
               - <<: *CrpOrg
                 Policies:
                   <<: *CrpOrgPolicies
                   Admins:
                     Type: Signature
                     Rule: "OR('CrpOrgMSP.member')" 
               - <<: *CrpetOrg
                 Policies:
                   <<: *CrpetOrgPolicies
                   Admins:
                     Type: Signature
                     Rule: "OR('CrpetOrgMSP.member')"

In chaincode deploy using the following command i got an error on only one of two peers:

peer lifecycle chaincode commit -o orderer.crpt.com:7050 --channelID crpchannel --name smart_contract_1 --version 1.0 --sequence 1 --tls true --cafile /root/network/organizations/ordererOrganizations/crpt.com/orderers/orderer.crpt.com/tls/ca.crt --peerAddresses peer.crpt.com:7051 --tlsRootCertFiles /root/network/organizations/peerOrganizations/crpt.com/peers/peer.crpt.com/tls/ca.crt --peerAddresses peer.crpet.it:7054 --tlsRootCertFiles /root/network/organizations/peerOrganizations/Crpet.it/peers/peer.crpet.it/tls/ca.crt --signature-policy "OR('CrpOrgMSP.peer', 'CrpetOrgMSP.peer')"

The first peer deploy (peer.crpt.com) logs seems goes well with some TLS errors

INFO [lifecycle] QueryChaincodeDefinition -> Successfully queried chaincode name 'smart_contract_1' with definition {sequence: 1, endorsement info: (version: '1.0', plugin: 'escc', init required: false), validation info: (plugin: 'vscc', policy: '0a34120c120a080112020800120208011a0f120d0a094372704f72674d535010031a1312110a0d436f72697065744f72674d53501003'), collections: ()},
INFO [lifecycle] QueryOrgApprovals -> org CrpetOrgMSP's mismatch items are []
INFO [lifecycle] QueryOrgApprovals -> org CrpOrgMSP's mismatch items are []
INFO [endorser] callChaincode -> finished chaincode: _lifecycle duration: 0ms channel=crpchannel txID=b8424f73
INFO [comm.grpc.server] 1 -> unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=172.28.0.1:53386 grpc.peer_subject="CN=peer.crpt.com,L=San Francisco,ST=California,C=US" grpc.code=OK grpc.call_duration=1.498406ms
INFO [grpc] Infof -> [transport] [server-transport 0xc003395520] Closing: EOF
INFO [grpc] Infof -> [transport] [server-transport 0xc003395520] loopyWriter exiting with error: transport closed by client
INFO [grpc] Infof -> [transport] [server-transport 0xc003395380] Closing: EOF
INFO [grpc] Infof -> [transport] [server-transport 0xc003395380] loopyWriter exiting with error: transport closed by client
ERRO [core.comm] ServerHandshake -> Server TLS handshake failed in 135.306152ms with error remote error: tls: internal error server=ChaincodeServer remoteaddress=172.28.0.1:55180
INFO [grpc] AddTraceEvent -> [core] [Server #2]grpc: Server.Serve failed to create ServerTransport: connection error: desc = "ServerHandshake(\"172.28.0.1:55180\") failed: remote error: tls: internal error"

Instead, the second peer (peer.crpet.it) fails chaincode execution:

WARN [lifecycle] Work -> could not launch chaincode 'smart_contract_1:d8a346da3120ddfddcc616633369aaa81649d8d7af6f3c35cdc413648e14a461': chaincode registration failed: container exited with 0

These are end logs of fail execution container:

sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Someone can help me to understand what i'm doing wrong ?


Solution

  • Solved! It was bad peer address:port mapping, changing these environment variables all chaincode peer containers goes up well:

    • CORE_PEER_CHAINCODEADDRESS=peer.coripet.it:7055
    • CORE_PEER_CHAINCODELISTENADDRESS=0.0.0.0:7052

    Peer peer.coripet.it has chaincode port (by default 7052) mapped as 7055 outside docker container.