How to test hitting an internal Network Load Balancer

Question

I have a requirement to use an internal NLB to communicate with other resources inside a VPC. I am able to create an NLB and an AWS Lambda function inside the VPC but for some reason my requests from Lambda hang and time out. I have tried putting the NLB and Lambda function inside separate subnets, in the same subnets, in separate SGs, in the same SGs, but nothing works. I'm not sure how you're supposed to hit internal NLBs to test them. Here is what I have spun up with CloudFormation:

  NLB:                               # The Network Load Balancer
    Type: "AWS::ElasticLoadBalancingV2::LoadBalancer"
    Properties:
      Scheme: "internal"
      Subnets: !Ref NlbSubnetIds     # I'm providing 3 subnets out of the 6 available
      Type: "network"
      LoadBalancerAttributes:
        - Key: access_logs.s3.enabled
          Value: true
        - Key: access_logs.s3.bucket 
          Value: !Ref AccessLogBucketNlb
        - Key: load_balancing.cross_zone.enabled
          Value: true
  NlbTargetGroup:
    Type: "AWS::ElasticLoadBalancingV2::TargetGroup"
    Properties:
        HealthCheckIntervalSeconds: 30
        HealthCheckProtocol: "TCP"
        HealthyThresholdCount: 3
        Matcher: !Ref "AWS::NoValue"
        Port: 80
        Protocol: "TCP"
        TargetType: "ip"
        UnhealthyThresholdCount: 3
        VpcId: !Ref VpcId

  NlbListener:
    Type: "AWS::ElasticLoadBalancingV2::Listener"
    Properties:
      DefaultActions:
      - Type: forward
        TargetGroupArn: !Ref NlbTargetGroup
      LoadBalancerArn: !Ref NLB
      Port: '80'
      Protocol: TCP

From my Lambda function in the same VPC:

import json
import urllib3

def lambda_handler(event, context):
    http = urllib3.PoolManager()
    r = http.request('GET', 'http://<NLB DNS Name>.elb.us-east-1.amazonaws.com',
                     headers={'Content-Type': 'application/json'})
    print(r.read())
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

All of the Inbound / Outbound rules in the ACLs allow for communication as well as the Security Groups so I'm certain it's not that. Any help would be appreciated.

Answer 1

Perhaps try using the VPC network analyzer to see connectivity between resources within your VPC

Blog https://aws.amazon.com/blogs/aws/new-vpc-insights-analyzes-reachability-and-visibility-in-vpcs/

Docs https://docs.aws.amazon.com/vpc/latest/reachability/what-is-reachability-analyzer.html

https://docs.aws.amazon.com/vpc/latest/network-access-analyzer/getting-started.html