AWS Lambda function that sends data to clickhouse suddenly stopped working

Question

I have a Lambda function that sends data to Clickhouse every 5 seconds (Kinesis Trigger). This was working fine for the last month until yesterday it stopped without any error.

The lambda started to time out (6 seconds), I raised it to one minute, but it's still timing out. I checked the lambda and I found out that code is blocked at the insert statement level, and it's timing out trying to connect to database.

I tried to connect using another external script (using same VPC config ) and it's working just fine.

I tried to create another lambda and I put inside just the client setup and a random query to database, it timed out too!

is there any way I can debug this ? specially that there's no error in database logs as the request is not being established at all.

this is my CDK code to deploy lambda

const test = new NodejsFunction(this, 'commonTest', {
  environment: {

  },
  vpc,
  vpcSubnets: {
    subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS,
  },
  securityGroups: [securityGroup],
  memorySize: 512,
  timeout: Duration.seconds(60),
  runtime: lambda.Runtime.NODEJS_18_X,
  entry: path.join(__dirname, `../resources/test/test.ts`),
  handler: 'handler'
});

this is the code inside lambda :

const clickhouse = new ClickHouse({
    url,
    port: 8123,
    debug: false,
    basicAuth: {
        username: "",
        password: "",
    },
    isUseGzip: false,
    trimQuery: false,
    usePost: false,
    format: "json", // "json" || "csv" || "tsv"
    config: {
        database: 'default',
    },

});
const a = await clickhouse.query(" select * from Record order by createdAt DESC LIMIT 1").toPromise();

Answer 1

The problem were that someone deleted the outbound rules in the security group that we we're using, that's why the http request hangs. (that someone is not me)