Send CloudWatch logs to database

Question

I am attempting to save our Cloudwatch logs in an on-premise Postgres database. I'm currently exporting logs to S3 and save in DynamoDB. My requirement now is to persist it in our DB, using node and AWS js-SDK. I'm not very strong on node and js-SDK, so I'll greatly appreciate any idea.

I tried a simple implementation.

const pools = require('../src/common/db'),
const AWS = require('aws-sdk');

// set the cwl
let cwl = new AWS.CloudWatchLogs({
  region: 'us-east-1',
  accessKeyId: 'ABCD1234',
  secretAccessKey: 'MNBV76543',
  Bucket: 'My_bucket'
});

// Get the events
cwl.getLogEvents({
  logGroupName: 'OurLogGroupname',
  logStreamName: 'specifiedLogstream'
}, (error, success ) =>{
  if(error){
    console.log(error)
  }
  console.log(success)
})

// Try saving to db
let sql = ''
pools.query_db('abc', 'INSERT INTO logging.aws_logs(request_id, duration, billed_duration) VALUES (?,?,?)', function(err, res){
  if(err) return callback(err);
  callback();
})

Answer 1

I would prefer the following way, if you really want to store all messages from Cloudwatch into a database:

• Add a subscription to your Cloudwatch LogGroup

• This subscription can be configured to trigger a Lambda

• The Lambda will have the following logic:

  1. extract the message from the event variable
  2. prepare your SQL statement
  3. connect to database (retry if not possible)
  4. execute the SQL statement (retry if not possible)
  5. done

One good example on how to extract the message of a Cloudwatch Subscription invocation would the one for sending those logs to Opensearch (search the blueprints)