Dynamic logs not coming on aws cloudwatch

Question

I am trying to integrate AWS Cloud watch with my application but facing multiple issues. I have tried multiple things but couldn't get success. Also, I don't see any good documentation for the aws cloudwatch. Any help is appreciated

• Unable to create a log stream
• Dynamic logs are not showing

Gemfile

gem 'aws-sdk'
gem 'lograge'

config/initializers/cloudwatch_log.rb

begin
    @cloudwatch_client = Aws::CloudWatchLogs::Client.new

    # Define the log group and log stream names
    @log_group_name = ENV.fetch('APPLICATION_LOG_GROUP_NAME', '/home/app/insight')
    @log_stream_name = "#{Time.now.strftime('%Y-%m-%d')}/$[LATEST]/#{Time.now.to_i}"
    Rails.logger.info "log_group_name: #{@log_group_name} and log_stream_name: #{@log_stream_name} in #{Rails.env} environment...!!!"
    
    @cloudwatch_client.create_log_stream({
        log_group_name: @log_group_name,
        log_stream_name: @log_stream_name
    })

    Rails.application.config.logger.extend(ActiveSupport::Logger.broadcast(Logger.new($stdout)))

    Rails.application.config.logger.extend(ActiveSupport::Logger.broadcast(@cloudwatch_client))
rescue Aws::Errors::MissingRegionError, Aws::Sigv4::Errors::MissingCredentialsError => e
    Rails.logger.info "be_admin: AWS credentials not found for CloudWatchLog...!!!"
end

production.rb

require "active_support/core_ext/integer/time"

Rails.application.configure do
  config.cache_classes = true

  config.eager_load = true

  config.consider_all_requests_local       = false
  config.action_controller.perform_caching = true

  config.public_file_server.enabled = true

  config.assets.compile = false

  config.flipper.preload = false

  config.active_storage.service = :local

  config.action_mailer.perform_caching = false

  config.active_support.report_deprecations = true

  logger           = ActiveSupport::Logger.new($stdout)
  logger.formatter = config.log_formatter
  config.logger = ActiveSupport::TaggedLogging.new(logger)

  # Set log level from LOG_LEVEL
  default_log_level = Rails.env.development? ? :debug : :info
  config_log_level  = ENV.fetch("LOG_LEVEL", default_log_level).to_s.downcase.to_sym
  config.log_level = if %i[debug info warn error fatal].include?(config_log_level)
                       config_log_level
                     else
                       default_log_level
                     end

  config.lograge.enabled = !Rails.env.development?

  config.lograge.formatter = Lograge::Formatters::Raw.new

  config.logger = Logger.new($stdout).tap do |logger|
    logger.formatter = config.log_formatter
    config.logger    = ActiveSupport::TaggedLogging.new(logger)
  end

  config.active_record.dump_schema_after_migration = false
end

Update 1

If I add this line it creates a log stream but still dynamic logs are not coming

@cloudwatch_client.put_log_events({log_group_name: @log_group_name, log_stream_name: @log_stream_name, log_events: [{timestamp: Time.now.to_i * 1000, message: "logger initialted"}]})

Answer 1

To anyone struggling to integrate aws cloudwatch with the Rails application. Here is the solution

Gemfile

gem 'aws-sdk'

Create a file inside lib cloudwatch_logger.rb. We are trying to override the Logger class with the cloud watch logger so to send dynamic logs

require 'aws-sdk-cloudwatchlogs'
class CloudwatchLogger < Logger
  def initialize(log_group_name, log_stream_name)
    super(STDOUT)
    @log_group_name = log_group_name
    @log_stream_name = log_stream_name
    @cloudwatch_logs = Aws::CloudWatchLogs::Client.new
    create_log_stream_if_not_exists
  end

  def add(severity, message = nil, progname = nil, &block)
    severity ||= UNKNOWN
    message ||= (block && block.call) || progname
    return true if severity < level

    message = format_message(severity, Time.now, progname, message)
    super(severity, message)

    @cloudwatch_logs.put_log_events(
      log_group_name: @log_group_name,
      log_stream_name: @log_stream_name,
      log_events: [
        {
          timestamp: Time.now.to_i * 1000,
          message: message
        }
      ]
    )
    true
  end

  def format_message(severity, timestamp, progname, message)
    "#{timestamp} #{severity} -- #{progname}: #{message}\n"
  end

  def create_log_stream_if_not_exists
    begin
      @cloudwatch_logs.create_log_stream(
        log_group_name: @log_group_name,
        log_stream_name: @log_stream_name
      )
    rescue Aws::CloudWatchLogs::Errors::ResourceNotFoundException => e
      puts "Error creating log stream: #{e.message}"
    end
  end
end

Finally, call the class from the initializer folder. Also, bind the CloudwatchLogger.new into try catch as if you are using docker so it won't be able to find region at docker build time cloudwatch_logger.rb

require 'cloudwatch_logger'

log_group_name =  'enter_your_log_group_name'
log_stream_name = "#{Time.now.strftime('%Y-%m-%d')}/$[LATEST]/#{Time.now.to_i}"
begin
  Rails.logger = CloudwatchLogger.new(log_group_name, log_stream_name)
rescue Aws::Errors::MissingRegionError => e
  puts "Error fetching at region: #{e.message}"
end