Log messages from Django application not uploaded in AWS CloudWatch

Question

I have added log messages in my Django application and it was successfully logging log messages to the log file.

Now, I tried to add log messages to AWS CloudWatch. When I run the application it creates log group in AWS CloudWatch but log stream is not created and log messages also not uploaded.

I have also manually created log stream in AWS CloudWatch but still log messages were not uploaded.

I have the following logging configuration in my Django application.

logger_boto3_client = boto3.client(
    "logs",
    aws_access_key_id=CLOUDWATCH_AWS_ID,
    aws_secret_access_key=CLOUDWATCH_AWS_KEY,
    region_name=AWS_DEFAULT_REGION,
)

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'simple': {
            'format': "[cid: %(cid)s] [%(asctime)s.%(msecs)03d] %(levelname)s [%(name)s:%(lineno)s] [%(funcName)s] %(message)s",
            'datefmt': '%Y-%m-%d %H:%M:%S',
        },
    },
    'handlers': {
        'logger': {
            'level': 'DEBUG',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': str(BASE_DIR) + '/logs/test.log',
            'formatter': 'simple',
            'filters': ['correlation'],
        },
        'watchtower': {
            "level": "DEBUG",
            "class": "watchtower.CloudWatchLogHandler",
            "boto3_client": logger_boto3_client,
            "log_group": "DemoLogs2",
            # Different stream for each environment
            "stream_name": "logs",
            "formatter": "simple",
            'filters': ['correlation'],
        }
    },
    'filters': {
        'correlation': {
            '()': 'cid.log.CidContextFilter'
        },
    },
    'loggers': {
        'root': {
            'handlers': ['logger', 'watchtower'],
            'level': 'DEBUG',
            'filters': ['correlation'],
            'propagate': True,
        }
    }
}

In my application I'm using logger like this,

import logging


logger = logging.getLogger(__name__)
logger.info("log message.")

My aws cloudwatch console.

Answer 1

The issue is with calling the logger in the API.

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'default': {
            'format': "[cid: %(cid)s] [%(asctime)s.%(msecs)03d] %(levelname)s [%(name)s:%(lineno)s] [%(funcName)s] %(message)s",
            'datefmt': '%Y-%m-%d %H:%M:%S',
        }
    },
    'handlers': {
        'file': {
            'level': 'DEBUG',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': str(BASE_DIR) + '/logs/test.log',
            'formatter': 'default',
            'filters': ['correlation'],
        }
        'cloudwatch': {
            "level": "DEBUG",
            "class": "watchtower.CloudWatchLogHandler",
            "boto3_client": logger_boto3_client,
            "log_group": CLOUDWATCH_LOG_GROUP,
            "stream_name": f"{ENV}-{CLOUDWATCH_DEFAULT_LOG_STREAM_NAME}",
            "formatter": "default",
            'filters': ['correlation'],
        }
    },
    'filters': {
        'correlation': {
            '()': 'cid.log.CidContextFilter'
        },
    },
    'loggers': {
        # previously named as 'root'
        'default': {
            'level': 'DEBUG',
            'handlers': ['file', 'cloudwatch'],
            'filters': ['correlation'],
            'propagate': False,
        },
    }
}

In the API,

import logging


# logger = logging.getLogger(__name__)
logger = logging.getLogger("default")
logger.info("log message.")

Previously I was passing __name__ to the getLogger method. Now, I passed the logger name default to the getLogger method and now I could able to see the log messages in both the log file and cloudwatch.