Search code examples
springspring-bootlogback

AWS CloudWatch logging with Spring Boot

What should be the approach to store logs of multiple spring boot application(s) in cloud watch?

Sample spring-boot - logback-spring.xml configuration file is below.

<?xml version="1.0" encoding="UTF-8"?>


<property name="LOGS" value="/logs/abc/" />

<appender name="Console" class="ch.qos.logback.core.ConsoleAppender">
    <layout class="ch.qos.logback.classic.PatternLayout">
        <Pattern> %black(%d{ISO8601}) %highlight(%-5level) [%blue(%t)] %yellow(%C{1.}): %msg%n%throwable </Pattern>
    </layout>
</appender>

<appender name="RollingFile" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>${LOGS}/abc-log.log</file>
    <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> 
    <Pattern>%d %p %C{1.} [%t] %m%n</Pattern> </encoder>
    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
        <!-- rollover daily and when the file reaches 10 MegaBytes -->
        <fileNamePattern>${LOGS}/archived/abc-log-%d{yyyy-MM-dd}.%i.log
        </fileNamePattern>
        <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
            <maxFileSize>5MB</maxFileSize>
        </timeBasedFileNamingAndTriggeringPolicy>
        <maxHistory>2</maxHistory>
        <totalSizeCap>10MB</totalSizeCap>
    </rollingPolicy>
</appender>

<!-- LOG everything at INFO level -->
<root level="info">
    <appender-ref ref="RollingFile" />
    <appender-ref ref="Console" />
</root>

<logger name="com.abc" level="trace" additivity="false">
    <appender-ref ref="RollingFile" />
    <appender-ref ref="Console" />
</logger>

Solution

  • I was able to achieve the same using Cloudwatch log agent.

    Step 1 - Create awslogs.conf file to point to the log location.

    [/logs/abcd/8080-abcd.log]
datetime_format = %Y-%m-%d %H:%M:%S
file = /logs/abcd/8080-abcd.log
buffer_duration = 5000
log_stream_name = {hostname}
initial_position = start_of_file
log_group_name = ABCD Group Name

[/logs/defg/8081-defg.log]
datetime_format = %Y-%m-%d %H:%M:%S
file = /logs/abcd/8081-defg.log
buffer_duration = 5000
log_stream_name = {hostname}
initial_position = start_of_file
log_group_name = DEFG Group Name

    Step 2 - Install cloud watch log agent as part of UserData/Bootstrap script for ec2.

    yum install wget -y
wget https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py
python ./awslogs-agent-setup.py --region $aws_region --non-interactive --configfile=/configlocation/awslogs.conf

    Step 3 - Need a role for ec2 machine to create logs in CloudWatch.