Search code examples
pythonelasticsearchlogginglogstashkibana

How to Send Structured Logs to ELK Stack with Custom Fields Using Python Logging?

I'm working on a Python application where I want to send logs to the ELK stack. However, I'm struggling to add custom fields to the log entries so that they appear as separate fields in Kibana, not just inside the "message" field. I'm relatively new to the ELK stack, so any guidance would be appreciated.

I tried using custom formatters, but all fields still end up inside the "message" field in Kibana. Here's my current Python script:

import logging

from logstash_async.formatter import LogstashFormatter

try:
    import json
except ImportError:
    import simplejson as json

from logstash_async.handler import AsynchronousLogstashHandler


class CustomLogstashFormatter(LogstashFormatter):
    def format(self, record):
        log_record = {
            "timestamp": self._format_timestamp(record.created),
            "level": record.levelname,
            "logger": record.name,
            "message": record.getMessage(),
            "filename": record.pathname,
            "funcName": record.funcName,
            "appName": "MyPythonApp"
        }
        if record.exc_info:
            log_record['exception'] = self._format_exception(record.exc_info)

        return json.dumps(log_record)


def activate_logging() -> None:
    logstash_handler = AsynchronousLogstashHandler("0.0.0.0", 5000, database_path=None)
    logstash_handler.setFormatter(CustomLogstashFormatter())
    logging.basicConfig(
        level=logging.INFO,
        force=True,
        handlers=[logstash_handler]
    )

Despite this setup, all log fields appear nested within the "message" field. enter image description here

I've researched and found some suggestions about changing the logstash.conf file to properly parse the log fields, but I can't modify the Logstash configuration at the moment.

Is there any way to achieve my goal directly from the application?

Any help or guidance would be greatly appreciated.

Solution

  • It turned out that actually the only way was to update the logstash .conf. What I didn't want to do was to add filters or plugins, but really it's simply a problem of how I defined the input part. My configuration was as follows:

    input {
    tcp {
        port => 5000
    }
}

output {
    elasticsearch {
        hosts => "elasticsearch:9200"
    }
}

    Since no codec was specified, the data was treated as plain text. By updating it to what follows, it manages to unpack the 'message' into fields.

    input {
    tcp {
        port => 5000
        type => syslog
        codec => json_lines
    }
}

output {
    elasticsearch {
        hosts => "elasticsearch:9200"
    }
}