Search code examples
springspring-bootkibanaelastic-stack

Issue with Kibana log entry using Spring Boot & MDC Filter

I'm playing with the ELK stack on elastic.co and I've been banging my head on something for a good 5 hours now (despite reading doc and doc).

So i have a spring boot API, which writes a JSON log file. That log file is taken by filebeat and shipped to Elastic.co ELK stack where I can see the data in Kibana.

In my API I have a Filter class which adds some Values such as Response code etc to the Logs (Shown below), however what I see is that in Kibana those MDC values are actually in the message rather than standalone fields.

Is that correct? Or do I need do a Grok filter? If Grok is the way to go I've constantly tried and fail to pull everything out of the message.

Can anyone help / shed some light before I actually lose my mind :D

@Order(1)
@Component
public class MDCFilter implements Filter {

    private final Logger LOGGER = LoggerFactory.getLogger(MDCFilter.class);
    private final String X_REQUEST_ID = "X-Request-ID";

    @Override
    public void doFilter(ServletRequest request,
                         ServletResponse response,
                         FilterChain chain) throws IOException, ServletException {
        HttpServletRequest req = (HttpServletRequest) request;
        HttpServletResponse res = (HttpServletResponse) response;
        try {
            addXRequestId(req);
            LOGGER.info("path: {}, method: {}, query {}",
                    req.getRequestURI(), req.getMethod(), req.getQueryString());
            res.setHeader(X_REQUEST_ID, MDC.get(X_REQUEST_ID));
            chain.doFilter(request, response);
        } finally {
            MDC.put("api.version", req.getContextPath());
            MDC.put("Server.IP", req.getServerName());
            MDC.put("API.Controller", req.getServletPath());
            MDC.put("Response.code", String.valueOf(res.getStatus()));
            MDC.put("Request.Method.Type", req.getMethod());

            LOGGER.info("statusCode {}, path: {}, method: {}, query {}, context {}, serverName {}, RequestURI {}, RemoteHost {}, Cookies {}",
                    res.getStatus(), req.getRequestURI(), req.getMethod(), req.getQueryString(), req.getContextPath(),
                    req.getServerName(), req.getRequestURI(), request.getRemoteAddr(),
                    req.getCookies());
            MDC.clear();
        }
    }

    private void addXRequestId(HttpServletRequest request) {
        String xRequestId = request.getHeader(X_REQUEST_ID);
        if (xRequestId == null) {
            MDC.put(X_REQUEST_ID, UUID.randomUUID().toString());
        } else {
            MDC.put(X_REQUEST_ID, xRequestId);
        }
    }

}

Here is what the output of my log is:

{"@timestamp":"2020-11-23T10:31:09.010Z", "log.level": "INFO", "message":"statusCode 200, path: /test/service/feed/, method: GET, query null, context /api/v1, serverName localhost, RequestURI /test/service/feed/, RemoteHost 0:0:0:0:0:0:0:1, Cookies null", "service.name":"test-spring-project","event.dataset":"test-service.log","process.thread.name":"http-nio-8080-exec-8","log.logger":"com.example.filter.MDCFilter","api.version":"/api/v1","API.Controller":"/test/service/feed/","Request.Method.Type":"GET","Response.code":"200","Server.IP":"localhost"}

Below is an image of what I see in ELK enter image description here

Solution

  • Ok I'll just add on to what Jan Mares mentioned in a comment.

    So when you have filebeat take the logs from Spring and send them directly to ELK you need to create an ingestion pipeline and tell it what field you want parsed.

    Now, once that is done head over to your index pattern and click on it. There's a tab for edit settings where you then need to add the following to the JSON setting:

    "index.default_pipeline": "INSERT YOUR INGESTION SETTING HERE"

    After that...everything should parse going forward.