NGINX, Extracting url path from http header

Question

So I'm sending my Nginx access logs to Datadog (an APM solution).

My log format looks like this

  log_format json_custom 
    '{'
      '"http.version":"$request",'
      '"http.status_code":$status,'
      '"http.method":"$request_method",'
      '"http.referer":"$http_referer",'
      '"http.useragent":"$http_user_agent",'
      '"time_local":"$time_local",'
      '"remote_addr":"$remote_addr",'
      '"remote_user":"$remote_user",'
      '"body_bytes_sent":"$body_bytes_sent",'
      '"request_time":$request_time,'
      '"response_content_type":"$sent_http_content_type",'
      '"X-Forwarded-For":"$proxy_add_x_forwarded_for",'
      '"custom_key":"custom_value"'
    '}';

I can extract the url from the referrer field and it looks like this

http://example.com/foo/bar 

I only want /foo/bar though. Is this something I have to modify in the log_format?

I saw an example from datadog docs where they're able to extract a url path attribute, but there's no example config.

Answer 1

If you already have an attribute that contains the url, a really easy way to do this would be to use the processing pipelines and add a processor of the "url parser" type to these logs. You just plug in the attribute that contains the url and an attribute path that you'd like to contain all the outputs from it (usually http.url_details), and then all new logs will get the extra url parsing applied.

If your logs have the "source:nginx" applied to them (configured in the log shipper), then you'll already have an out-of-the-box Nginx processing pipeline that Datadog has for structuring standard Nginx syntax logs. You can clone that and then just add your new url parser there. Or, if your syntax is similar to the standard syntax, you can just modify their default suggested parsers (in the cloned pipeline). In any case, it'd be worth looking at that default pipeline for inspiration for other valuable things to do beyond url parsing.