I would like to modify the IIS logs for further transfer to the destination. Now I am parsing the IIS log with the xm_csv module, as in the template. UndefValue is disabled to not get empty.
How can I interact with parsed data from w3c_parser?
For example, I want to combine into a variable $request = '"' + $cs-method + ' ' + $cs-uri-stem + ' ' + $cs-version + '"'; such a value, but I get an error. When I try to write a field from w3c_parser to $raw_event, I also get an error. Any other data is added without error.
For example $raw_event = $c-ip -- error
$raw_event = $EventTime + ' ' + $http_host -- no error
Example error, logs and config file below
2022-03-23 16:49:56 ERROR Couldn't parse Exec block at C:\Program Files\nxlog\conf\nxlog.conf:59; couldn't parse statement at line 71, character 32 in C:\Program Files\nxlog\conf\nxlog.conf; syntax error, unexpected +, expecting (
2022-03-23 16:49:56 ERROR module 'iis_w3c' has configuration errors, not adding to route 'uds_to_file' at C:\Program Files\nxlog\conf\nxlog.conf:84
2022-03-23 16:49:56 ERROR route uds_to_file is not functional without input modules, ignored at C:\Program Files\nxlog\conf\nxlog.conf:84
2022-03-23 16:49:56 WARNING no routes defined!
2022-03-23 16:49:56 WARNING not starting unused module iis_w3c
2022-03-23 16:49:56 WARNING not starting unused module file
2022-03-23 16:49:56 INFO nxlog-ce-3.0.2272 started
Current log format
date time s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-bytes cs-bytes time-taken
2022-03-23 08:00:01 HOST.DOMAIN 99.XX.XX.4 GET /AnalyticsService - 443
- XX.XX.XX.XXX HTTP/1.1 Zabbix - - site.host.domain 200 3918 144 4
Required log format
$http_host $remote_addr $remote_user [$time_local] UNIX-TIME-$msec "$request" $status "$sent_http_content_type" $body_bytes_sent "$http_referer" "$http_user_agent" "$http_cookie" $request_time "$upstream_addr" NGINX-CACHE-$upstream_cache_status "$request_id" "$request_body"
host.domain 99.99.99.249 - [11/Mar/2022:20:09:56+0300] UNIX-TIME-1647018596.031 "GET /api/company.php?id=853747 HTTP/1.1" 200 "text/xml; charset=UTF-8" 1455 "-" "-" "20b6b325ea192383cb1244412247c5ea=3002538ef353c9daab4f742176a840; etpsid=f488b343a23d1a4a2332e089a0" 0.059 0.059 "10.10.10.111:80" NGINX-CACHE-- "d0b5ac12cf82671067aa5e6c5c" "-"
Panic Soft
#NoFreeOnExit TRUE
define ROOT C:\Program Files\nxlog
define CERTDIR %ROOT%\cert
define CONFDIR %ROOT%\conf\nxlog.d
define LOGDIR %ROOT%\data
define LOGFILE %LOGDIR%\nxlog.log
LogFile %LOGFILE%
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
<Extension _syslog>
Module xm_syslog
</Extension>
<Extension fileop>
Module xm_fileop
</Extension>
<Extension _charconv>
Module xm_charconv
AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32
</Extension>
<Extension _exec>
Module xm_exec
</Extension>
#Create the parse rule for IIS logs. You can copy these from the header of the IIS log file.
<Extension w3c_parser>
Module xm_csv
Fields $date, $time, $s-computername, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs-version, $cs(User-Agent), $cs(Cookie), $cs(Referer), $cs-host, $sc-status, $sc-bytes, $cs-bytes, $time-taken
FieldTypes string, string, string, string, string, string, string, integer, string, string, string, string, string, string, string, integer, integer, integer, integer
Delimiter ' '
EscapeChar '"'
QuoteChar '"'
EscapeControl FALSE
# UndefValue -
</Extension>
<Extension w3c_out>
Module xm_csv
Fields $http_host, $c-ip, $cs-username, $EventTime1, $sc-status, $Unix
FieldTypes string, string, string, string, string, string
Delimiter ' '
# UndefValue -
QuoteMethod None
</Extension>
<Input iis_w3c>
Module im_file
File 'C:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log'
SavePos TRUE
<Exec>
if $raw_event =~ /^#/ drop();
else
{
w3c_parser->parse_csv();
$EventTime = parsedate($date + " " + $time);
$EventTime = $EventTime + (3 * 3600);
$EventTime1 = strftime($EventTime, '[%d/%b/%Y:%H:%M:%S]');
# $EventTime1 = '$EventTime1' + ' +0003]';
$Unix = integer($EventTime);
$Unix = 'UNIX-TIME-' + $Unix;
$http_host = "site.host.domain";
# $request = '"' + $cs-method + ' ' + $cs-uri-stem + ' ' + $cs-version + '"';
# $request = $cs-method;
w3c_out->to_csv();
}
</Exec>
</Input>
<Output file>
Module om_file
File 'C:\inetpub\logs\LogFiles\Parser\w3c.txt'
</Output>
<Route uds_to_file>
Path iis_w3c => file
</Route>
let's start with NXLog language in the conf files. Dashes in explicit format are not allowed - you can check: https://nxlog.co/docs/nxlog-ce/nxlog-reference-manual.html#lang_fields
Hence, one needs to apply curly braces to reach the goal ({}). If I understand your issue correctly, this may help with most of your issues.