Trying to find an example of how to configure the Confluent syslog connector to accept data in CEF format. I have the syslog connector up and running, but seeing these kinds of errors in the connect.log, never makes it to my topic:
[2020-03-24 23:56:09,181] WARN decode() - Could not parse message. request = 'SyslogRequest{receivedDate=2020-03-24T23:56:09.181, remoteAddress=/<some ip>, rawMessage=Wed Mar 25 00:31:30 2020 1/1/e1 CEF:0|Gigamon|metadata|unknown|4|metadata generation|6|GigamonMdataDeltaBytesRcvd=40 GigamonMdataDeltaPKtsRcvd=1 GigamonMdataFlowStartSysUpTime=264840090 GigamonMdataFlowEndSysUpTime=264840090 proto=6 src=<some ip> dst=<some ip> spt=34846 dpt=9997 }' (io.confluent.connect.syslog.message.SyslogMessageParsingHandler)
I configure connector through REST like this:
curl localhost:8083/connectors -X POST -H "Content-Type: application/json" -d '{"name": "connect-syslog-netflow","config": { "connector.class": "io.confluent.connect.syslog.SyslogSourceConnector", "tasks.max": "1", "topics":"syslog-netflow","syslog.port": "2055","confluent.topic.bootstrap.servers": "localhost:9092", "confluent.topic.replication.factor": "1", "type": "CEF" } }'
type CEF was weak attempt at telling it incoming format.
Running 5.0.1 confluent distro, one node dev cluster.
So after working with confluent support we determined problem:
rawMessage=Wed Mar 25 00:31:30 2020 ...
It works by changing that date to:
rawMessage=Mar 25 00:31:30 ...
Stripping off day of week and year.
Unfortunately I don't control the netflow source, so I guess I'll need to have some kind of custom transform on data before sending on to connector.