I'm looking for a Docker image of Fluentd with fluent-plugin-splunk-hec plugin to send data to Splunk.
I found that some time ago Fluentd provided such an image, but it's discontinued for some/no reason. Fluentd official documentation points at the outdated repository as well.
Finally, I found something promising at Docker Hub splunk/fluentd-hec which states:
By splunk • Updated 24 minutes ago This image contains fluentd along with the fluent-plugin-splunk-hec.
When I tried to use this image to send data to Splunk like this:
<match **>
@type splunk_hec
hec_host XX.XX.XX.XX
hec_port 8088
hec_token 00000000-0000-0000-0000-000000000000
</match>
I received an error message:
2022-08-09 10:12:55 +0000 [info]: fluent/log.rb:330:info: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2022-08-09 10:12:55 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-concat' version '2.4.0'
2022-08-09 10:12:55 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-jq' version '0.5.1'
2022-08-09 10:12:55 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.9.4'
2022-08-09 10:12:55 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-prometheus' version '2.0.2'
2022-08-09 10:12:55 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-record-modifier' version '2.1.0'
2022-08-09 10:12:55 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-systemd' version '1.0.2'
2022-08-09 10:12:55 +0000 [info]: fluent/log.rb:330:info: gem 'fluentd' version '1.14.6'
2022-08-09 10:12:55 +0000 [error]: fluent/log.rb:372:error: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::NotFoundPluginError error="Unknown output plugin 'splunk_hec'. Run 'gem search -rd fluent-plugin' to find plugins"
When I checked the image itself I found that there's no fluent-plugin-splunk-hec plugin available inside.
There's yet another option available - build own image using this repo, but maybe I missed something important.
EDIT
I've created a new image using https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/docker-image/v1.15/debian-stackdriver/Dockerfile, but added plugin to the Gemfile as a single line gem "fluent-plugin-splunk-hec". It works as expected:
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluentd' version '1.15.1'
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-concat' version '2.5.0'
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-google-cloud' version '0.4.10'
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-grok-parser' version '2.6.2'
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.13.0'
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-parser-cri' version '0.1.1'
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-prometheus' version '2.0.3'
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-record-modifier' version '2.1.0'
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-splunk-hec' version '1.2.13'
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-systemd' version '1.0.5'
2022-08-11 12:34:22 +0000 [debug]: fluent/log.rb:309:debug: No fluent logger for internal event
2022-08-11 12:34:22 +0000 [info]: fluent/log.rb:330:info: using configuration file: <ROOT>
In compare to the example above you can clearly see that fluent-plugin-splunk-hec plugin was loaded.
/EDIT
What's wrong with the image from Splunk or with my config? What's the easiest way to make it working?
As it turned out the Docker image splunk/fluentd-hec:1.2.13 uses bundle to start fluentd. So, instead of a command:
fluentd -c /fluentd/etc/fluent.conf -v
you should use bundle:
bundle exec fluentd -c /fluentd/etc/fluent.conf
As a result, everything works as expected:
2022-08-16 13:03:28 +0000 [info]: fluent/log.rb:330:info: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2022-08-16 13:03:28 +0000 [info]: fluent/log.rb:330:info: gem 'fluentd' version '1.14.6'
2022-08-16 13:03:28 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-concat' version '2.4.0'
2022-08-16 13:03:28 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-jq' version '0.5.1'
2022-08-16 13:03:28 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.9.4'
2022-08-16 13:03:28 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-prometheus' version '2.0.2'
2022-08-16 13:03:28 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-record-modifier' version '2.1.0'
2022-08-16 13:03:28 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-splunk-hec' version '1.2.13'
2022-08-16 13:03:28 +0000 [info]: fluent/log.rb:330:info: gem 'fluent-plugin-systemd' version '1.0.2'
2022-08-16 13:03:28 +0000 [debug]: fluent/log.rb:309:debug: No fluent logger for internal event