how to add the plugin fluent-plugin-opensearch to docker

I'm trying to send logs from fluentd (installed using docker) to opensearch.

In configuration file, there's @type opensearch that uses the plugin fluent-plugin-opensearch which I installed locally as a Ruby gem.

I get the following error:

2022-04-22 15:47:10 +0000 [error]: config error file="/fluentd/etc/fluentd.conf" error_class=Fluent::NotFoundPluginError error="Unknown output plugin 'opensearch'. Run 'gem search -rd fluent-plugin' to find plugins"

As a solution, I found out that I need to add the plugin to the fluentd docker container, but I couldn't find a way to do that.

Any way to add the plugin to docker or an alternative to this solution would be appreciated.

Solution

The comments already gave a hint, you will need to build your own Docker image. Depending on the infrastructure you have available, you can either build the image, store it in some registry and then use it in your compose file, or build it on the machine that you use docker on.

The Dockerfile

Common to both approaches is that you'll need a Dockerfile. I am using Calyptias Docker image as a base, but you can use whatever fluentd image you like to. My docker file looks as follows:

FROM ghcr.io/calyptia/fluentd:v1.14.6-debian-1.0

USER root

RUN gem install fluent-plugin-opensearch
RUN fluent-gem install fluent-plugin-rewrite-tag-filter fluent-plugin-multi-format-parser

USER fluent
ENTRYPOINT ["tini",  "--", "/bin/entrypoint.sh"]
CMD ["fluentd"]

As you can see it installs a few more plugins, but the first RUN line is the important one for you.

Option 1

If you have a container registry available, you can build the image and push it there, either using a CI/CD pipeline or simply locally. Then you can reference this custom image instead of whatever other fluentd image you're using today as such:

fluentd:
    image: registry.your-domain.xyz/public-projects/fluentd-opensearch:<tag|latest>
    container_name: fluentd
    ports:
      - ...
    restart: unless-stopped
    volumes:
      - ...

Adjust the config to your needs.

Option 2

You can also have docker-compose build the container locally for you. For this, create a directory fluentd in the same folder where you store your docker-compose.yml and place the Dockerfile there.

fluentd:
        build: ./fluentd
        container_name: fluentd
        ports:
          - ...
        restart: unless-stopped
        volumes:
          - ...

Instead of referencing the image from some registry, you can reference a local build directory. This should get you started.