I'm playing a bit with Monk.io and Kafka - Meltano integration. So, I would like to create a Monk.io Kafka cluster and provision a new connection on Meltano. I'm using https://github.com/lensesio/fast-data-dev for Kafka env. What would be the best approach and have the most sense? I've planed to do it this way: Create runnables of Kafka and Meltano and create Monk actions for Meltano template. Those actions would have custom Meltano loader that would provide pipe to Kafka.
I haven't tested the integration with Kafka but should be similar to Postgres in terms of execution in actions i.e to add a loader or extractor if those Kafka plugins exist for Meltano.
actions:
defines: actions
add:
arguments:
resource-type:
type: string
resource-name:
type: string
code: exec("meltano", "/bin/sh", "-c", `cd elt && meltano add ${args["resource-type"]} ${args["resource-name"]}`)
The workflow and template are abstracted here https://github.com/monk-io/monk-dataops so it's more a question if Meltano has a good Kafka plugin