Search code examples
sql-serverapache-kafka-connectdebezium

Is debezium sqlServer connector task multi-threading?


I'm working with debezium to fetch data from an Microsoft SQL Server database...

I would like to know if the task of the debezium connector is using multiple threads or not.

Looking at the code of debezium I can't find anything clear that specify the use of multi-threading..


Solution

  • Debezium is built on the Kafka Connect framework, which uses a property tasks.max, which can be configured to start multiple threads. However, I don't know if that's recommended, as it'll start multiple database reads

    Stated in the documentation

    MySQL connector always uses a single task and therefore does not use this value, so the default (1) is always acceptable

    If you're trying to read multiple tables, you can create independent connector configs and run them in parallel, thereby starting more than one thread in the Connect worker cluster