Search code examples
clickhouseapache-pulsar

Apache Pulsar Clickhouse Sink - does it have intervals between inserts?


Clickhouse allows high performance writes but only if they are done in bulk and with intervals (recommended is at least 1 second interval between inserts). In the documentation to JDBC connector for Clickhouse batchSize option exists but there is nothing about intervals between inserts and I didn't manage to find insertion logic in the code (I am not a Java guy though).

Does it mean there are no intervals and Pulsar simply does inserts as frequently as it can?


Solution

  • I know nothing about Pulsar.

    recommended is at least 1 second interval between inserts

    That recommendation is "one insert per second". Nothing about 1 second to sleep.

    This recommendation is too basic and very vague.

    Every project is unique, has own environment and requirements.

    In one project I insert 10mil. very wide rows per minute with RAID 10 with HDD disks.

    In another project I do 1000 inserts with ~100 narrow rows each per second using In-memory parts with single NVME disk.