Kafka - increase partition count of existing topic through Confluent REST Api

Question

I need to add partitions to an existing Kafka topic. I'm aware that it is possible to use the /bin/kafka-topics.sh script to achieve this, but I would prefer to do this through the Confluent REST api.

As far as I see there is no documented endpoint in the api reference, but I wonder if someone else here was able to make this work.

Edit: As it does seem to be impossible to use the REST api here, I wonder what the best practice is for adding partitions to an existing topic in a containerized setup. E.g. if there is a custom partioning scheme that maps customer ids to specific partitions. In this case the app container would need to adjust the partition count of the kafka container.

Answer 1

This is the solution I ended up with:

• Create a small http service that is is deployed within the kafka docker image
• The http service accepts requests to increase the partition count and directs the requests to the kafka admin scripts (bin/kafka/kafka-topics.sh)
  • Something similar could have been achieved by using the AdminClient NewPartitions api in the Java Kafka lib. This solution has the advantage that the kafka docker image does not have to be changed, because the AdminClient can connect through network from another container.

For a production setup the AdminClient is preferrable, I decided for the integrated script approach because of the chosen language (rust).