Search code examples
apache-kafkadiagram

How to document Kafka Streams?

How could I draw a diagram of my topci/stream relation ? Is there a standard for that ?

I could use the stream's topology and use a viewer like this one : https://zz85.github.io/kafka-streams-viz/. But I find this too low level.

I want a more abstract diagram representing the topic and the "business" operations between them. Something that do not require a proper knowledge of Kafka to understand.

Solution

  • Kafka streams visualizer just works by creating an image from the topology description (Topology#describe()).

    I want a more abstract diagram representing the topic and the "business" operations between them. Something that do not require a proper knowledge of Kafka to understand.

    I hope you would want to avoid KSTREAM-SOURCE, KSTREAM-TRANSFORM etc from your diagram.

    Topology
Sub-topologies:
Sub-topology: 0
    Source:  KSTREAM-SOURCE-0000000000 (topics: [conversation-meta])
    --> KSTREAM-TRANSFORM-0000000001
    Processor: KSTREAM-TRANSFORM-0000000001 (stores: [conversation-meta-state])
    --> KSTREAM-KEY-SELECT-0000000002
    <-- KSTREAM-SOURCE-0000000000
    Processor: KSTREAM-KEY-SELECT-0000000002 (stores: [])
    --> KSTREAM-FILTER-0000000005
    <-- KSTREAM-TRANSFORM-0000000001
    ...
    omitted for the sake of brevity

    You can simply Replace All the KSTREAM-xxx with appropriate names. For example,

    Topology
Sub-topologies:
Sub-topology: 0
    Source:  Conversations (topics: [conversation-meta])
    --> TransformConversation
    Processor: TransformConversation (stores: [conversation-meta-state])
    --> KSTREAM-KEY-SELECT-0000000002
    <-- KSTREAM-SOURCE-0000000000
    Processor: KSTREAM-KEY-SELECT-0000000002 (stores: [])
    --> KSTREAM-FILTER-0000000005
    <-- TransformConversation

    Here KSTREAM-TRANSFORM-0000000001 has been changed to TransformConversation and KSTREAM-SOURCE-0000000000 has been changed to Conversations

    If you use Processor API in your Kafka streams application and build the Topology and do a describe() the topology will come with appropriate processor names because in Processor API you have the facility to give names for processors.