Search code examples
apache-kafkamesosmesospheredcos

difference between dcos-kafka-service and mesos-kafka

I’m doing a POC to deploy Kafka as an application on Mesos Cluster. I came across these 2 codebases on github. One developed by apache-mesos (github page) & other developed by mesosphere and can run only on DCOS (github page).

Question: Would like to know if there are any differences between DCOS-Kafka & mesos-Kafka in terms of features and extended functionality.

Regarding Mesos-Kafka: I don’t see active participation on github (and some open issues) for mesos-kafka in the past months. Can I assume that the service is robust enough that I can use in production environment? Any Inputs on this would be helpful.

Solution

  • kakfa-mesos is a package that includes a release of Kafka and a custom mesos scheduler that was meant to work around issues with running Kafka as a stateful service on Marathon. I think post but confluent is useful. It also includes a RESTful api for doing ops tasks and aims to include these features in the future (this is pulled from the article I linked)

    • Integrating Kafka commands (e.g. kafka-topics, etc) into the Scheduler so it can be used through the CLI and REST API.
    • Auto-scaling clusters (including auto reassignment of partitions) so that the resources (CPU, RAM, etc.) that brokers are using can be used elsewhere in known valleys of traffic.
    • Rack-aware partition assignment for fault tolerance.
    • Hooks so that producers and consumers can also be launched from the Scheduler and managed with the cluster.
    • Automated partition reassignment based on load and traffic

    I haven't used it in a production environment myself but it has the support of Confluent which is a good sign.

    DC/OS Kafka on the other hand is a DC/OS service which will probably only be useful if you are already running or plan on running services through Mesosphere's DC/OS. It also includes an API and a CLI management tool but is less ambitious with additional features. It's current feature set includes

    • Single-command installation for rapid provisioning
    • Multiple clusters for multiple tenancy with DC/OS
    • High availability runtime configuration and software updates
    • Storage volumes for enhanced data durability, known as Mesos Dynamic * * Reservations and Persistent Volumes
    • Integration with syslog-compatible logging services for diagnostics and troubleshooting
    • Integration with statsd-compatible metrics services for capacity and performance monitoring