How to automatically start Kafka upon system startup in Ubuntu?

Question

Does Kafka have an official way (e.g. an init.d script) to start Kafka on system start up?

The only official method to start Kafka I have seen is:

nohup ~/kafka/bin/kafka-server-start.sh ~/kafka/config/server.properties > ~/kafka/kafka.log 2>&1 &

I have tried to use a @reboot task in crontab -e however it did not start Kafka. Some people have also written custom init.d

There are also custom init.d scripts available (e.g. one, two, three) however they are all different and I am not familiar enough with init.d to understand which one, if any to implement.

How to start Kafka on system startup?

Answer 1

Here's how I configure Kafka to start automatically on Ubuntu 14.04:

sudo su
cp -R ~/kafka_2.11-0.10.0.1 /opt
ln -s /opt/kafka_2.11-0.10.0.1 /opt/kafka

Copy the following init script to /etc/init.d/kafka:

DAEMON_PATH=/opt/kafka/
PATH=$PATH:$DAEMON_PATH/bin

# See how we were called.
case "$1" in
  start)
        # Start daemon.
        echo "Starting Zookeeper";
        nohup $DAEMON_PATH/bin/zookeeper-server-start.sh -daemon /$DAEMON_PATH/config/zookeeper.properties 2> /dev/null && \
        echo "Starting Kafka";
        nohup $DAEMON_PATH/bin/kafka-server-start.sh -daemon /$DAEMON_PATH/config/server.properties 2> /dev/null
        ;;
  stop)
        # Stop daemons.
        echo "Shutting down Zookeeper";
        pid=`ps ax | grep -i 'org.apache.zookeeper.server' | grep -v grep | awk '{print $1}'`
        if [ -n "$pid" ]
          then
          kill -9 $pid
        else
          echo "Zookeeper was not Running"
        fi
        echo "Shutting down Kafka";
        pid=`ps ax | grep -i 'kafka.Kafka' | grep -v grep | awk '{print $1}'`
        if [ -n "$pid" ]
          then
          kill -9 $pid
        else
          echo "Kafka was not Running"
        fi
        ;;
  restart)
        $0 stop
        sleep 2
        $0 start
        ;;
  status)
        pid=`ps ax | grep -i 'org.apache.zookeeper.server' | grep -v grep | awk '{print $1}'`
        if [ -n "$pid" ]
          then
          echo "Zookeeper is Running as PID: $pid"
        else
          echo "Zookeeper is not Running"
        fi
        pid=`ps ax | grep -i 'kafka.Kafka' | grep -v grep | awk '{print $1}'`
        if [ -n "$pid" ]
          then
          echo "Kafka is Running as PID: $pid"
        else
          echo "Kafka is not Running"
        fi
        ;;
  *)
        echo "Usage: $0 {start|stop|restart|status}"
        exit 1
esac

exit 0

Make the kafka service with these commands:

chmod 755 /etc/init.d/kafka
update-rc.d kafka defaults

Now you should be able to start and stop the kafka service like this:

sudo service kafka start
sudo service kafka status
sudo service kafka stop

If you want to remove the Kafka service later, run update-rc.d -f kafka remove.