Search code examples
apache-kafkamapr

mapR Kafka cannot start second time round

To date I have either used an existing professional installation for Hadoop with components running, or, installed Kafka and used the also-supplied Zookeeper in a native VM.

I am trying to get the mapR Community Edition Sandbox to run now.

  • There is a KAFKA library on mapR, but here is no kafka shown when using jps. Seems odd? I managed to get KAFKA to start once.
  • There is a Zookeeper service on mapR but it uses port 5181, not 2181.

  • Kafka uses port 9092.

  • The log.dirs for kafka was set to /tmp/kafka-logs, I changed that to /opt/kafka-logs

  • The dataDir was also set to /tmp/zookeeper, I changed that to /opt/zookeeper

  • I also changed the Zookeeper port to 5181 as that is what mapR uses.

It ran once, and then I re-started and I still get this type of error:

java.io.FileNotFoundException: /tmp/kafka-logs/.lock (Permission denied)

I have done chmod 777 where required I think, but I changed the paths to /opt/... from /tmp. So why is it picking /tmp up again?

I have the impression that it keeps on point to /tmp regardless of the updates to the configurations.

I also see a warning - although I do not think this is an issue:

[2019-01-14 13:26:46,355] WARN No meta.properties file under dir /tmp/kafka-logs/meta.properties (kafka.server.BrokerMetadataCheckpoint)

May be because of the mapR Streams I cannot influence it so as to run natively?

Solution

  • OK, I could delete the question as I solved it, but for those on mapR I deduced:

    • You need to update the port 2181 to 5181 on server.properties immediately. In this case we integrate with an existing zookeeper instance.
    • Likewise, update the log.dirs for Kafka from /tmp/kafka-logs asap to /opt/kafka-logs.
    • Likewise, update the dataDir from /tmp/zookeeper asap to /opt/zookeeper.

    Trying to fix latterly otherwise leads to all sorts of issues. I ended up just re-installing and doing it right from scratch.

    mapR has a faster version called mapR Streams which implements Kafka. I was not wanting to use that for what I was wanting to do, but mapR Sandbox has a lot of up-to-date items straight out of the box -certainly compared to Cloudera.