Set up kafka.service and zookeeper.service in AWS EC2 Ubuntu

On my own Ubuntu 20.04 machine, with a user kafka, I can successfully set up the following service files (referencing to this tutorial):

sudo vi /etc/systemd/system/zookeeper.service

[Unit]
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
User=kafka
ExecStart=/home/kafka/kafka/bin/zookeeper-server-start.sh /home/kafka/kafka/config/zookeeper.properties
ExecStop=/home/kafka/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

sudo vi /etc/systemd/system/kafka.service

[Unit]
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
User=kafka
ExecStart=/bin/sh -c '/home/kafka/kafka/bin/kafka-server-start.sh /home/kafka/kafka/config/server.properties > ~/kafka/kafka.log 2>&1'
ExecStop=/home/kafka/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

And then successfully start Kafka and check its status:

sudo systemctl start kafka
sudo systemctl status kafka

Now I want to set up in an AWS EC2 Ubuntu instance with the same folder structure:

sudo vi /etc/systemd/system/zookeeper.service

[Unit]
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
User=ubuntu
ExecStart=/home/ubuntu/kafka/bin/zookeeper-server-start.sh /home/ubuntu/kafka/config/zookeeper.properties
ExecStop=/home/ubuntu/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

sudo vi /etc/systemd/system/kafka.service

[Unit]
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
User=ubuntu
ExecStart=/bin/sh -c '/home/ubuntu/kafka/bin/kafka-server-start.sh /home/ubuntu/kafka/config/server.properties > /home/ubuntu/kafka/kafka.log 2>&1'
ExecStop=/home/ubuntu/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

The ubuntu is my username for AWS EC2 Ubuntu AMI according to AWS document.

But when I start kafka.service, I get the following error:

kafka.service
     Loaded: loaded (/etc/systemd/system/kafka.service; disabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2021-06-10 18:58:26 UTC; 7s ago
    Process: 40816 ExecStart=/bin/sh -c /home/ubuntu/kafka/bin/kafka-server-start.sh /home/ubuntu/kafka/config/server.properties > /home/ubuntu/kafka/kafka.log 2>&1 (code=exited, status=2)
   Main PID: 40816 (code=exited, status=2)

Jun 10 18:58:26 ip-xxx-xx-xx-xxx systemd[1]: Started kafka.service.
Jun 10 18:58:26 ip-xxx-xx-xx-xxx sh[40816]: /bin/sh: 1: cannot create /home/ubuntu/kafka/kafka.log: Directory nonexistent
Jun 10 18:58:26 ip-xxx-xx-xx-xxx systemd[1]: kafka.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jun 10 18:58:26 ip-xxx-xx-xx-xxx systemd[1]: kafka.service: Failed with result 'exit-code'.

Under directory ~/kafka, I don't see the file kafka.log. But when setting up on my own machine, I did not encounter this error.

Am I missing anything when setting this up in AWS EC2 Ubuntu?

More details: I just realized that with my user ubuntu access, I cannot create a folder without sudo. In other word:

$ mkdir test_folder

gives me Permission denied. I have to do:

$sudo mkdir test_folder

So, how do I modify the service files to account for this?

Solution

with my user ubuntu access, I cannot create a folder without sudo

Unclear how you added your ubuntu user, but running mkdir -p /home/ubuntu as ec2-user wouldn't be correct.

You can fix the permissions with these commands, but you might want to re-visit the useradd command options

sudo chown -R ubuntu:ubuntu /home/ubuntu
sudo chmod -R a+rwX,o-w /home/ubuntu

You should also prefer to have separate user accounts for Kafka and Zookeeper rather than use some ubuntu user, which is what would happen if you use APT to install Confluent Platform, for example