I have a cassandra cluster where I recently added two new nodes . Looking at the stats I observed that disk I/O in these newly added machines is way higher than the already present machines . On checking I found that the read_ahead_kb OS configuration on these machines is 4096 whereas other machines it is 4 .
I changed the value but the disk I/O is still same . Do we need to restart the machines for these OS configuration changes to take effect ?
Also if there is any other setting I need to look at .
It depends how you set the readahead value. The following command will set the readahead for /dev/sda to 4kb, and will take effect immediately (no reboot necessary):
sudo blockdev --setra 4 /dev/sda
I recommend configuring a udev rule (as described here), as otherwise the change will be lost after a reboot.
Read ahead is one of the most important performance tweaks regarding disk I/O & throughput. Some other things that are important for read performance:
This guide is a few years old, but many of the OS tuning & hardware recommendations still apply to Cassandra 3.x: https://tobert.github.io/pages/als-cassandra-21-tuning-guide.html