I'm using the NFS gateway service to ingest some data into HDFS (CDH 5.4.5).
All seemed to be well until I received a message warning informing me that the logging directory free space was running very low. I had a quick look for big log files:
sudo find /var/log -type f -size +100000k -exec ls -lh {} \; | awk '{ print $9 ": " $5 }'
... and noticed a 34GB file called /var/log/hadoop-hdfs/hadoop-cmf-hdfs-NAMENODE-[fqdn-of-name-node].log.out. A quick peek inside showed that almost entirely consisted of this warning:
2015-11-30 13:41:15,535 WARN org.apache.hadoop.security.UserGroupInformation: No groups available for user [some_user]
Seeing this, I added the some_user user to the hdfs group, i.e. usermod -a -G hdfs some_user on the node running the NFS gateway, and then restarted the NFS gateway service. Unfortunately, the "No groups available for user ..." message is still getting logged at a ferocious rate.
Can you see what I'm doing wrong? Does the some_user account need to be setup on each node, and not just the NFS gateway? Do I need to do something more than just restart the service for the changes to be effective?
The user and its group need to be set up on the namenode. By default, Hadoop gets the group mappings from the OS settings there. See for example here: https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#Group_Mapping