We are using logstash in linux to do perform an initial run of our jobs which then creates the last_run metadata files for each JDBC input we have. This works well but we noticed that the last run meta data files are created with an owner of root and group association of root. The config file is executed through SSH to the server under a user called logstash.
sudo time /usr/share/logstash/bin/logstash -f /home/ubuntu/manual-pipeline.conf --path.settings /etc/logstash/ --path.data /var/lib/log
We then have another logstash config which we are running as a service. The only difference is that it contains scheduled jobs that will be using the same last run metadata files from our first config. We noticed that when it is able to update the last run metadata, the owner of the file is logstash. We now have a number of metadata files that are under root but are not getting updated. If we manually update the owner to logstash then the scheduled jobs are able to update the file without problems.
sudo chown logstash .last_run
All our metadata files are stored in /etc/logstash which we configured to owner root and group logstash. We are using the same SSH connection for running the manual-pipeline.conf and copying scheduled-pipeline.conf to /etc/logstash/conf.d and starting the service using sudo service logstash start.
logstash user only has a group of logstash and adm. I would also add that we are only using the timestamp from the last run metadata and we use the :sql_last_value in the where clause of our SQL queries.
Questions:
Sorry if the info is all over the place. I'm just new to logstash and all of this. I believe we are using Logstash 5.4.1 if it helps.
Why is the initial creation of the last run metadata files owned by root? Can we configure this?
The last run metadata files will be owned by the user that runs logstash, if they are first owned by the root user, then someone ran logstash as root, using sudo.
Why is our service using the logstash user and not root? (I'm new to this)
This is the default, the logstash service uses the user logstash, not root. It is advised to not run logstash as root, for security reasons.
We know we can work around this by changing the owner but is there a better solution to this problem?
In your example you are using sudo without the -u logstash parameter, so it will run as the root user, not as the logstash user.
You should try to run logstash as the logstash user, just guarantee that the logstash user has access to your config file inside the directory /home/ubuntu or move this file to another directory where the logstash user has access.