I have a slurm cluster of 8 separate rhel9 server nodes to serve as master, compute, and database nodes. The master and compute nodes are up and talking, but I have not activated the database node yet. I know that the slurmdbd node can be configured easily for Job Accounting Storage and Job Completion Logs via the slurm.conf file parameters. However, I cannot find any resources or documentation for how I would store the slurmd computational output files themselves, nor do any of the LLM at my disposal have any idea.
Is there built-in functionality within slurm to connect to a database server and automatically send the output files to that server? If not, any tips or pointers for automatically storing the output files from each node in one database that I can access remotely?
Not important, but if possible I was hoping I would be able to use postgreql and pgadmin4 in particular on this project, due to already having a psql server running on my database node.
The Slurm database configuration is only meant for Slurm's internal use (accounting, reporting, limits and priority, etc.). It offers no functionality to push the results into a database.
If you want to store the results of your jobs in a database, you can add the functionality at the end of your submission script, but that is something you will have to do explicitly, either using the mysql/pgsql command line utility, or some other way if the software you ran has that functionality enabled, and you will have to configure a dedicated database and manage the access rights, etc.
Some workflow management systems can help you build workflows where the ingress/outgress of the data from/to a database are managed automatically.