I know how to recieve emails when my job begins, ends, and/or fails, or, when doing a job array, how to do it for each job in the array vs. once for the array as a whole. I want to substitute email for text. Ideally, I'll get a text when my job starts, ends, or fails, like the following email:
FW: Slurm Job_id=66967693 Name=testing_Tappen Ended, Run time
00:01:55, CANCELLED, ExitCode 0
________________________________________
From: Slurm Controller - <[email protected]>
Sent: Thursday, January 1, 2025 5:18:02 PM (UTC-06:00) Central Time (US & Canada)
To: Jane Doe
Subject: Slurm Job_id=123456789 Name=my_job_name Ended, Run time 05:00:55, CANCELLED, ExitCode 0
This is an automated message requested by a user-submitted Slurm batch job. If you did not request this message, please reply so we can remove you from further messages ASAP.
I know you can email via SMS, given you know the phone's provider. I tried submitting the following job, but despite completing, it didn't end up sending anything:
#!/bin/bash --login
#SBATCH --mail-type=BEGIN,END,FAIL
#SBATCH [email protected]
#SBATCH --time=00:03:00 # walltime
#SBATCH --ntasks-per-node=2 # number of processor cores (i.e. tasks)
#SBATCH --nodes=1 # number of nodes
#SBATCH --mem=64G # memory per CPU core
#SBATCH -J "get_a_text" # job name
#SBATCH --output=%x_%j.out
srun python3 src/jane_doe_script.py
I also tried sending it to my email, then forwarding all emails from SLURM to my mobile and that worked, but with a ~5-15 minute delay and it doesn't let me control which SLURM jobs I want on my phone or don't.
Thoughts?
Additional info:
scontrol --version
--> slurm 23.11.6
This is not related to Slurm rather related to how your Mail Transport Agent (MTA) is configured. Slurm will just pass your mail id and data to sent to the mail program ( which typically can be seen by the command scontrol show config | grep -i mail
), and based on the configuration of the agent the mail is sent.
Following command can be used to see which agent is used to sent the email.
ps aux | grep -E 'postfix|sendmail|exim'
If it is Postfix (often default on many modern Linux distros like Ubuntu or CentOS) then you can see your configuration information via the following command:
postconf -n
and this would lead to some results like this:
smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
So, if your mail agent is configured to not sent it to certain domains (have some restrictions), giving it in the slurm script won't make any difference.
For example, defer_unauth_destination
means if the client is neither in mynetworks
nor SASL-authenticated
(e.g., via username and password), and is attempting to send mail to an external (non-local) domain, defer that delivery (essentially refusing to relay for unauthenticated users).