Batch job on lustre is not working: awk: cmd. line:1 "unexpected newline or end of string"

Question

I have recently started running Python batch jobs on a Lustre system.

I have recently made changes to the shell script, resulting in the error:

awk: cmd. line:1: NR==
awk: cmd. line:1:     ^ unexpected newline or end of string

The script is as follows:

#$ -cwd


# Define parallel environment for N cores
#$ -pe openmp 4



# Validation level (e = reject on all problems)
#$ -w e

# Merge stdout and stderr streams: yes/no
#$ -j yes

#name of job (human-friendly)
#$ -N experiment

# array job parameters below:
# which job IDs in the array to run, eg 1-5 or 3
#$ -t 1
# how many jobs of the array to run at a time
#$ -tc 15


MYFILE=all_experiment_parameters.txt.$RANDOM
python Get_Parameter_Settings.py > $MYFILE
SEED=$(awk "NR==$SGE_TASK_ID" $MYFILE)
echo $SEED

python RunExperiment.py $SEED

This is similar to the syntax in the example given in section 'Pulling data from the ith line of a file' here: http://wiki.gridengine.info/wiki/index.php/Simple-Job-Array-Howto.

Could someone advise on the meaning of the error message (I don't understand why it is warning of 'newline or end of string', when pointing at the middle of my code), and how to fix this? I'm sure it is a trivial change I have made by mistake, as it was working earlier.

Answer 1

I have realised why SGE_TASK_ID wasn't getting set. I was running the shell script using

./shell.sh

instead of submitting it to the queue with

qsub shell.sh

So a total beginner's mistake.