The job script `job.ksh`¶

The job script is a Korn-shell script which will be created at runtime for each flex_extract execution in the application modes remote and gateway.

It is based on the submitscript.template template file stored in the Templates directory. This template is generated in the installation process from a jobscript.template template file.

Flex_extract uses the Python package genshi to generate the Korn-shell script from the template files by substituting the individual parameters. These individual parameters are marked by $$ in jobscript.template.

The job script has a number of settings for the batch system which are fixed, and differentiates between the ecgate and the cca/ccb server system to load the necessary modules for the environment when submitted to the batch queue.

The submission is done by the ECaccess tool from within flex_extract with the command ecaccess-job-submit.

What does the job script do?¶

It sets necessary batch system parameters.

It prepares the job environment at the ECMWF servers by loading the necessary library modules.

It sets some environment variables for the single session.

It creates the directory structure in the user’s $SCRATCH file system.

It creates a CONTROL file on the ECMWF servers whith the parameters set before creating the job.ksh. Flex_extract has a set of parameters which are passed to the job script with their default or the user-defined values. It also sets CONTROL as an environment variable.

Flex_extract is started from within the work directory of the new directory structure by calling the submit.py script. It sets new paths for input and output directories and the recently generated CONTROL file.

At the end, it checks whether the script has returned an error or not, and emails the log file to the user.

Example `job.ksh`¶

#!/bin/ksh

# ON ECGB:
# start with ecaccess-job-submit -queueName ecgb NAME_OF_THIS_FILE  on gateway server
# start with sbatch NAME_OF_THIS_FILE directly on machine

#SBATCH --workdir=/scratch/ms/at/km4a
#SBATCH --qos=normal
#SBATCH --job-name=flex_ecmwf
#SBATCH --output=flex_ecmwf.%j.out
#SBATCH --error=flex_ecmwf.%j.out
#SBATCH --mail-type=FAIL
#SBATCH --time=12:00:00

## CRAY specific batch requests
##PBS -N flex_ecmwf
##PBS -q np
##PBS -S /usr/bin/ksh
## -o /scratch/ms/at/km4a/flex_ecmwf.${PBS_JOBID}.out
## job output is in .ecaccess_DO_NOT_REMOVE
##PBS -j oe
##PBS -V
##PBS -l EC_threads_per_task=24
##PBS -l EC_memory_per_task=32000MB

set -x
export VERSION=7.1
case ${HOST} in
  *ecg*)
  module unload grib_api
  module unload emos
  module load python3
  module load eccodes
  module load emos/455-r64
  export PATH=${PATH}:${HOME}/flex_extract_v7.1/Source/Python
  ;;
  *cca*)
  module switch PrgEnv-cray PrgEnv-intel
  module load python3
  module load eccodes
  module load emos/455-r64
  export SCRATCH=${TMPDIR}
  export PATH=${PATH}:${HOME}/flex_extract_v7.1/Source/Python
  ;;
esac

cd ${SCRATCH}
mkdir -p python$$
cd python$$

export CONTROL=CONTROL

cat >${CONTROL}<<EOF
accmaxstep 24
acctime 18
acctype FC
accuracy 24
addpar None
area 74.0/-24.0/10.0/60.0
basetime None
cds_api None
controlfile CONTROL_CERA
cwc 1
dataset None
date_chunk 3
debug 1
destination <specificname>@genericSftp
doubleelda 0
dpdeta 1
dtime 3
ec_api None
ecfsdir ectmp:/${USER}/econdemand/
ecgid at
ecstorage 0
ectrans 1
ecuid km4a
end_date 20000809
eta 1
etadiff 0
etapar 77
expver 1
format GRIB1
gateway srvx8.img.univie.ac.at
gauss 0
gaussian
grib2flexpart 0
grid 1.0/1.0
inputdir <path-to-flex_extract>/flex_extract_v7.1/run/workspace
install_target None
job_chunk 1
job_template job.temp
left -24.
level 91
levelist 1/to/91
logicals gauss omega omegadiff eta etadiff dpdeta cwc wrf grib2flexpart ecstorage ectrans debug oper request public purefc rrint doubleelda
lower 10.
mailfail ${USER}
mailops ${USER}
marsclass EP
maxstep 0
number 000
omega 0
omegadiff 0
oper 0
outputdir <path-to-flex_extract>/flex_extract_v7.1/run/workspace
prefix CE
public 0
purefc 0
queue ecgate
request 2
resol 159
right 60.
rrint 0
smooth 0
start_date 20000809
step 00 00 00 00 00 00 00 00
stream ENDA
time 00 03 06 09 12 15 18 21
type AN AN AN AN AN AN AN AN
upper 74.
wrf 0

EOF


submit.py --controlfile=${CONTROL} --inputdir=./work --outputdir=./work 1> prot 2>&1

if [ $? -eq 0 ] ; then
  l=0
  for muser in `grep -i MAILOPS ${CONTROL}`; do
      if [ ${l} -gt 0 ] ; then
         mail -s flex.${HOST}.$$ ${muser} <prot
      fi
      l=$((${l}+1))
  done
else
  l=0
  for muser in `grep -i MAILFAIL ${CONTROL}`; do
      if [ ${l} -gt 0 ] ; then
         mail -s "ERROR! flex.${HOST}.$$" ${muser} <prot
      fi
      l=$((${l}+1))
  done
fi