Auto-generated documentation for the Python scripts¶
Programs¶
install¶
This script installs the flex_extract program.
Depending on the selected installation environment (locally or on the ECMWF server ecgate or cca) the program extracts the command line arguments and the CONTROL file parameter and prepares the corresponding environment. The necessary files are collected in a tar ball and placed at the target location. There, is is untared, the environment variables are set, and the Fortran code is compiled. If the ECMWF environment is selected, a job script is prepared and submitted for the remaining configurations after putting the tar ball on the target ECMWF server.
Type: install.py –help to get information about command line parameters. Read the documentation for usage instructions.
-
install.
check_install_conditions
(c)[source]¶ Checks necessary attributes and conditions for the installation, e.g. whether they exist and contain values. Otherwise set default values.
- Parameters
c (ControlFile) – Contains all the parameters of CONTROL file and command line.
-
install.
del_fortran_build
(src_path)[source]¶ Clean up the Fortran source directory and remove all build files (e.g. *.o, *.mod and FORTRAN EXECUTABLE)
- Parameters
src_path (str) – Path to the fortran source directory.
-
install.
get_install_cmdline_args
()[source]¶ Decomposes the command line arguments and assigns them to variables. Apply default values for arguments not present.
- Returns
args – Contains the commandline arguments from script/program call.
- Return type
Namespace
-
install.
install_local
(c)[source]¶ Perform the actual installation on a local machine.
- Parameters
c (ControlFile) – Contains all the parameters of CONTROL file and command line.
-
install.
install_via_gateway
(c)[source]¶ Prepare data transfer to remote gateway and submit a job script which will install everything on the remote gateway.
- Parameters
c (ControlFile) – Contains all the parameters of CONTROL file and command line.
-
install.
main
()[source]¶ Controls the installation process. Calls the installation function if target is specified.
-
install.
mk_compilejob
(makefile, ecuid, ecgid, fp_root)[source]¶ Modifies the original job template file so that it is specified for the user and the environment were it will be applied. Result is stored in a new file “job.temp” in the python directory.
- Parameters
makefile (str) – Name of the makefile which should be used to compile the Fortran program.
ecuid (str) – The user id on ECMWF server.
ecgid (str) – The group id on ECMWF server.
fp_root (str) – Path to the root directory of FLEXPART environment or flex_extract environment.
-
install.
mk_env_vars
(ecuid, ecgid, gateway, destination)[source]¶ Creates a file named ECMWF_ENV which contains the necessary environmental variables at ECMWF servers. It is based on the template ECMWF_ENV.template.
- Parameters
ecuid (str) – The user id on ECMWF server.
ecgid (str) – The group id on ECMWF server.
gateway (str) – The gateway server the user is using.
destination (str) – The remote destination which is used to transfer files from ECMWF server to local gateway server.
-
install.
mk_fortran_build
(src_path, makefile)[source]¶ Compiles the Fortran code and generates the executable.
- Parameters
src_path (str) – Path to the fortran source directory.
makefile (str) – The name of the makefile which should be used.
-
install.
mk_job_template
(ecuid, ecgid, fp_root)[source]¶ Modifies the original job template file so that it is specified for the user and the environment were it will be applied. Result is stored in a new file.
- Parameters
ecuid (str) – The user id on ECMWF server.
ecgid (str) – The group id on ECMWF server.
fp_root (str) – Path to the root directory of FLEXPART environment or flex_extract environment.
-
install.
mk_tarball
(tarball_path, target)[source]¶ Creates a tarball with all necessary files which need to be sent to the installation directory. It does not matter whether this is local or remote. Collects all Python files, the Fortran source and makefiles, the ECMWF_ENV file, the CONTROL files as well as the template files.
- Parameters
tarball_path (str) – The complete path to the tar file which will contain all relevant data for flex_extract.
target (str) – The queue where the job is submitted to.
submit¶
This script allows the user to extract meteorological fields from the ECMWF.
It prepares the settings for retrieving the data from ECMWF servers and process the resulting files to prepare them for the use with FLEXPART or FLEXTRA.
If it is supposed to work locally then it works through the necessary functions get_mars_data and prepare_flexpart. Otherwise it prepares a job script (korn shell) which will do the necessary work on the ECMWF server. This script will de submitted via the ecaccess command ecaccess-job-submit.
This file can also be imported as a module which then contains the following functions:
main - the main function of the script
submit - calls mk_jobscript depending on operation mode and submits its
mk_jobscript - creates the job script from a template
Type: submit.py –help to get information about command line parameters. Read the documentation for usage instructions.
-
submit.
main
()[source]¶ Get the arguments from script call and from CONTROL file. Decides from the argument “queue” if the local version is done “queue=None” or the gateway version with “queue=ecgate” or “queue=cca”.
-
submit.
mk_jobscript
(jtemplate, job_file, clist)[source]¶ Creates the job script from template.
- Parameters
jtemplate (str) – Job template file from sub-directory “Templates” for submission to ECMWF. It contains all necessary module and variable settings for the ECMWF environment as well as the job call and mail report instructions. Default is _config.TEMPFILE_JOB.
job_file (str) – Path to the job script file.
clist (list of str) – Contains all necessary parameters for ECMWF CONTROL file.
-
submit.
submit
(jtemplate, c, queue)[source]¶ Prepares the job script and submits it to the specified queue.
- Parameters
jtemplate (str) – Job template file from sub-directory “_templates” for submission to ECMWF. It contains all necessary module and variable settings for the ECMWF environment as well as the job call and mail report instructions. Default is _config.TEMPFILE_JOB.
c (ControlFile) – Contains all the parameters of CONTROL file and command line.
queue (str) – Name of queue for submission to ECMWF (e.g. ecgate or cca )
Classes¶
ControlFile¶
-
class
ControlFile.
ControlFile
(filename)[source]¶ Contains the information which are stored in the CONTROL files.
The CONTROL file is the steering part of the FLEXPART extraction software. All necessary parameters needed to retrieve the data fields from the MARS archive for driving FLEXPART are set in a CONTROL file. Some specific parameters like the start and end dates can be overwritten by the command line parameters, but in generall all parameters needed for a complete set of fields for FLEXPART can be set in the CONTROL file.
-
controlfile
¶ The name of the control file to be processed. Default value is the filename passed to the init function when initialised.
- Type
str
-
start_date
¶ The first day of the retrieval period. Default value is None.
- Type
str
-
end_date
¶ The last day of the retrieval period. Default value is None.
- Type
str
-
date_chunk
¶ Length of period for a single mars retrieval. Default value is 3.
- Type
int
-
dtime
¶ The time step in hours. Default value is None.
- Type
str
-
basetime
¶ The time for a half day retrieval. The 12 hours upfront are to be retrieved. Default value is None.
- Type
int
-
maxstep
¶ The maximum forecast step for non flux data. Default value is None.
- Type
int
-
type
¶ List of field type per retrieving hour. Default value is None.
- Type
list of str
-
time
¶ List of retrieving times in hours. Default valuer is None.
- Type
list of str
-
step
¶ List of forecast time steps in hours for non flux data. Default value is None.
- Type
list of str or str
-
acctype
¶ The field type for the accumulated forecast fields. Default value is None.
- Type
str
-
acctime
¶ The starting time of the accumulated forecasts. Default value is None.
- Type
str
-
accmaxstep
¶ The maximum forecast step for the accumulated forecast fields (flux data). Default value is None.
- Type
int
-
marsclass
¶ Characterisation of dataset. Default value is None.
- Type
str
-
dataset
¶ For public datasets there is the specific naming and parameter dataset which has to be used to characterize the type of data. Default value is None.
- Type
str
-
stream
¶ Identifies the forecasting system used to generate the data. Default value is None.
- Type
str
-
number
¶ Selects the member in ensemble forecast run. Default value is ‘OFF’.
- Type
str
-
expver
¶ The version number of the dataset. Default value is ‘1’.
- Type
str
-
gaussian
¶ This parameter is deprecated and should no longer be used. Specifies the desired type of Gaussian grid for the output. Default value is an empty string ‘’.
- Type
str
-
grid
¶ Specifies the output grid which can be either a Gaussian grid or a Latitude/Longitude grid. Default value is None.
- Type
str
-
area
¶ Specifies the desired sub-area of data to be extracted. Default value is None.
- Type
str
-
left
¶ The western most longitude of the area to be extracted. Default value is None.
- Type
str
-
lower
¶ The southern most latitude of the area to be extracted. Default value is None.
- Type
str
-
upper
¶ The northern most latitued of the area to be extracted. Default value is None.
- Type
str
-
right
¶ The eastern most longitude of the area to be extracted. Default value is None.
- Type
str
-
level
¶ Specifies the maximum level. Default value is None.
- Type
str
-
levelist
¶ Specifies the required level list. Default value is None.
- Type
str
-
resol
¶ Specifies the desired triangular truncation of retrieved data, before carrying out any other selected post-processing. Default value is None.
- Type
str
-
gauss
¶ Switch to select gaussian fields (1) or regular lat/lon (0). Default value is 0.
- Type
int
-
accuracy
¶ Specifies the number of bits per value to be used in the generated GRIB coded fields. Default value is 24.
- Type
int
-
omega
¶ Switch to select omega retrieval (1) or not (0). Default value is 0.
- Type
int
-
omegadiff
¶ Switch to decide to calculate Omega and Dps/Dt from continuity equation for diagnostic purposes (1) or not (0). Default value is 0.
- Type
int
-
eta
¶ Switch to select direct retrieval of etadot from MARS (1) or wether it has to be calculated (0). Then Default value is 0.
- Type
int
-
etadiff
¶ Switch to select calculation of etadot and Dps/Dt from continuity equation for diagnostic purposes (1) or not (0). Default value is 0.
- Type
int
-
etapar
¶ GRIB parameter Id for etadot fields. Default value is 77.
- Type
int
-
dpdeta
¶ Switch to select multiplication of etadot with dpdeta. Default value is 1.
- Type
int
-
smooth
¶ Spectral truncation of ETADOT after calculation on Gaussian grid. Default value is 0.
- Type
int
-
format
¶ The format of the GRIB data. Default value is ‘GRIB1’.
- Type
str
-
addpar
¶ List of additional surface level ECMWF parameter to be retrieved. Default value is None.
- Type
str
-
prefix
¶ Prefix string for the final FLEXPART/FLEXTRA ready input files. Default value is ‘EN’.
- Type
str
-
cwc
¶ Switch to select wether the sum of cloud liquid water content and cloud ice water content should be retrieved. Default value is 0.
- Type
int
-
wrf
¶ Switch to select further parameters for retrievment to support WRF simulations. Default value is 0.
- Type
int
-
ecfsdir
¶ Path to the ECMWF storage ‘ectmp:/${USER}/econdemand/’
- Type
str
-
mailfail
¶ Email list for sending error log files from ECMWF servers. The email addresses should be seperated by a comma. Default value is [‘${USER}’].
- Type
list of str
-
mailops
¶ Email list for sending operational log files from ECMWF servers. The email addresses should be seperated by a comma. Default value is [‘${USER}’].
- Type
list of str
-
ecstorage
¶ Switch to select storage of FLEXPART ready output files in the ECFS file system. Default value is 0.
- Type
int
-
ectrans
¶ Switch to select the transfer of FLEXPART ready output files to the gateway server. Default value is 0.
- Type
int
-
inputdir
¶ Path to the temporary directory for the retrieval grib files and other processing files. Default value is _config.PATH_INPUT_DIR.
- Type
str
-
outputdir
¶ Path to the final directory where the final FLEXPART ready input files are stored. Default value is None.
- Type
str
-
flexextractdir
¶ Path to the flex_extract root directory. Default value is _config.PATH_FLEXEXTRACT_DIR.
- Type
str
-
exedir
¶ Path to the FORTRAN executable file. Default value is _config.PATH_FORTRAN_SRC.
- Type
str
-
installdir
¶ Path to a FLEXPART root directory. Default value is None.
- Type
str
-
makefile
¶ Name of the makefile to be used for the Fortran program. Default value is None.
- Type
str
-
destination
¶ The remote destination which is used to transfer files from ECMWF server to local gateway server. Default value is None.
- Type
str
-
gateway
¶ The gateway server the user is using. Default value is None.
- Type
str
-
ecuid
¶ The user id on ECMWF server. Default value is None.
- Type
str
-
ecgid
¶ The group id on ECMWF server. Default value is None.
- Type
str
-
install_target
¶ Defines the location where the installation is to be done. Default value is None.
- Type
str
-
debug
¶ Switch to keep temporary files at the end of postprocessing (1) or to delete all temporary files except the final output files (0). Default value is 0.
- Type
int
-
oper
¶ Switch to prepare the operational job script. Start date, end date and basetime will be prepared with environment variables. Default value is 0.
- Type
int
-
request
¶ Switch to select between just retrieving the data (0), writing the mars parameter values to a csv file (1) or doing both (2). Default value is 0.
- Type
int
-
public
¶ Switch to select kind of ECMWF Web Api access and the possible data sets. Public data sets (1) and Memberstate data sets (0). Default value is 0.
- Type
int
-
ec_api
¶ Tells wether the ECMWF Web API was able to load or not. Default value is None.
- Type
boolean
-
cds_api
¶ Tells wether the CDS API was able to load or not. Default value is None.
- Type
boolean
-
purefc
¶ Switch to decide wether the job is a pure forecast retrieval or coupled with analysis data. Default value is 0.
- Type
int
-
rrint
¶ Switch to select between old precipitation disaggregation method (0) or the new IA3 disaggegration method (1). Default value is 0.
- Type
int
-
doubleelda
¶ Switch to select the calculation of extra ensemble members for the ELDA stream. It doubles the amount of retrieved ensemble members.
- Type
int
-
logicals
¶ List of the names of logical switches which controls the flow of the program. Default list is [‘gauss’, ‘omega’, ‘omegadiff’, ‘eta’, ‘etadiff’, ‘dpdeta’, ‘cwc’, ‘wrf’, ‘ecstorage’, ‘ectrans’, ‘debug’, ‘request’, ‘public’, ‘purefc’, ‘rrint’, ‘doubleelda’]
- Type
list of str
-
assign_args_to_control
(args)[source]¶ Overwrites the existing ControlFile instance attributes with the command line arguments.
- Parameters
args (Namespace) – Contains the commandline arguments from script/program call.
-
assign_envs_to_control
(envs)[source]¶ Assigns the ECMWF environment parameter.
- Parameters
envs (dict of str) – Contains the ECMWF environment parameternames “ECUID”, “ECGID”, “DESTINATION” and “GATEWAY” with its corresponding values. They were read from the file “ECMWF_ENV”.
-
check_conditions
(queue)[source]¶ Checks a couple of necessary attributes and conditions, such as if they exist and contain values. Otherwise set default values.
- Parameters
queue (str) – Name of the queue if submitted to the ECMWF servers. Used to check if ecuid, ecgid, gateway and destination are set correctly and are not empty.
-
to_list
()[source]¶ Just generates a list of strings containing the attributes and assigned values except the attributes “_expanded”, “exedir”, “flexextractdir” and “installdir”.
- Returns
l – A sorted list of the all ControlFile class attributes with their values except the attributes “_expanded”, “exedir”, “flexextractdir” and “installdir”.
- Return type
list of *
-
EcFlexpart¶
-
class
EcFlexpart.
EcFlexpart
(c, fluxes=False)[source]¶ Class to represent FLEXPART specific ECMWF data.
FLEXPART needs grib files in a specifc format. All necessary data fields for one time step are stored in a single file. The class represents an instance with all the parameter and settings necessary for retrieving MARS data and modifing them so they are fitting FLEXPART needs. The class is able to disaggregate the fluxes and convert grid types to the one needed by FLEXPART, therefore using the FORTRAN program.
-
mreq_count
¶ Counter for the number of generated mars requests.
- Type
int
-
inputdir
¶ Path to the directory where the retrieved data is stored.
- Type
str
-
dataset
¶ For public datasets there is the specific naming and parameter dataset which has to be used to characterize the type of data.
- Type
str
-
basetime
¶ The time for a half day retrieval. The 12 hours upfront are to be retrieved.
- Type
int
-
dtime
¶ Time step in hours.
- Type
str
-
acctype
¶ The field type for the accumulated forecast fields.
- Type
str
-
acctime
¶ The starting time from the accumulated forecasts.
- Type
str
-
accmaxstep
¶ The maximum forecast step for the accumulated forecast fields.
- Type
str
-
marsclass
¶ Characterisation of dataset.
- Type
str
-
stream
¶ Identifies the forecasting system used to generate the data.
- Type
str
-
number
¶ Selects the member in ensemble forecast run.
- Type
str
-
resol
¶ Specifies the desired triangular truncation of retrieved data, before carrying out any other selected post-processing.
- Type
str
-
accuracy
¶ Specifies the number of bits per value to be used in the generated GRIB coded fields.
- Type
str
-
addpar
¶ List of additional parameters to be retrieved.
- Type
str
-
level
¶ Specifies the maximum level.
- Type
str
-
expver
¶ The version of the dataset.
- Type
str
-
levelist
¶ Specifies the required levels.
- Type
str
-
glevelist
¶ Specifies the required levels for gaussian grids.
- Type
str
-
gaussian
¶ This parameter is deprecated and should no longer be used. Specifies the desired type of Gaussian grid for the output.
- Type
str
-
grid
¶ Specifies the output grid which can be either a Gaussian grid or a Latitude/Longitude grid.
- Type
str
-
area
¶ Specifies the desired sub-area of data to be extracted.
- Type
str
-
purefc
¶ Switch for definition of pure forecast mode or not.
- Type
int
-
outputfilelist
¶ The final list of FLEXPART ready input files.
- Type
list of str
-
types
¶ Determines the combination of type of fields, time and forecast step to be retrieved.
- Type
dictionary
-
params
¶ Collection of grid types and their corresponding parameters, levels, level types and the grid definition.
- Type
dictionary
-
server
¶ This is the connection to the ECMWF data servers.
- Type
ECMWFService or ECMWFDataServer
-
public
¶ Decides which Web API Server version is used.
- Type
int
-
dates
¶ Contains start and end date of the retrieval in the format “YYYYMMDD/to/YYYYMMDD”
- Type
str
-
calc_extra_elda
(path, prefix)[source]¶ Calculates extra ensemble members for ELDA - Stream.
This is a specific feature which doubles the number of ensemble members for the ELDA Stream.
- Parameters
path (str) – Path to the output files.
prefix (str) – The prefix of the output filenames as defined in Control file.
-
create
(inputfiles, c)[source]¶ An index file will be created which depends on the combination of “date”, “time” and “stepRange” values. This is used to iterate over all messages in each grib file which were passed through the parameter “inputfiles” to seperate specific parameters into fort.* files. Afterwards the FORTRAN program is called to convert the data fields all to the same grid and put them in one file per unique time step (combination of “date”, “time” and “stepRange”).
Note
This method is based on the ECMWF example index.py https://software.ecmwf.int/wiki/display/GRIB/index.py
- Parameters
inputfiles (UioFiles) – Contains a list of files.
c (ControlFile) – Contains all the parameters of CONTROL file and command line.
-
deacc_fluxes
(inputfiles, c)[source]¶ De-accumulate and disaggregate flux data.
Goes through all flux fields in ordered time and de-accumulate the fields. Afterwards the fields are disaggregated in time. Different versions of disaggregation is provided for rainfall data (darain, modified linear) and the surface fluxes and stress data (dapoly, cubic polynomial).
- Parameters
inputfiles (UioFiles) – Contains the list of files that contain flux data.
c (ControlFile) – Contains all the parameters of CONTROL file and command line.
-
process_output
(c)[source]¶ Postprocessing of FLEXPART input files.
The grib files are postprocessed depending on the selection in CONTROL file. The resulting files are moved to the output directory if its not equal to the input directory. The following modifications might be done if properly switched in CONTROL file: GRIB2 - Conversion to GRIB2 ECTRANS - Transfer of files to gateway server ECSTORAGE - Storage at ECMWF server
- Parameters
c (ControlFile) – Contains all the parameters of CONTROL file and command line.
-
retrieve
(server, dates, public, request, inputdir='.')[source]¶ Finalizing the retrieval information by setting final details depending on grid type. Prepares MARS retrievals per grid type and submits them.
- Parameters
server (ECMWFService or ECMWFDataServer) – The connection to the ECMWF server. This is different for member state users which have full access and non member state users which have only access to the public data sets. The decision is made from command line argument “public”; for public access its True (ECMWFDataServer) for member state users its False (ECMWFService)
dates (str) – Contains start and end date of the retrieval in the format “YYYYMMDD/to/YYYYMMDD”
request (int) – Selects the mode of retrieval. 0: Retrieves the data from ECMWF. 1: Prints the mars requests to an output file. 2: Retrieves the data and prints the mars request.
inputdir (str, optional) – Path to the directory where the retrieved data is about to be stored. The default is the current directory (‘.’).
-
write_namelist
(c)[source]¶ Creates a namelist file in the temporary directory and writes the following values to it: maxl, maxb, mlevel, mlevelist, mnauf, metapar, rlo0, rlo1, rla0, rla1, momega, momegadiff, mgauss, msmooth, meta, metadiff, mdpdeta
- Parameters
c (ControlFile) – Contains all the parameters of CONTROL file and command line.
filename (str) – Name of the namelist file.
-
GribUtil¶
-
class
GribUtil.
GribUtil
(filenames)[source]¶ Class for GRIB utilities (new methods) based on GRIB API
The GRIB API provides all necessary tools to work directly with the grib files. Nevertheless, the GRIB API tools are very basic and are in direct connection with the grib files. This class provides some higher functions which apply a set of GRIB API tools together in the respective context. So, the class initially contains a list of grib files (their names) and the using program then applies the methods directly on the class objects without having to think about how the actual GRIB API tools have to be arranged.
-
copy_dummy_msg
(filename_in, keynames, keyvalues, selectwhere=True, filemode='wb')[source]¶ Add the content of another input grib file to the objects file but only messages corresponding to keys/values passed to the function. The selectWhere switch decides if to copy the keys equal to (True) or different to (False) the keynames/keyvalues list passed to the function.
- Parameters
filename_in (
string
) – Filename of the input file to read the grib messages from.selectwhere (
boolean
, optional) – Decides if to copy the keynames and values equal to (True) or different to (False) the keynames/keyvalues list passed to the function. Default is True.keynames (
list
ofstring
) – List of keynames.keyvalues (
list
ofstring
) – List of keyvalues.filemode (
string
, optional) – Sets the mode for the output file. Default is “wb”.
-
get_keys
(keynames, wherekeynames, wherekeyvalues)[source]¶ Get keyvalues for a given list of keynames a where statement can be given (list of key and list of values)
- Parameters
keynames (
list
ofstring
) – List of keynames.wherekeynames (
list
ofstring
) – List of key names for indexing grib message parameter.wherekeyvalues (
list
ofstring
) – List of key values corresponding the key names.
- Returns
return_list – List of keyvalues for given keynames.
- Return type
list
ofstring
-
index
(index_keys, index_file='my.idx')[source]¶ Create index file from a list of files if it does not exist or read an index file.
- Parameters
index_keys (
list
ofstring
) – Contains the list of key parameter names from which the index is to be created.index_file (
string
, optional) – Filename where the indices are stored. Default is “my.idx”.
- Returns
iid – Grib index id.
- Return type
integer
-
set_keys
(fromfile, filemode='wb', keynames=[], keyvalues=[], wherekeynames=[], wherekeyvalues=[])[source]¶ Opens the file to read the grib messages and then write the selected messages (with wherekeys) to a new output file. Also, the keyvalues of the passed list of keynames are set.
- Parameters
fromfile (
string
) – Filename of the input file to read the grib messages from.keynames (
list
ofstring
) – List of keynames to set in the selected messages. Default is an empty list.keyvalues (
list
ofstring
) – List of keyvalues to set in the selected messages. Default is an empty list.wherekeynames (
list
ofstring
) – List of keynames to select correct message.wherekeyvalues (
list
ofstring
) – List of keyvalues for keynames to select correct message.filemode (
string
, optional) – Sets the mode for the output file. Default is “wb”.
-
MarsRetrieval¶
-
class
MarsRetrieval.
MarsRetrieval
(server, public, marsclass='EA', dataset='', type='', levtype='', levelist='', repres='', date='', resol='', stream='', area='', time='', step='', expver='1', number='', accuracy='', grid='', gaussian='', target='', param='')[source]¶ Specific syntax and content for submission of MARS retrievals.
A MARS revtrieval has a specific syntax with a selection of keywords and their corresponding values. This class provides the necessary functions by displaying the selected parameters and their values and the actual retrievement of the data through a mars request or a Python web api interface. The initialization already expects all the keyword values.
A description of MARS keywords/arguments and examples of their values can be found here: https://software.ecmwf.int/wiki/display/UDOC/ Identification+keywords#Identificationkeywords-class
-
server
¶ This is the connection to the ECMWF data servers.
- Type
ECMWFService or ECMWFDataServer
-
public
¶ Decides which Web API Server version is used.
- Type
int
-
marsclass
¶ Characterisation of dataset.
- Type
str, optional
-
dataset
¶ For public datasets there is the specific naming and parameter dataset which has to be used to characterize the type of data.
- Type
str, optional
-
type
¶ Determines the type of fields to be retrieved.
- Type
str, optional
-
levtype
¶ Denotes type of level.
- Type
str, optional
-
levelist
¶ Specifies the required levels.
- Type
str, optional
-
repres
¶ Selects the representation of the archived data.
- Type
str, optional
-
date
¶ Specifies the Analysis date, the Forecast base date or Observations date.
- Type
str, optional
-
resol
¶ Specifies the desired triangular truncation of retrieved data, before carrying out any other selected post-processing.
- Type
str, optional
-
stream
¶ Identifies the forecasting system used to generate the data.
- Type
str, optional
-
area
¶ Specifies the desired sub-area of data to be extracted.
- Type
str, optional
-
time
¶ Specifies the time of the data in hours and minutes.
- Type
str, optional
-
step
¶ Specifies the forecast time step from forecast base time.
- Type
str, optional
-
expver
¶ The version of the dataset.
- Type
str, optional
-
number
¶ Selects the member in ensemble forecast run.
- Type
str, optional
-
accuracy
¶ Specifies the number of bits per value to be used in the generated GRIB coded fields.
- Type
str, optional
-
grid
¶ Specifies the output grid which can be either a Gaussian grid or a Latitude/Longitude grid.
- Type
str, optional
-
gaussian
¶ This parameter is deprecated and should no longer be used. Specifies the desired type of Gaussian grid for the output.
- Type
str, optional
-
target
¶ Specifies a file into which data is to be written after retrieval or manipulation.
- Type
str, optional
-
param
¶ Specifies the meteorological parameter.
- Type
str, optional
-
UioFiles¶
-
class
UioFiles.
UioFiles
(path, pattern)[source]¶ Collection of files matching a specific pattern.
The pattern can contain regular expressions for the files. The files are listed and can be transformed to a single string or they can be deleted.
-
path
¶ Directory where to list the files.
- Type
str
-
files
¶ List of files matching the pattern in the path.
- Type
list of str
-
Modules¶
get_mars_data¶
This script extracts MARS data from ECMWF.
At first, the necessary parameters from command line and CONTROL files are extracted. They define the data set to be extracted from MARS.
This file can also be imported as a module and contains the following functions:
main - the main function of the script
get_mars_data - overall control of ECMWF data retrievment
write_reqheader - writes the header into the mars_request file
mk_server - creates the server connection to ECMWF servers
mk_dates - defines the start and end date
remove_old - deletes old retrieved grib files
do_retrieval - creates individual retrievals
Type get_mars_data.py –help to get information about command line parameters. Read the documentation for usage instructions.
-
get_mars_data.
check_dates_for_nonflux_fc_times
(types, times)[source]¶ Checks if the time 18UTC corresponds to forecast field.
- Parameters
types (list of str) – List of field types.
times (list of str or str) – The time in hours of the field.
- Returns
- Return type
True or False
-
get_mars_data.
do_retrievement
(c, server, start, end, delta_t, fluxes=False)[source]¶ Divides the total retrieval period into smaller chunks and retrieves the data from MARS.
- Parameters
c (ControlFile) – Contains all the parameters of CONTROL file and command line.
server (ECMWFService or ECMWFDataServer) – The server connection to ECMWF.
start (datetime) – The start date of the retrieval.
end (datetime) – The end date of the retrieval.
delta_t (datetime) – Delta_t + 1 is the maximum time period of a single retrieval.
fluxes (boolean, optional) – Decides if the flux parameters are to be retrieved or the rest of the parameter list. Default value is False.
-
get_mars_data.
get_mars_data
(c)[source]¶ Retrieves the ECMWF data required for a FLEXPART simulation.
Start and end dates for retrieval period are set. Retrievals are divided into shorter periods if necessary and if datechunk parameter is set.
- Parameters
c (ControlFile) – Contains all the parameters of CONTROL file and command line.
-
get_mars_data.
main
()[source]¶ Controls the program to retrieve data from MARS.
This is done if called directly from command line. Then, arguments and control file are taken as input.
-
get_mars_data.
mk_dates
(c, fluxes)[source]¶ Prepares start and end date depending on flux or non-flux type of data.
If forecasts for a maximum of one day (24 h) are to be retrieved, then collect accumulation data (flux data) with additional days in the beginning and at the end (needed for complete disaggregation of original period)
If forecast data for more than +24 h are to be retrieved, then collect accumulation data (flux data) with the exact start and end date (disaggregation will be done for the exact time period with boundary conditions)
Since for basetime the extraction contains the 12 hours upfront, if basetime is 0, the starting date has to be the day before
- Parameters
c (ControlFile) – Contains all the parameters of CONTROL file and command line.
fluxes (boolean, optional) – Decides if the flux parameter settings are stored or the rest of the parameter list. Default value is False.
- Returns
start (datetime) – The start date of the retrieving data set.
end (datetime) – The end date of the retrieving data set.
chunk (datetime) – Time period in days for one single mars retrieval.
-
get_mars_data.
mk_server
(c)[source]¶ Creates a server connection with available Python API.
The API selected depends on availability and the data set to be retrieved. The CDS API is used for ERA5 data, no matter whether the user is a member-state or a public user. ECMWF WebAPI is used for all other available datasets.
- Parameters
c (ControlFile) – Contains all the parameters of CONTROL file and command line.
- Returns
server – Connection to ECMWF server via python interface ECMWF WebAPI or CDS API.
- Return type
ECMWFDataServer, ECMWFService or Client
-
get_mars_data.
remove_old
(pattern, inputdir)[source]¶ Deletes old retrieval files from current input directory matching the pattern.
- Parameters
pattern (str) – The substring pattern which identifies the files to be deleted.
inputdir (str, optional) – Path to the directory where the retrieved data are stored.
prepare_flexpart¶
This script prepares the final version of the grib files which are then used by FLEXPART.
It converts the bunch of grib files extracted via get_mars_data before, by doing the necessary conversion to obtain consistent grids or the disaggregation of flux data. Finally, the data fields are combined in files per hour available with the naming convention xxYYMMDDHH, where xx should be 2 arbitrary letters (mostly xx is chosen to be “EN”).
This file can also be imported as a module which then contains the following functions:
main
prepare_flexpart
Type: prepare_flexpart.py –help to get information about command line parameters. Read the documentation for usage instructions.
-
prepare_flexpart.
main
()[source]¶ Controls the program to prepare FLEXPART input files from MARS data.
This is done if called directly from the command line. Then, arguments and control file are taken as input.
-
prepare_flexpart.
prepare_flexpart
(ppid, c)[source]¶ Converts the MARS data into files ready as input for FLEXPART.
Certain fields are converted to a different grid and the flux data are disaggregated. Fields are collected by hour and stored in a file with a specific naming convention.
- Parameters
ppid (int) – Contains the ppid number of the current ECMWF job. It will be None if the method was called within this module.
c (ControlFile) – Contains all the parameters of CONTROL file and command line.
tools¶
This module contains a collection of diverse tasks within flex_extract.
-
tools.
clean_up
(c)[source]¶ Remove files from the intermediate directory (inputdir).
It keeps the final FLEXPART input files if program runs without ECMWF API and keywords “ectrans” or “ecstorage” are set to “1”.
- Parameters
c (ControlFile) – Contains all the parameters of CONTROL file and command line.
-
tools.
execute_subprocess
(cmd_list, error_msg='SUBPROCESS FAILED!')[source]¶ Executes a command via a subprocess.
Error handling is done if an error occures.
- Parameters
cmd_list (list of str) – A list of the components for the command line execution. They will be concatenated with blank space for the command to be submitted, like [‘mv’, file1, file2] for mv file1 file2.
- Returns
error_msg – Error message if the subprocess fails. By default it will just say “SUBPROCESS FAILED!”.
- Return type
str, optional
-
tools.
generate_retrieval_period_boundary
(c)[source]¶ Generates retrieval period boundary datetimes from CONTROL information.
- Parameters
c (ControlFile) – Contains all the parameters of CONTROL file and command line.
- Returns
start_period (datetime) – The first timestamp of the actual retrieval period disregarding the temporary times which were used for processing reasons.
end_period (datetime) – The last timestamp of the actual retrieval period disregarding the temporary times which were used for processing reasons.
-
tools.
get_cmdline_args
()[source]¶ Decomposes the command line arguments and assigns them to variables. Apply default values for arguments not present.
- Returns
args – Contains the command line arguments from the script / program call.
- Return type
Namespace
-
tools.
get_dimensions
(info, purefc, dtime, index_vals, start_date, end_date)[source]¶ This function specifies the correct dimensions for x, y, and t.
- Parameters
info (dict) – Contains basic informations of the ECMWF grib files, e.g. ‘Ni’, ‘Nj’, ‘latitudeOfFirstGridPointInDegrees’, ‘longitudeOfFirstGridPointInDegrees’, ‘latitudeOfLastGridPointInDegrees’, ‘longitudeOfLastGridPointInDegrees’, ‘jDirectionIncrementInDegrees’, ‘iDirectionIncrementInDegrees’, ‘missingValue’
purefc (int) – Switch for definition of pure forecast mode or not.
dtime (str) – Time step in hours.
index_vals (list of list of str) – Contains the values from the keys used for a distinct selection of GRIB messages in processing the grib files. Content looks like e.g.: index_vals[0]: (‘20171106’, ‘20171107’, ‘20171108’) ; date index_vals[1]: (‘0’, ‘1200’, ‘1800’, ‘600’) ; time index_vals[2]: (‘0’, ‘12’, ‘3’, ‘6’, ‘9’) ; stepRange
start_date (str) – The start date of the retrieval job.
end_date (str) – The end date of the retrieval job.
- Returns
(ix, jy, it) – Dimension in x-direction, y-direction and in time.
- Return type
tuple of int
-
tools.
get_informations
(filename)[source]¶ Extracts basic information from a sample grib file.
This information is needed for later use and the initialization of numpy arrays where data are stored.
- Parameters
filename (str) – Name of the file which will be opened to extract basic information.
- Returns
data – Contains basic informations of the ECMWF grib files, e.g. ‘Ni’, ‘Nj’, ‘latitudeOfFirstGridPointInDegrees’, ‘longitudeOfFirstGridPointInDegrees’, ‘latitudeOfLastGridPointInDegrees’, ‘longitudeOfLastGridPointInDegrees’, ‘jDirectionIncrementInDegrees’, ‘iDirectionIncrementInDegrees’, ‘missingValue’
- Return type
dict
-
tools.
get_list_as_string
(list_obj, concatenate_sign=', ')[source]¶ Converts a list of arbitrary content into a single string using a given concatenation character.
- Parameters
list_obj (list of *) – A list with arbitrary content.
concatenate_sign (str, optional) – A string which is used to concatenate the single list elements. Default value is “, “.
- Returns
str_of_list – The content of the list as a single string.
- Return type
str
-
tools.
init128
(filepath)[source]¶ Opens and reads the grib file with table 128 information.
- Parameters
filepath (str) – Path to file of ECMWF grib table number 128.
- Returns
table128 – Contains the ECMWF grib table 128 information. The key is the parameter number and the value is the short name of the parameter.
- Return type
dict
-
tools.
make_dir
(directory)[source]¶ Creates a directory.
If the directory already exists, an information is printed and the creation skipped. The program stops only if there is another problem.
- Parameters
directory (str) – The path to directory which should be created.
-
tools.
my_error
(message='ERROR')[source]¶ Prints a specified error message which can be passed to the function before exiting the program.
- Parameters
message (str, optional) – Error message. Default value is “ERROR”.
-
tools.
none_or_int
(value)[source]¶ Converts the input string into Pythons None-type if it contains string “None”; otherwise it is converted to an integer value.
- Parameters
value (str) – String to be checked for the “None” word.
- Returns
Return depends on the content of the input value. If it was “None”, then the python type None is returned. Otherwise the string is converted into an integer value.
- Return type
None or int(value)
-
tools.
none_or_str
(value)[source]¶ Converts the input string into Pythons None type if it contains the string “None”.
- Parameters
value (str) – String to be checked for the “None” word.
- Returns
Return depends on the content of the input value. If it was “None”, then the Python type None is returned, otherwise the string itself.
- Return type
None or value
-
tools.
normal_exit
(message='Done!')[source]¶ Prints a specific exit message which can be passed to the function.
- Parameters
message (str, optional) – Message for exiting program. Default value is “Done!”.
-
tools.
product
(*args, **kwds)[source]¶ Creates combinations of all passed arguments.
This method combines the single characters of the passed arguments with each other in a way that each character of each argument value will be combined with each character of the other arguments as a tuple.
Note
This method is taken from an example at the ECMWF wiki website. https://software.ecmwf.int/wiki/display/GRIB/index.py; 2018-03-16
It was released under the following license: https://confluence.ecmwf.int/display/ECC/License
Example
product(‘ABCD’, ‘xy’) –> Ax Ay Bx By Cx Cy Dx Dy
product(range(2), repeat = 3) –> 000 001 010 011 100 101 110 111
- Parameters
*args (list or str) – Positional arguments (arbitrary number).
**kwds (dict) – Contains all the keyword arguments from *args.
- Returns
prod – Return will be done with “yield”. A tuple of combined arguments. See example in description above.
- Return type
tuple
-
tools.
put_file_to_ecserver
(ecd, filename, target, ecuid, ecgid)[source]¶ Uses the ecaccess-file-put command to send a file to the ECMWF servers.
Note
The return value is just for testing reasons. It does not have to be used from the calling function since the whole error handling is done in here.
- Parameters
ecd (str) – The path were the file is stored.
filename (str) – The name of the file to send to the ECMWF server.
target (str) – The target queue where the file should be sent to.
ecuid (str) – The user id on ECMWF server.
ecgid (str) – The group id on ECMWF server.
-
tools.
read_ecenv
(filepath)[source]¶ Reads the file into a dictionary where the key values are the parameter names.
- Parameters
filepath (str) – Path to file where the ECMWF environment parameters are stored.
- Returns
envs – Contains the environment parameter ecuid, ecgid, gateway and destination for ECMWF server environments.
- Return type
dict
-
tools.
send_mail
(users, success_mode, message)[source]¶ Prints a specific exit message which can be passed to the function.
- Parameters
users (list of str) – Contains all email addresses which should be notified. It might also contain just the ecmwf user name which wil trigger mailing to the associated email address for this user.
success_mode (str) – States the exit mode of the program to put into the mail subject line.
message (str, optional) – Message for exiting program. Default value is “Done!”.
-
tools.
setup_controldata
()[source]¶ Collects, stores and checks controlling arguments from command line, CONTROL file and ECMWF_ENV file.
- Returns
c (ControlFile) – Contains all the parameters of CONTROL file and command line.
ppid (str) – Parent process id.
queue (str) – Name of queue for submission to ECMWF (e.g. ecgate or cca )
job_template (str) – Name of the job template file for submission to ECMWF server.
-
tools.
silent_remove
(filename)[source]¶ Remove file if it exists. The function does not fail if the file does not exist.
- Parameters
filename (str) – The name of the file to be removed without notification.
-
tools.
submit_job_to_ecserver
(target, jobname)[source]¶ Uses ecaccess-job-submit command to submit a job to the ECMWF server.
Note
The return value is just for testing reasons. It does not have to be used from the calling function since the whole error handling is done in here.
- Parameters
target (str) – The target where the file should be sent to, e.g. the queue.
jobname (str) – The name of the jobfile to be submitted to the ECMWF server.
- Returns
job_id – The id number of the job as a reference at the ECMWF server.
- Return type
int
-
tools.
to_param_id
(pars, table)[source]¶ Transform parameter names to parameter ids with ECMWF grib table 128.
- Parameters
pars (str) – Addpar argument from CONTROL file in the format of parameter names instead of IDs. The parameter short names are separated by “/” and passed as one single string.
table (dict) – Contains the ECMWF grib table 128 information. The key is the parameter number and the value is the short name of the parameter.
- Returns
ipar – List of addpar parameters from CONTROL file transformed to parameter ids in the format of integer.
- Return type
list of int
-
tools.
to_param_id_with_tablenumber
(pars, table)[source]¶ Transform parameter names to parameter IDs and add table ID.
Conversion with ECMWF grib table 128.
- Parameters
pars (str) – Addpar argument from CONTROL file in the format of parameter names instead of ID. The parameter short names are separated by “/” and passed as one single string.
table (dict) – Contains the ECMWF grib table 128 information. The key is the parameter number and the value is the short name of the parameter.
- Returns
spar – List of addpar parameters from CONTROL file transformed to parameter IDs in the format of integer.
- Return type
str
checks¶
This module contains check methods for the CONTROL paramaters.
-
checks.
check_accmaxstep
(accmaxstep, marsclass, purefc, maxstep)[source]¶ Guarantees that the accumulation forecast step was set.
- Parameters
accmaxstep (str) – The maximum forecast step for the accumulated forecast fields.
marsclass (str) – ECMWF data classification identifier.
purefc (int) – Switch for definition of pure forecast mode or not.
maxstep (str) – The maximum forecast time step in hours from the forecast base time. This is the maximum step for non-flux (accumulated) forecast data.
- Returns
accmaxstep – The maximum forecast step for the accumulated forecast fields.
- Return type
str
-
checks.
check_acctime
(acctime, marsclass, purefc, time)[source]¶ Guarantees that the accumulation forecast times were set.
If not set, setting the value to some of the most commonly used data sets is attempted. Otherwise, an eror is raised.
- Parameters
acctime (str) – The starting time for the accumulated forecasts.
marsclass (str) – ECMWF data classification identifier.
purefc (int) – Switch for definition of pure forecast mode or not.
- Returns
acctime – The starting time for the accumulated forecasts.
- Return type
str
-
checks.
check_acctype
(acctype, ftype)[source]¶ Guarantees that the accumulation field type is set.
If not set, it is derived with the old method (TYPE[1]).
- Parameters
acctype (str) – The field type for the accumulated forecast fields.
ftype (list of str) – List of field types.
- Returns
acctype – The field type for the accumulated forecast fields.
- Return type
str
-
checks.
check_addpar
(addpar)[source]¶ Check that addpar has correct format of additional parameters in a single string, so that it can be easily appended to the hard-coded parameters retrieved in any case.
- Parameters
addpar (str or list of str) – List of additional parameters to be retrieved.
- Returns
addpar – List of additional parameters to be retrieved.
- Return type
str
-
checks.
check_area
(grid, area, upper, lower, left, right)[source]¶ Defines the correct area string.
Checks the format of the four area components wether it is on the order of 1000 or 1. Also checks wether area was already set on command line, then the four components are overwritten. Convert to correct format of the order of magnitude “1” and sets the area parameter (North/West/South/East). E.g.: -5./20./10./10.
- Parameters
grid (str) – Contains grid information.
area (str) – Contains area informtion.
upper (str) – The northernmost latitude.
lower (str) – The southernmost latitude.
left (str) – The westernmost longitude.
right (str) – The easternmost longitude.
- Returns
grid – Contains grid in format lat/lon. E.g. 0.1/0.1
- Return type
str
-
checks.
check_basetime
(basetime)[source]¶ Check if basetime is set and contains one of the two possible values (0, 12).
- Parameters
basetime (int or str or None) – The time for a half day retrieval. The 12 hours upfront are to be retrieved.
- Returns
basetime – The time for a half day retrieval. The 12 hours upfront are to be retrieved.
- Return type
int or None
-
checks.
check_dates
(start, end)[source]¶ Checks if there is at least a start date for a one-day retrieval.
Checks if end date lies after start date and end date is set.
- Parameters
start (str) – The start date of the retrieval job.
end (str) – The end date of the retrieval job.
- Returns
start (str) – The start date of the retrieval job.
end (str) – The end date of the retrieval job.
-
checks.
check_grid
(grid)[source]¶ Convert grid into correct Lat/Lon format. E.g. ‘0.5/0.5’
Checks on format of original grid. Wether it is in the order of 1000 or 1. Convert to correct grid format and substitute into “Lat/Lon” format string.
- Parameters
grid (str) – Contains grid information
- Returns
grid – Contains grid in format Lat/lon. E.g. 0.1/0.1
- Return type
str
-
checks.
check_job_chunk
(job_chunk)[source]¶ Checks that if job chunk is set, the number is positive and nonzero.
- Parameters
job_chunk (int) – The number of days for a single job script.
- Returns
job_chunk – The number of days for a single job script.
- Return type
int
-
checks.
check_len_type_time_step
(ftype, ftime, steps, maxstep, purefc)[source]¶ Check if
- Parameters
ftype (list of str) – List of field types.
ftime (list of str or str) – The time in hours of the field.
steps (str) – Specifies the forecast time step from forecast base time. Valid values are hours (HH) from forecast base time.
maxstep (int) – The maximum forecast time step in hours from the forecast base time. This is the maximum step for non-flux (not accumulated) forecast data.
purefc (int) – Switch for definition of pure forecast mode or not.
- Returns
ftype (list of str) – List of field types.
ftime (list of str) – The time in hours of the field.
steps (str) – Specifies the forecast time step from forecast base time. Valid values are hours (HH) from forecast base time.
-
checks.
check_levels
(levelist, level)[source]¶ Defines correct level list and guarantees that the maximum level is one of the available maximum levels.
- Parameters
levelist (str) – Specifies the level list. Examples: model level: 1/to/137, pressure levels: 500/to/1000
level (str) – Specifies the maximum level.
- Returns
levelist (str) – Specifies the required levels. It has to have a valid correspondence to the selected levtype. Examples: model level: 1/to/137, pressure levels: 500/to/1000
level (str) – Specifies the maximum level. It has to be one of the available maximum level numbers as contained in the variable MAX_LEVEL_LIST in “_config”: [16, 19, 31, 40, 50, 60, 62, 91, 137]
-
checks.
check_logicals_type
(c, logicals)[source]¶ Check that the logical variables have correct type integer.
- Parameters
c (ControlFile) – Contains all the parameters of CONTROL file and command line.
logicals (list of (str or int)) – Names of the switches that are used to control the flow of the program.
-
checks.
check_mail
(mail)[source]¶ Check the string of mail addresses, separate them and convert to list.
- Parameters
mail (list of str or str) – Contains email addresses for notifications. It might also contain just the ecmwf user name which will trigger mailing to the associated email address for this user.
- Returns
mail – Contains email addresses for notifications. It might also contain just the ecmwf user name which will trigger mailing to the associated email address for this user.
- Return type
list of str
-
checks.
check_maxstep
(maxstep, steps)[source]¶ Convert maxstep into integer if it is already given. Otherwise, select maxstep by going through the steps list.
- Parameters
maxstep (str) – The maximum forecast time step in hours from the forecast base time. This is the maximum step for non flux (accumulated) forecast data.
steps (str) – Specifies the forecast time step from forecast base time. Valid values are hours (HH) from forecast base time.
- Returns
maxstep – The maximum forecast time step in hours from the forecast base time. This is the maximum step for non flux (accumulated) forecast data.
- Return type
int
-
checks.
check_number
(number)[source]¶ Check for correct string format of ensemble member numbers.
- Parameters
number (str) – List of ensemble member forecast runs.
- Returns
number – String with list of ensemble member forecast runs. E.g. ‘01/02/03/04’
- Return type
str
-
checks.
check_pathes
(idir, odir, fpdir, fedir)[source]¶ Check if output and flexpart pathes are set.
- Parameters
idir (str) – Path to the temporary directory for MARS retrieval data.
odir (str) – Path to the final output directory where the FLEXPART input files will be stored.
fpdir (str) – Path to FLEXPART root directory.
fedir (str) – Path to flex_extract root directory.
- Returns
odir (str) – Path to the final output directory where the FLEXPART input files will be stored.
fpdir (str) – Path to FLEXPART root directory.
-
checks.
check_ppid
(c, ppid)[source]¶ Sets the current PPID.
- Parameters
c (ControlFile) – Contains all the parameters of CONTROL file and command line.
ppid (int or None) – Contains the ppid number provided by the command line parameter of is None otherwise.
-
checks.
check_public
(public, dataset, marsclass)[source]¶ Check wether the dataset parameter is set to a public data set.
- Parameters
public (int) – Specifies if public data are to be retrieved or not.
dataset (str) – Specific name which identifies the public dataset.
-
checks.
check_purefc
(ftype)[source]¶ Check for a pure forecast mode.
- Parameters
ftype (list of str) – List of field types.
- Returns
True if pure forecasts are to be retrieved. False if there are analysis fields in between.
- Return type
True or False
-
checks.
check_queue
(queue, gateway, destination, ecuid, ecgid)[source]¶ Check if the necessary ECMWF parameters are set if the queue is one of the QUEUES_LIST (in _config).
- Parameters
queue (str) – Name of the queue if submitted to the ECMWF servers. Used to check if ecuid, ecgid, gateway and destination are set correctly and are not empty.
gateway (str) – The address of the gateway server.
destination (str) – The name of the destination of the gateway server for data transfer through ectrans. E.g. name@genericSftp
ecuid (str) – ECMWF user id.
ecgid (str) – ECMWF group id.
-
checks.
check_request
(request, marsfile)[source]¶ Check if there is an old MARS request file; if so, remove it.
- Parameters
request (int) – Selects the mode of retrieval. 0: Retrieves the data from ECMWF. 1: Prints the mars requests to an output file. 2: Retrieves the data and prints the mars request.
marsfile (str) – Path to the mars request file.
-
checks.
check_step
(step)[source]¶ Checks on step format and convert into a list of steps.
If the steps were defined with “to” and “by” they are converted into a list of steps. If the steps were set in a string, it is converted into a list.
- Parameters
step (list of str or str) – Specifies the forecast time step from forecast base time. Valid values are hours (HH) from forecast base time.
- Returns
step – List of forecast steps in format e.g. [001, 002, …]
- Return type
list of str
-
checks.
check_time
(ftime)[source]¶ Check if time variable is of type list. Otherwise convert to list.
- Parameters
ftime (list of str or str) – The time in hours of the field.
- Returns
ftime – The time in hours of the field.
- Return type
list of str
-
checks.
check_type
(ftype, steps)[source]¶ Check if type variable is of type list and if analysis field has forecast step 0.
- Parameters
ftype (list of str or str) – List of field types.
steps (str) – Specifies the forecast time step from forecast base time. Valid values are hours (HH) from forecast base time.
- Returns
ftype – List of field types.
- Return type
list of str
disaggregation¶
Disaggregation of deaccumulated flux data from an ECMWF model FG field.
- Initially the flux data to be concerned are:
large-scale precipitation
convective precipitation
surface sensible heat flux
surface solar radiation
u stress
v stress
Different versions of disaggregation is provided for rainfall data (darain, modified linear) and the surface fluxes and stress data (dapoly, cubic polynomial).
-
disaggregation.
IA3
(g)[source]¶ Interpolation with a non-negative geometric mean based algorithm.
The original grid is reconstructed by adding two sampling points in each data series interval. This subgrid is used to keep all information during the interpolation within the associated interval. Additionally, an advanced monotonicity filter is applied to improve the monotonicity properties of the series.
Note
(C) Copyright 2017-2019 Sabine Hittmeir, Anne Philipp, Petra Seibert
This work is licensed under the Creative Commons Attribution 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ or send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.
- Parameters
g (list of float) – Complete data series that will be interpolated having the dimension of the original raw series.
- Returns
f – The interpolated data series with additional subgrid points. Its dimension is equal to the length of the input data series times three.
- Return type
list of float
References
For more information see article: Hittmeir, S.; Philipp, A.; Seibert, P. (2017): A conservative interpolation scheme for extensive quantities with application to the Lagrangian particle dispersion model FLEXPART., Geoscientific Model Development
-
disaggregation.
dapoly
(alist)[source]¶ Cubic polynomial interpolation of deaccumulated fluxes.
Interpolation of deaccumulated fluxes of an ECMWF model FG field using a cubic polynomial solution which conserves the integrals of the fluxes within each timespan. Disaggregation is done for 4 accumluated timespans which generates a new, disaggregated value which is output at the central point of the 4 accumulation timespans. This new point is used for linear interpolation of the complete timeseries afterwards.
- Parameters
alist (list of array of float) – List of 4 timespans as 2-dimensional, horizontal fields. E.g. [[array_t1], [array_t2], [array_t3], [array_t4]]
- Returns
nfield – Interpolated flux at central point of accumulation timespan.
- Return type
array of float
Note
- March 2000P. JAMES
Original author
- June 2003A. BECK
Adaptations
- November 2015Leopold Haimberger (University of Vienna)
Migration from Fortran to Python
-
disaggregation.
darain
(alist)[source]¶ Linear interpolation of deaccumulated fluxes.
Interpolation of deaccumulated fluxes of an ECMWF model FG rainfall field using a modified linear solution which conserves the integrals of the fluxes within each timespan. Disaggregation is done for 4 accumluated timespans which generates a new, disaggregated value which is output at the central point of the 4 accumulation timespans. This new point is used for linear interpolation of the complete timeseries afterwards.
- Parameters
alist (list of array of float) – List of 4 timespans as 2-dimensional, horizontal fields. E.g. [[array_t1], [array_t2], [array_t3], [array_t4]]
- Returns
nfield – Interpolated flux at central point of accumulation timespan.
- Return type
array of float
Note
- March 2000P. JAMES
Original author
- June 2003A. BECK
Adaptations
- November 2015Leopold Haimberger (University of Vienna)
Migration from Fortran to Python