APPENDIX
Meteorological Archival and Retrieval System (MARS) is an archiving system for meteorological data developed at ECMWF.
At ECMWF both observations and model data are handled, here we only take care of model data in GRIB. For further reference see the documentation at ECMWF.
The mars archive is now accessible from all nodes on dunder, gimle, bore, tornado and vagn. Since the conversion from HIRLAM to MARS request files may consume large amounts of system memory on the computer, we now recommend doing large archiving jobs on cluster nodes where Out-of-Memory incidents do not affect so many users. Instructions on how to allocate nodes for interactive work can be found in the user guides of the respective clusters.
Also, for usage statistics you can check out the stats page.
For support please use mars-support@nsc.liu.se.
First make sure you have all MARS components loaded by
module load mars
Next you will need to generate data to submit when you apply for an account described in the following section.
First time you contact MARS a pair of SSH keys will be generated. These keys are used as the means for authentication on the MARS servers. The most convenient way to initiate this first
contact is described in the following section, but as a spoiler for the
impatient, this can be accomplished by issuing "expid --class=re -l" or "mars_verify_auth".
Following this first contact you will get on-screen instructions on how to proceed to get an account on the MARS servers, and thereby access to the MARS service.
If you are getting a MARS account for a non-NSC resource, such as for instance
Ekman at PDC, you have two options when connecting to the MARS servers: Either
set the environment variable MARS_USER on that resource to the user name you
have on the MARS servers, for instance
export MARS_USER=my_user_name_on_the_mars_servers
in bash, or add your user name when you call the servers directly, that is
mars --ssh-user=my_user_name_on_the_mars_servers ...
Since MARS does not store any ownership details within its database, an experiment id is used to distinguish between the data owned by different users. This experiment id is exactly 4 characters long and may consist of any combination of the digits 0-9 and the ASCII character subset a-v.
There are three things especially noteworthy regarding the experiment id's
- An experiment id is only unique to a certain MARS database, therefore the MARS class must always be specified as a mandatory argument.
- An allocated id is stored on a MARS server together with the user,
date created and a supplied (up to 43 character long) description of
the id.
- An allocated id can not be altered/removed by any other than the owner of the id.
Some examples:
- List configured MARS classes:
expid --list--classes
- Allocate a new experiment id for the server that maintains the
MARS class re:
expid --class=re --new --description="This is a description of my experiment id"
- Change description for a previously allocated id 1234:
expid --class=re --new --id=1234 --description="Not used anymore"
- Remove a previously allocated id:
expid --class=re --remove --id=1234
- List all allocated id:s for the server that maintains the MARS class re:
expid --class=re --list
- List all allocated id:s for the server that maintains the MARS class re by the user exampleuser
expid --class=re --list --user=exampleuser
- Check that this experiment ID is really yours ( bash syntax )
expid -l -u $USER --class=re --id=test | grep -q "test $USER" || { echo "test is not your experiment in the database list" ; expid -l -u $USER --class=re ; exit 1 ; }
The meta data used in MARS has to be stored in the header of the GRIB files as well as in the data base of the archive.
Therefore the meta data for your particular data has to be written in the header of your GRIB file using a filter, hirmar.x.
The marsfilter needs at minimum the following information:
|
EXPNAME, |
as generated by expid |
|
CLASS, |
defines the class you belong to |
|
STREAM, |
type of data |
|
TYPE, |
type of integration |
|
MODEL, |
type of model
|
A full list of options available for the filter is given by hirmar.x -h
In general we would set CLASS=re, STREAM=oper, TYPE=fc for a forecast experiment. Read on about meta data valid for your model >>>
CLASS=op is restricted to operational use only. Please check you experiment ID carefully before you archive anything!
The keywords are given in a fortran namelist with the following content:
&namin
mars_exp ='test',
mars_class ='re',
mars_model ='hirlam',
mars_type ='fc',
mars_stream ='oper',
/
Call the filter by :
hirmar.x -n namelist_file GRIBFILE
The filter program scans through your GRIB file(s), adds the extra information to the header, and split the data to smaller units depending on e.g. level type. The reason for splitting your file into smaller pieces is that there has to be a one to one correspondence between the archive request and the file content. A file, archive_list, specifying you archive sequence is created. This is meant to be used when you run the actual archiving:
mars archive_list
You may include/exclude fields from your file by specifying include_field or exclude_field in the namelist.
Include/exclude one specific field:
*_field(1)%table = 001
*_field(1)%param = 011
*_field(1)%lev = 040
*_field(1)%levtype = 109
Include/exclude one specific levtype:
*_field(1)%table = -1
*_field(1)%param = -1
*_field(1)%lev = -1
*_field(1)%levtype = 109
Include/exclude all levels for a specific parameter and levtype
*_field(1)%table = 001
*_field(1)%param = 011
*_field(1)%lev = -1
*_field(1)%levtype = 109
Example:
&namin
include_field(1:2)%table = 001,001,
include_field(1:2)%param = 011,001,
include_field(1:2)%lev = -1,000,
include_field(1:2)%levtype = 109,105,
mars_exp ='test',
mars_class ='re',
mars_model ='hirlam',
mars_type ='fc',
mars_stream ='oper',
/
For general information on how to retrieve data from MARS read the ECMWF user guide: http://www.ecmwf.int/publications/manuals/mars/
The marsfilter creates a mirror file to the archive_list, retrieve_list. This may be used to extract your recently archived data.
mars retrieve_list
It is possible to list the content of an experiment by .e.g. the following MARS request
list,
CLASS = OP,
MODEL = hirlam,
TYPE = FC,
STREAM = DA,
EXPVER = 0001,
DATE = 20081029/to/20081030,
TIME = ALL,
STEP = ALL,
LEVTYPE = ALL,
LEVELIST = ALL,
PARAM = ALL
A template file for listing, list_mars, is created when you run your hirmar.x. This is a good start for browsing your archived data. Use as
mars list_mars
It is not possible for a user to remove data from the data himself. There will be a yearly review of the content of the archive.
For MARS to work the username supplied from the client must match that recorded on the server, meaning that from non-NSC sites you will have to supply your NSC username on the command line when the mars client is invoked.
There are some syntax differences between ECMWF and NSC on how a MARS request is written. The differences are introduced to allow a more general use of data.
At ECMWF all surface data are archived as LEVTYPE=sfc with the only valid LEVEL=0. In e.g. HIRLAM we have different levels for LEVTYPE=105. In order not to have to change the original data we have introduced handling of any leveltype. The archive/retrieve syntax has therefore changed:
LEVTYPE=pl -> LEVTYPE=100
and so on. Note that LEVTYPE is still handled as a string in MARS. i.e. "008" is not the same as "8".
At ECMWF different model parameters can be retrieved using real names, like TEMPERATURE or MSLP. Since this names are linked to the ECMWF usaged of GRIB table 2 version 128 we have to stick to the parameter numbers instead. A field is then referred to with it's parameter number and table version such as
PARAMETER = 011.001,033.011,034.001,...
for e.g. temperature and wind components.
|
Server |
Database |
Identification |
User group |
|
Mars2 |
mars2.nsc.liu.se |
CLASS=op |
Operational data |
|
Mars1 |
mars1.nsc.liu.se |
CLASS=re/ECMWF_CLASSES |
Research and re-archiving of ECMWF data
|
|
CLASS |
MODEL |
STREAM |
TYPE |
|
op |
HIRLAM |
oper |
an |
|
re |
RCA |
mnth |
fc |
|
MATCH |
moda |
3v |
|
HIROMB |
mdfa |
4v |
|
HARMONIE |
dame |
si |
|
IFS |
secy |
|
|
|
seme |
|
|
|
|
|
|
* |
ECMWF-IFS |
* |
* |
|
TYPE |
GRIB_CODE |
Meaning |
|
an |
02 |
Analysis |
|
3v |
05 |
3DVAR analysis |
|
4v |
06 |
4DVAR analys |
|
fc |
09 |
Forecast |
|
si |
21 |
Climate simulation |
|
CLASS |
GRIB_CODE |
Description |
|
op |
80 |
SMHI operations |
|
re |
81 |
SMHI research |
|
od |
01 |
ECMWF operations |
|
rd |
02 |
ECMWF research department |
|
STREAM |
GRIB_CODE |
Description |
|
oper |
1025 |
Operational deterministic model daily archive |
|
mnth |
1043 |
Monthly means |
|
moda |
1071 |
Monthly means of daily means |
|
mdfa |
1075 |
Monthly means of daily forecast accumulations |
|
dame |
4000 |
Daily means |
|
seme |
4001 |
Seasonal means |
|
secy |
4002 |
Seasonal daily cycle |
|
MODEL |
GRIB_CODE |
Description |
|
HIRLAM |
30 |
NWP LAM |
|
RCA |
31 |
Climate model |
|
HIROMB |
40 |
Ocean model |
|
MATCH |
50 |
Atmospheric dispersion model |
|
HARMONIE |
20 |
NH-NWP LAM |
|
IFS |
10 |
Global NWP |
|
MESAN |
50 |
Mesoscale analysis system |
|
SWAN |
60 |
Wave model |
To be able to handle the extra parameters stored in the grib headers to new local definition tables have been created
Make sure your LOCAL_DEFINITION_TABLES points to /usr/share/emos/gribtemplates/.
The new tables can be found under the directory defined by the environment $LOCAL_DEFINITION_TABLES. In the general case the changes does not effect your normal usage of the GRIB files. In the MATCH case, please contact Lennart Robertsson.
Use the "anytoany" utility to convert to an extended GRIB1
format including the ECMWF keys.
$ anytoany -i ingrib -ext -o outgrib
$ anytoany -i MATCH_2009051200+000H00M -ext -o MARS_2009051200+000H00M
You may find the anytoany utility on gimle/dunder at /home/sm_lrobe/work/metgraf/bin/anytoany,
or checkout metgraf and compile the utility:
$ co metgraf
$ cd metgraf/src
$ configure
$ make ../bin/anytoany
For further information about MATCH conversions please contact Lennart Robertson.
Page last modified: 2010-07-01 10:43
For more information contact us at
info@nsc.liu.se.
