ERA 5 download script UEMS - ds633.0

Questions and other topics related to UEMS 18.
Post Reply
marquezchile
Posts: 2
Joined: Fri Apr 17, 2020 12:21 am

ERA 5 download script UEMS - ds633.0

Post by marquezchile » Fri Apr 17, 2020 1:08 am

Hi all,

I have uems v19.8 and I have a problem using the download_rda_ds630.0_era5 script, it cannot find the files because they are not available. Probably because they changed the repository from 630.0 to 633.0 (https://rda.ucar.edu/datasets/ds630.0/).
Please note: Please use ds633.0 to access RDA maintained ERA-5 data, see ERA5 Reanalysis (0.25 Degree Latitude-Longitude Grid), RDA dataset ds633.0. This dataset is no longer being updated, and web access has been removed.
Does anyone have an updated script to download the files?

I will take advantage of asking about the possibility of using this dataset (https://rda.ucar.edu/datasets/ds084.1/) to carry out an annual historical analysis with UEMS. It's possible?


Thanks for your time and any help is welcome.
Very grateful for all the information provided by this forum.

Camilo

Canuck64
Posts: 41
Joined: Tue Sep 04, 2012 10:41 pm
Contact:

Re: ERA 5 download script UEMS - ds633.0

Post by Canuck64 » Sat Apr 25, 2020 2:00 am

Hello Camilo,

I haven't tested this out yet myself, but I think you can make a change in the {your_directory}/uems/util/extras/era5/download_rda_ds630.0_era5.pl file. Near the bottom of the script, there is a line for the site, starting with 'http'. Change the line from 'ds630.0' to 'ds633.0'.

As I said it may work, but after reading the notes at the UCAR RDA 633.0 website, mention is made that only the NETCDF/HDF5 version is available on-line, and that the grib1 version is stored on an internal server. In which case, this script will require significant modifications, and then you'll need software to convert the netdf/hdf5 files to grib1. Not an easy task.

Cheers!

Brian

smartie
Posts: 100
Joined: Sat May 21, 2011 7:34 am

Re: ERA 5 download script UEMS - ds633.0

Post by smartie » Mon Apr 27, 2020 7:00 pm

An easy way to get ERA5 on pressure levels is to:
-register for a free account at CDS
https://cds.climate.copernicus.eu/#!/home
Download and install the CDS API
https://confluence.ecmwf.int/display/CK ... nload+ERA5
You can then download via Python scripts running in a terminal, examples for pressure levels in Grib format:

Code: Select all

#Download sub-setted ERA5 pressure level data from CDS to initialise WRF NAE12 domain every 3hr.

import cdsapi

c = cdsapi.Client()

c.retrieve(
    'reanalysis-era5-pressure-levels',
    {
        'product_type':'reanalysis',
        'format':'grib',
        'variable':[
            'geopotential','relative_humidity','temperature',
            'u_component_of_wind','v_component_of_wind'
        ],
        'pressure_level':[
            '50','100','150',
            '200','250','300',
            '350','400','450',
            '500','550','600',
            '650','700','750',
            '775','800','825',
            '850','875','900',
            '925','950','975',
            '1000'
        ],
# Area: North/West/South/East
        'area':'80/-90/20/45',
        'year':'2014',
        'month':'02',
        'day':['11','12','13'],
        'time':['00:00','03:00','06:00','09:00','12:00','15:00','18:00','21:00'
        ],                    
    },
    'gribprs.grib')
and surface...

Code: Select all

#Download sub-setted ERA5 surface level data from CDS to initialise WRF NAE12 domain every 3hr.

import cdsapi

c = cdsapi.Client()

c.retrieve(
    'reanalysis-era5-single-levels',
    {
        'product_type':'reanalysis',
        'format':'grib',
        'variable':[
            '10m_u_component_of_wind','10m_v_component_of_wind','2m_dewpoint_temperature',
            '2m_temperature','land_sea_mask','mean_sea_level_pressure',
            'sea_ice_cover','sea_surface_temperature','skin_temperature',
            'snow_density','snow_depth','soil_temperature_level_1',
            'soil_temperature_level_2','soil_temperature_level_3','soil_temperature_level_4',
            'surface_pressure','volumetric_soil_water_layer_1','volumetric_soil_water_layer_2',
            'volumetric_soil_water_layer_3','volumetric_soil_water_layer_4'
        ],
# Area: North/West/South/East
        'area':'80/-90/20/45',
        'year':'2014',
        'month':'02',
        'day':['11','12','13'],
        'time':['00:00','03:00','06:00','09:00','12:00','15:00','18:00','21:00'
        ],                    
    },
    'gribsfc.grib')
modify pressure levels/area/date/time fields to your needs or call variables from a 'control' script. i need only 3hr data on selected levels.
Join the the two files together-

Code: Select all

cat gribprs.grib gribsfc.grib  > ec.grb 
then subset the bulk file into individual time files

Code: Select all

ecmwf_subset_dates.pl ec.grb
this script used to be included in EMS (I don't if it still is) for ERA Interim but still works on this data.

Code: Select all

#!/usr/bin/perl
#
#  This routine subsets the ECMWF ERA Interim data set into individual
#  grib files according to date/time, which is what the WRF EMS expects.
#
#  If you are interested in using these data for initializing the WRF EMS you will
#  need to do the following:
#
#  1/2. Access the ECMWF ERA Interim data set (1.5 degree 4x daily analyses):
#
#         http://data-portal.ecmwf.int/data/d/interim_daily
#
#       All data should be in grib 1 format.
#
#    1. Obtain the following PRESSURE LEVEL data sets (all levels) for all
#       available analysis times (00, 06, 12, 18 UTC)
#
#       a. Geopotential
#       b. Temperature
#       c. U-Wind
#       d. V-Wind
#       e. Relative Humidity
#
#    2. Obtain the following SURFACE LEVEL data sets for the same analysis
#       times (00, 06, 12, 18 UTC)
#
#       * Note that all necessary fields are available in the "Step 0"
#         group (analysis time). You may simply select "Step 0" and then
#         "Select All" at the bottom of the Parameters window.
#
#       a. 10m U-Wind
#       b. 10m V-wind
#       c. 2m Temperature
#       d. 2m Dewpoint Temperature
#       e. Sea-level Pressure
#       f. Surface Pressure
#       g. Sea-Surface Temperature
#       h. Skin Temperature
#       i. Sea Ice
#       j. Water Equivalent of Accumulated Snow Depth
#       k. All 4 leves of Soil Temperature
#       l. All 4 levels of soil moisture
#
#    3. After downloading, combine the 2 grib 1 data sets, combine into a single data set:
#
#         % cat dataset1 dataset2 >  ecmwfdata.grb
#
#    4. Run this file passing ecmwfdata.grb as the lone argument:
#
#         % subset_dates_ecmwf_grib1.pl ecmwfdata.grb
#
#       Running the above command assumes that you have EMS EMS installed and have
#       access to the wgrib routine.
#
#    5. The single monolithic fine will be subet into individual grib 1 files for
#       each date/time in the file. The WRF EMS is configured to look for these
#       individual files at the specified location.
#
#    6. Edit the wrfems/conf/grib_info/ecmwf_gribinfo.conf file and make sure that
#       the location of the grib files are correctly specified by a line in the
#       SERVERS section. The line should start with:
#
#         SEVERS-NFS  = LOCAL:<path to files>/<file naming convention>
#
#       *  Note that the file naming convention must use "HH" rather then "FF"
#          for the analysis hour.
#
#    7. When running ems_prep be sure to include the "--analysis" option, which tells
#       the routine that this is a series of analysis rather than forecasts.
#
#         % ems_prep --dset ecmwf::local  --date <your date> --cycle <start hour> --length <hours> --analysis
#
#---------------------------------------------------------------------------------------------------
#
use strict;

    my $tdir = "."; # Change target directory as necessary
#   my $tdir = "/data/ecmwf/$yr/$mo";

    print "\n\n  WRF EMS ECMWF ERA Interim data subsetting routine\n";
    my $grib   = shift @ARGV;
    exit unless -e $grib;

    #  Start by extracting only the analysis fields from the original
    #  GRIB 1 file.
    #   
    #  Now get the vertfication dates in the file
    #
    print "\n    Step 1. Extracting analysis fields from $grib - This step may take a while\n";
    `wgrib -s $grib | grep \":anl:\"  | wgrib -i -grib $grib -o newgrib`;

    #  Now collect all the dates/time in the grib file. These will be used
    #  in the subsetting of the file.
    #
    print "\n    Step 2. Collecting verification date and times in file\n";
    my %dates;
    foreach (`wgrib -verf -4yr newgrib | cut -d":" -f3`) {
        chomp;
        s/d=//g;
        substr $_, 5, 8;
        $dates{$_} = 1;
    }

    #  Now loop through each date/time and extract only those fields that
    #  are valid at that time. Write to new grib 1 file.
    #
    #  File naming convention is YYYYMMDDHH.ecmwf.grb
    #
    print "\n    Step 3. Processing into individual grib files\n\n";
    foreach my $date (sort keys %dates) {
        my $yr  = substr $date, 0, 4;
        my $mo  = substr $date, 4, 2;
        my $fd  = substr $date, 2, 8;
        my $nf  = "$date.ecmwf.grb";
        system "mkdir -p $tdir >& /dev/null";
        print "            * Creating : $tdir/$nf\n";
#       print "wgrib -verf -4yr newgrib | grep \"d=$date\" | wgrib -i -grib newgrib -o $tdir/$nf\n";
        `wgrib -verf -4yr newgrib | grep \"d=$date\" | wgrib -i -grib newgrib -o $tdir/$nf`;
    }
    system "rm -f newgrib";

print "\n  The $0 routine is complete - Now go model something!\n\n"; 
exit;
Put the data in a structured directory. Create a new ERA5 grib info file pointing to the ecmwf on pressure levels vtable and to the data folders. You can then call this-

Code: Select all

ems_prep --dset era5 --analysis ...
In my experience the CDS system is very fast- serving up a a few days data in minutes.
Last edited by smartie on Mon Apr 27, 2020 7:35 pm, edited 3 times in total.

smartie
Posts: 100
Joined: Sat May 21, 2011 7:34 am

Re: ERA 5 download script UEMS - ds633.0

Post by smartie » Mon Apr 27, 2020 7:08 pm

I've put the required scripts here-
https://www.dropbox.com/s/rdlal9a51vum0 ... ar.gz?dl=0

JMeans
Posts: 23
Joined: Wed Jan 16, 2013 7:34 am

Re: ERA 5 download script UEMS - ds633.0

Post by JMeans » Mon Apr 27, 2020 11:17 pm

There are more changes required to the download and process files than simply a change in the directory from 630 to 633. The file names have also changed somewhat, and the 2006 invariant files are not available (as far as I can tell), so you can use the 1979 ones instead. You need to have both wgrib and wgrib2 in the directory for it to work. I have sent updated files to Robert Rozumalski, so maybe they'll be included in the next release.

As for the approach using the Copernicus Climate Data Store, I couldn't get the API to work, but the code did work in their online GUI. Unfortunately I didn't know what else I needed to do with the data (the processing that smartie outlined) and ems_prep failed on the unprocessed data. Robert's perl scripts work well, if you make the file name an directory changes.

Post Reply