How to get rid of gridpoint storms (WRF-NMM)

Compiling NMM/ARW code for EMS system, upgrading WRF cores on your own, changing EMS scripts to suit your particular needs, and other modifications to original EMS distribution goes in this forum. These are officially unsupported actions.
Post Reply
beowrf
Posts: 39
Joined: Sat Jul 23, 2011 3:07 pm

How to get rid of gridpoint storms (WRF-NMM)

Post by beowrf » Mon Nov 14, 2011 10:26 pm

Dear WRF-EMS-users,

Maybe you are also concerned, what concerns the well-known problem of so-called "grid point storms". They happen quite often (especially in instable atmospheric conditions, in summer) over steep orographic slopes.

What can be done? Well. You can either swith to WRF-ARW (which is slower) or modify WRF-NMM yourself manually.
I opted for the second solution as I was very happy about the modeled results of my WRF-NMM configuration (excepted the rain, of course).

This is a step-to-step approach, and this tutorial might be updated later on if things are unclear to modelers wanting to upgrade their WRF.

1. Go to the intel homepage and download the Fortran & C - Compilers. They are for free for non-commercial use (here http://software.intel.com/en-us/article ... -download/). Source the file called "ifortvars.csh" or "ifortvars.sh" located in the bin folder of the just installed Intel compiler.
You know that the IFC & ICC have been installed if you get no error message like "not found" if you type ifort or icc.
2. Go to the MPICH2 homepage (here http://www.mcs.anl.gov/research/project ... =downloads) and download + install the MPICH2 library & binaries. It is best practice to configure with ./configure --prefix=/opt/mpich2 so that you have your mpich2 libraries in one known folder. MPICH2 has been successfully installed if in /opt/mpich2/bin there are binaries like mpif90, mpicc or mpiexec. Add this folder to your $PATH.
3. Go and get the WRF-NMM v3.3 http://www.mmm.ucar.edu/wrf/src/WRFV3.3.TAR.gz and untar the file.
4. You know have to modify two source files of the WRF-NMM:

File 1: dyn_nmm/module_ADVECTION.F
Action : Average the vertical motion generally. This will decrease the intensity of your storms, but I did not see any alarming decrease.

Change

Code: Select all

LOGICAL,SAVE :: TRADITIONAL=.TRUE.
to

Code: Select all

LOGICAL,SAVE :: TRADITIONAL=.FALSE.
File 2: dyn_nmm/module_DIFFUSION_NMM.F
Action : Re-define the maximum allowable slope to the model surface for which lateral diffusion is permitted. Additionally, you will enable a weak background diffusion in points were topography is so steep that normally, there would be no horizontal diffusion at all normally.

Change around line 136

Code: Select all

SLOPHC=SLOPHT*SQRT(2.)*0.5*18
and also here (yes, twice):

Code: Select all

IF(SLOP<SLOPHC.OR.CILINE.OR.WATSLOP)THEN
SNE(I,J)=1.
ELSE
SNE(I,J)=0. ! ->> change to SNE(I,J)=0.1 ??
ENDIF


IF(SLOP<SLOPHC.OR.CILINE.OR.WATSLOP)THEN
SSE(I,J)=1.
ELSE
SSE(I,J)=0. ! -->> change to SSE(I,J)=0.1 ??
ENDIF
5. Re-compile the whole WRF-NMM with the just installed MPICH2-ifort compiler environment. This here might help you if you fail: http://software.intel.com/en-us/article ... compilers/

6. Once you succeeded in doing that (congrats by the way! Good job until now) , you will have to modify a bit the WRF-EMS environment:
a) Backup your WRF-EMS environment!!
b) Copy all the *TBL files from the WRF v3.3 run directory to the data/tables/wrf/, directory.
c) Modify namelist.nmm: Add

Code: Select all

euler_adv=.false.
to the section &DYNAMICS
d) Copy the WRF executables from your WRFv3.3 main/ directory to the bin/ WRF-EMS directory. The files are called wrf.exe (copy to wrfm_nmm.exe) and real_nmm.exe (copy to real_nmm.exe).
e) go to ./strc/ems_run/run_main.pm and go to line 2800.
Add

Code: Select all

    $EMSrun{WRFNL}{DYNAMICS}{euler_adv}[0] = '.false.';  
in the section called
"The DYNAMICS namelist section", in the "else" statement.

7. Launch your simulation. If you reach the point that everything works well - you're the hero of the month!

8. Should you have been unable to launch the simulation as there are errors, please contact me by PM or post a message here.

Cheers & have fun!

Julien




I forgot to mention that: Thanks to the WRF-FORUM & to Francesco. He was the one giving me the idea to modify my WRF-NMM-EMS environment! Link: http://forum.wrfforum.com/viewtopic.php ... 6&start=10

meteoadriatic
Posts: 1512
Joined: Wed Aug 19, 2009 10:05 am

Re: How to get rid of gridpoint storms (WRF-NMM)

Post by meteoadriatic » Sat Nov 19, 2011 2:15 pm

Great!

Hi, I tried and managed to compile nmm core but I have problems with MPI.

I compiled hdf5, netcdf and NMM using icc/ifort and suceeded to build wrf.exe and real_nmm.exe but something probably went wrong during building. I probably set wrong environment variables or didn't compiled properly for MPICH2...

I have clearly problems with MPI...

here are my logs for real_nmm log (it fails; If I use original real_nmm.exe then it suceed but wrfm_nmm.exe fails with similar MPI related errors)...

Code: Select all

Fatal error in MPI_Init: Other MPI error, error stack:
MPIR_Init_thread(392)..............: 
MPID_Init(139).....................: channel initialization failed
MPIDI_CH3_Init(38).................: 
MPID_nem_init(234).................: 
MPID_nem_tcp_init(108).............: 
MPID_nem_tcp_get_business_card(346): 
MPID_nem_tcp_init(305).............: gethostbyname failed, coldplug (errno 3)
forrtl: error (69): process interrupted (SIGINT)
Image              PC                Routine            Line        Source             
real_nmm.exe       00000000014277DA  Unknown               Unknown  Unknown
real_nmm.exe       0000000001426D6D  Unknown               Unknown  Unknown
real_nmm.exe       0000000001434AEC  Unknown               Unknown  Unknown
real_nmm.exe       000000000142A626  Unknown               Unknown  Unknown
real_nmm.exe       0000000001434640  Unknown               Unknown  Unknown
real_nmm.exe       000000000142BD95  Unknown               Unknown  Unknown
real_nmm.exe       000000000142A626  Unknown               Unknown  Unknown
real_nmm.exe       000000000142BF2C  Unknown               Unknown  Unknown
real_nmm.exe       0000000001421E4E  Unknown               Unknown  Unknown
real_nmm.exe       0000000001421EAC  Unknown               Unknown  Unknown
real_nmm.exe       0000000001401BA9  Unknown               Unknown  Unknown
real_nmm.exe       0000000001401853  Unknown       forrtl: error (69): process interrupted (SIGINT)
Image              PC                Routine            Line        Source             
real_nmm.exe       00000000014287F1  Unknown               Unknown  Unknown
real_nmm.exe       0000000001429C56  Unknown               Unknown  Unknown
real_nmm.exe       0000000001434C09  Unknown               Unknown  Unknown
real_nmm.exe       000000000142A626  Unknown               Unknown  Unknown
real_nmm.exe       0000000001434640  Unknown               Unknown  Unknown
real_nmm.exe       000000000142BD95  Unknown               Unknown  Unknown
real_nmm.exe       000000000142A626  Unknown               Unknown  Unknown
real_nmm.exe       000000000142BF2C  Unknown               Unknown  Unknown
real_nmm.exe       0000000001421E4E  Unknown               Unknown  Unknown
real_nmm.exe       0000000001421F44  Unknown               Unknown  Unknown
real_nmm.exe       0000000001401BA9  Unknown               Unknown  Unknown
real_nmm.exe       0000000001401853  Unknown       forrtl: error (69): process interrupted (SIGINT)
Image              PC                Routine            Line        Source             
real_nmm.exe       00000000013FD987  Unknown               Unknown  Unknown
real_nmm.exe       00000000012AEC04  Unknown               Unknown  Unknown
real_nmm.exe       00000000012AE0AA  Unknown               Unknown  Unknown
real_nmm.exe       00000000012BF075  Unknown               Unknown  Unknown
real_nmm.exe       000000000124B50E  Unknown               Unknown  Unknown
real_nmm.exe       00000000012245D5  Unknown               Unknown  Unknown
real_nmm.exe       00000000012830F3  Unknown               Unknown  Unknown
real_nmm.exe       000000000122E13D  Unknown               Unknown  Unknown
real_nmm.exe       0000000000604D2E  Unknown               Unknown  Unknown
real_nmm.exe       00000000008EE8CA  Unknown               Unknown  Unknown
real_nmm.exe       00000000004003FB  Unknown               Unknown  Unknown
real_nmm.exe       00000000004002EC  Unknown               Unknown  Unknown
real_nmm.exe       00000000012B4555  Unknown               Unknown  Unknown
real_nmm.exe       00000000012AE352  Unknown               Unknown  Unknown
real_nmm.exe       00000000012BF075  Unknown               Unknown  Unknown
real_nmm.exe       000000000124B50E  Unknown               Unknown  Unknown
real_nmm.exe       00000000012245D5  Unknown               Unknown  Unknown
real_nmm.exe       00000000012830F3  Unknown               Unknown  Unknown
real_nmm.exe       000000000122E13D  Unknown               Unknown  Unknown
real_nmm.exe       0000000000604D2E  Unknown               Unknown  Unknown
real_nmm.exe       00000000008EE8CA  Unknown               Unknown  Unknown
real_nmm.exe       00000000004003FB  Unknown               Unknown  Unknown
real_nmm.exe       00000000004002EC  Unknown               Unknown  Unknown
real_nmm.exe       00000000013CCB90  Unknown               Unknown  Unknown
real_nmm.exe       00000000004001B9  Unknown               Unknown  Unknown
        Unknown  Unknown
real_nmm.exe       00000000012B4555  Unknown               Unknown  Unknown
real_nmm.exe       00000000012AE352  Unknown               Unknown  Unknown
real_nmm.exe       00000000012BF075  Unknown               Unknown  Unknown
real_nmm.exe       000000000124B50E  Unknown               Unknown  Unknown
real_nmm.exe       00000000012245D5  Unknown               Unknown  Unknown
real_nmm.exe       00000000012830F3  Unknown               Unknown  Unknown
real_nmm.exe       000000000122E13D  Unknown               Unknown  Unknown
real_nmm.exe       0000000000604D2E  Unknown               Unknown  Unknown
real_nmm.exe       00000000008EE8CA  Unknown               Unknown  Unknown
real_nmm.exe       00000000004003FB  Unknown               Unknown  Unknown
real_nmm.exe       00000000004002EC  Unknown               Unknown  Unknown
real_nmm.exe       00000000013CCB90  Unknown               Unknown  Unknown
real_nmm.exe       00000000004001B9  Unknown               Unknown  Unknown
        Unknown  Unknown
real_nmm.exe       00000000013CCB90  Unknown               Unknown  Unknown
real_nmm.exe       00000000004001B9  Unknown               Unknown  Unknown
However, I tried this on v3.2 ems beta installation. You probably used v3.1 as I can guess from your post where you mentioned run_main.pm file and it's contents (on 3.2 beta it is a bit different).

Any suggestion about these errors? Is there anything special I need to do for compiling NMM against MPICH2, except setting correct path?

Can you post your configure.wrf file please?

Thanks!

beowrf
Posts: 39
Joined: Sat Jul 23, 2011 3:07 pm

Re: How to get rid of gridpoint storms (WRF-NMM)

Post by beowrf » Sun Nov 20, 2011 7:39 pm

Good evening,

Hm. It would help if you could compile with -g -O0 -traceback ... ;-)
I guess you have an uncomplete / corrupted namelist. Or you did not configure well your WRF makefiles (./configure) before compiling. Did you take smpar, dmpar or both? Then, another caveat is a mix of mpich2 versions.
If you compiled with 1.4.3 and your mpiexec used is the one coming with WRF-EMS (utils/mpichs/bin) then you should the mpiexec you installed yourself.

Here's my namelist:

Code: Select all

&time_control                                                                                                                                                                                
 start_year                 = 2011, 2011, 2011
 start_month                = 11, 11, 11
 start_day                  = 20, 20, 20
 start_hour                 = 12, 12, 12
 start_minute               = 00, 00, 00
 start_second               = 00, 00, 00
 tstart                     = 00, 00, 00
 end_year                   = 2011, 2011, 2011
 end_month                  = 11, 11, 11
 end_day                    = 23, 23, 22
 end_hour                   = 12, 12, 12
 end_minute                 = 00, 00, 00
 end_second                 = 00, 00, 00
 interval_seconds           = 10800
 history_interval           = 60, 30, 30
 history_outname            = "wrfout_d<domain>_<date>"
 frames_per_outfile         = 1, 1, 1
 restart                    = F
 restart_interval           = 360
 reset_simulation_start     = F
 io_form_input              = 2
 io_form_history            = 2
 io_form_restart            = 2
 io_form_boundary           = 2
 auxhist1_outname           = "sfcout_d<domain>_<date>"
 auxhist1_interval          = 0, 0, 0
 frames_per_auxhist1        = 1, 1, 1
 io_form_auxhist1           = 2
 auxinput1_inname           = "met_nmm.d<domain>.<date>"
 auxhist2_outname           = "auxhist2_d<domain>_<date>"
 auxhist2_interval          = 0, 0, 0
 frames_per_auxhist2        = 1, 1, 1
 io_form_auxhist2           = 5
 write_input                = F
 adjust_output_times        = T
 debug_level                = 0
/

&domains
 time_step                  = 28
time_step_fract_num        = 8
 time_step_fract_den        = 100
 max_dom                    = 3
 grid_allowed               = T, T, T
 s_we                       = 1, 1, 1
 e_we                       = 36, 64, 55
 s_sn                       = 1, 1, 1
 e_sn                       = 62, 82, 94
 s_vert                     = 1, 1, 1
 e_vert                     = 45, 45, 45
 dx                         = 0.0897880, 0.0299293, 0.0099764
 dy                         = 0.0896900, 0.0298967, 0.0099656
 grid_id                    = 1, 2, 3
 parent_id                  = 0, 1, 2
 i_parent_start             = 1, 7, 15
 j_parent_start             = 1, 16, 28
 parent_grid_ratio          = 1, 3, 3
 parent_time_step_ratio     = 1, 3, 3
 num_metgrid_levels         = 27
 num_metgrid_soil_levels    = 4
 p_top_requested            = 5000
 ptsgm                      = 28000.
 feedback                   = 0
 smooth_option              = 0
 numtiles                   = 1
 nproc_x                    = -1
 nproc_y                    = -1
/

&physics
 mp_physics                 = 8, 8, 8
 ra_lw_physics              = 1, 1, 1
 ra_sw_physics              = 99, 99, 99
 nrads                      = 64, 192, 576
 nradl                      = 64, 192, 576
 co2tf                      = 1
 sf_sfclay_physics          = 2, 2, 2
 sf_surface_physics         = 2, 2, 2
 sf_urban_physics           = 0, 0, 0
num_land_cat               = 24
 num_soil_cat               = 16
 bl_pbl_physics             = 2, 2, 2
 nphs                       = 4, 12, 36
 cu_physics                 = 2, 0, 0
 ncnvc                      = 4, 12, 36
 tprec                      = 73, 73, 73
 theat                      = 73, 73, 73
 tclod                      = 73, 73, 73
 trdsw                      = 73, 73, 73
 trdlw                      = 73, 73, 73
 tsrfc                      = 73, 73, 73
 pcpflg                     = F, F, F
 icloud                     = 1
 num_soil_layers            = 4
 usemonalb                  = T
 rdmaxalb                   = T
 rdlai2d                    = F
 seaice_threshold           = 271
 fractional_seaice          = 0
 gwd_opt                    = 0, 0, 0
/

&dynamics
 non_hydrostatic            = T
 diff_opt                   = 1
 euler_adv                  = .false.
/

&bdy_control
 spec_bdy_width             = 1
 specified                  = T, F, F
 nested                     = F, T, T
/

&grib2
/

&namelist_quilt
 nio_tasks_per_group        = 0
 nio_groups                 = 1
/

meteoadriatic
Posts: 1512
Joined: Wed Aug 19, 2009 10:05 am

Re: How to get rid of gridpoint storms (WRF-NMM)

Post by meteoadriatic » Wed Nov 23, 2011 9:25 pm

Hello, thanks!

Well...

yes, I recompiled mpich without optimization flags and then recompiled wrf (with old configure.wrf) and now it started to work (at least). OK, I'll post news when I test this thoroughly!

But, can you explain little bit what euler_adv = .false. does in comparision to T ?
beowrf wrote: Here's my namelist:

Code: Select all

 mp_physics                 = 8, 8, 8
That is probably the reason why you use euler_adv=.false. ?
Did you found this scheme better than default one (Ferrier)?

I ask, because I googled that and found that True becomes default from NMM v3.3, so it might be better to leave it on True if I use Ferrier?

beowrf
Posts: 39
Joined: Sat Jul 23, 2011 3:07 pm

Re: How to get rid of gridpoint storms (WRF-NMM)

Post by beowrf » Sun Nov 27, 2011 12:07 pm

Hello!

Great you secceeded, congrats!
euler_adv uses another avection scheme. I think that this is kind of a ferrier-replacement, but I didn't really had a look at the code.
Fact is that Euler_ADV is a new tool in the latest WRF version, which was never used before.

It would be a good thing to try to move to Euler. At least maybe for performance reasons, e.g.

Did you already try it?

Cheers,

J

meteoadriatic
Posts: 1512
Joined: Wed Aug 19, 2009 10:05 am

Re: How to get rid of gridpoint storms (WRF-NMM)

Post by meteoadriatic » Sun Nov 27, 2011 6:29 pm

Well, I don't know! If it was turned on in original wrfems 3.1 then yes, if not then I didn't tried it :mrgreen: Really this is the first time I got in touch with this parameter. But what I could find about it is this:
http://www.dtcenter.org/wrf-nmm/users/o ... s_v3.3.php

So it is default from WRF NMM v3.3, but only works with Ferrier. So it looks it can't work with your Microphysics scheme (8). However I found Ferrier much faster than number 8 and it is default one so I stick with it all the time. I use it even with ARW here http://maps.meteoadriatic.net/ because it speeds up model execution a lot.

Here http://www.ezadar.hr/prognoza/meteo-karte/hrvatska I use default MP scheme with ARW (2). Never used New Thompson (8) yet. What do you think about forecast reliability of this (8) scheme?

About euler_adv, I will test it if it will work with my models. Is it only for NMM? I'll report my findings.

meteoadriatic
Posts: 1512
Joined: Wed Aug 19, 2009 10:05 am

Re: How to get rid of gridpoint storms (WRF-NMM)

Post by meteoadriatic » Tue Nov 29, 2011 6:10 am

meteoadriatic wrote:About euler_adv, I will test it if it will work with my models. Is it only for NMM? I'll report my findings.
I turned it on by changing those in NMM namelist from .false. to T. There are differences, and significant. The most important one is that model runs much slower when euler_adv is used. About 25% slower I would say. However, there is one parameter that can control this little bit. It is idtadt and it say how often to call this advection in respect to model time steps. So default is 1, but I set it to 2 and model is now faster but still around 15% slower than without euler_adv.

Forecast impact? Well it seems to me, there is some difference. Precipitation areas are little bit more "soft" and "natural", in fact, it maybe even helps to control grid point storms bit more, but I'm not sure. I did just 2 runs with it now. Maybe I will make some test with parallel runs with/without to see if I'm right.

However:
NMM 3.3.1 with few changes as you and Francesco proposed (didn't applied them all yet), with euler_adv and BMJ Cu scheme, 0.77deg (12km), accumulated precipitation in 6 hours:
http://maps.meteoadriatic.net/wrf-nmm-3 ... orine.html

And 4km nest from it, CU off:
http://maps.meteoadriatic.net/wrf-nmm-3 ... orine.html

newman66
Posts: 1
Joined: Sun Aug 12, 2012 4:41 am

Re: How to get rid of gridpoint storms (WRF-NMM)

Post by newman66 » Sun Aug 12, 2012 4:49 am

[color=#333333]Los Angeles lawyer[/color] wrote: Hello everyone. Here is what you need to do:

2. Go to the MPICH2 homepage (here http://www.mcs.anl.gov/research/project ... =downloads) and download + install the MPICH2 library & binaries. It is best practice to configure with ./configure --prefix=/opt/mpich2 so that you have your mpich2 libraries in one known folder. MPICH2 has been successfully installed if in /opt/mpich2/bin there are binaries like mpif90, mpicc or mpiexec. Add this folder to your $PATH.
Hello Julien,

I followed your directions here. Step 1 went ok. step 2, I don't know why, but when I go to that link, it does not work. It says something about bad port or something. Does it work on your end?
Last edited by newman66 on Sun Aug 26, 2012 2:23 am, edited 2 times in total.
Thanks

beowrf
Posts: 39
Joined: Sat Jul 23, 2011 3:07 pm

Re: How to get rid of gridpoint storms (WRF-NMM)

Post by beowrf » Sun Aug 12, 2012 1:48 pm

Hi newman66,

Hm - that's strange. I just accessed the webpage of MPICH2 now and it works without problems. Maybe the server was down?

Best regards,

Julien

meteoadriatic
Posts: 1512
Joined: Wed Aug 19, 2009 10:05 am

Re: How to get rid of gridpoint storms (WRF-NMM)

Post by meteoadriatic » Sun Aug 12, 2012 1:51 pm

Hi,

also, you can try direct link if you still have problems:
http://www.mcs.anl.gov/research/project ... 1p1.tar.gz

Post Reply

Who is online

Users browsing this forum: No registered users and 0 guests