Commit c8faf15e authored by Patryk Kiepas's avatar Patryk Kiepas 🐦
Browse files

Formatting Valentin's README as Markdown

parent 9696b7d8
# HOW TO RUN ZOOMED-NUDGED LMDZ SIMULATIONS ON JEAN-ZAY WITH TUTORIAL_PROD
> N.B. : the following instructions are specific for running simulations on the Jean-Zay server.
LMDZ in the Jean-Zay server is divided between 3 main folders:
1. `$STORE`: actually your working directory, where your scripts are stored
2. `$WORK`: the LMDZ source code (Fortran files) and such are stored there
3. `$SCRATCH`: the simulation outputs will be stored there.
[[_TOC_]]
## I. INSTALLATION AND COMPILATION
* Go to your `$STORE` folder: `cd $STORE`
* Copy a `TUTORIAL_PROD` archive: `cp -p /gpfsstore/rech/gzi/rdzt896/test_prod_v20211006.tar`.
This one is probably outdated, but as we'll update the model later it's not a problem (the latest [archive](https://www.lmd.jussieu.fr/~lmdz/pub/Training/tutorial_prod.tar)).
* Extract the archive: `tar -xvf test_prod_v20211006.tar`
* cd `TEST_PROD_v20211006/`
In the `lmdz_env.sh` file, check that your active Jean-Zay computation project is the one you want to use.
To know your default group type `idrproj`.
You can change `groupe=$groupeLog` by `groupe=your_project`.
* Choose your grid resolution in `main.sh` (example: `resol=64x64x95`)
Horizontal resolutions should be multiples of `2^ngroups` (usually `ngroups=3` or `4`) and vertical resolutions should correspond to a `L*.def` file (37, 47, 59, 79, 95).
* Choose your start and end dates (you should begin with a 1-month simulation for a test)
* Still in `main.sh`, check that `install="-install"`, `xios="-xios"`, `init=1` and `nudging="-nudging"`
* Run `./main.sh`. That simple user-friendly script launches in fact the more complex `setup.sh` script.
It may take some tens of minutes.
> Note : Each time you'll want to change the grid resolution in the future, you'll have to recompile the model by editing and running the `compile.sh` script in `$WORK/LMDZ20210927.trunk/modipsl/modeles/LMDZ/`.
Once finished :
* Go to the LMDZ source codes folder in your `$WORK`:
`cd $WORK/LMDZ20210927.trunk/modipsl/modeles/LMDZ/libf`. Then type:
```sh
module load svn
svn info # (describe actual version)
svn update
svn info # (updated version)
```
LMDZ is now up to date.
## II. INITIALISATION
### II.a : Configuring the grid
* Set `install=""` in `main.sh`
* Check that `init=1` in `main.sh`
* Go into the DEF folder. Open `gcm.def`.
* `day_step`: number of dynamical steps in a day.
A finer grid needs more timesteps. If you've got N cells along the longitudes, and a coarser reference simulation had M:
`day_step(N) = day_step(M) * N/M`.
Moreover, a zoomed simulation also refines the grid and requires more timesteps, using the relation:
`day_step(zoomed) = day_step(regular) * max(grossismx, grossismy)`.
Mine has 14400 timesteps for 64 longitude cells.
Dynamical timestep: `dt_dyn = 86400 / day_step`. In my case, `dt_dyn = 6s`.
* `iperiod`: I let it at 5.
* `idissip`: I let it at 5.
* `iphysiq`: I wanted a physical timestep of 10 minutes = 600s.
As: `dt_phys = iphysiq * dt_dyn` then `iphysiq = dt_phys / dt_dyn = 600 / 6 = 100`.
So the model calls the physics every 100 dynamical timesteps.
The physical timestep is also, I believe, the highest possible output frequency.
* Dissipation parameters:
By default, `tetagdiv = 600.`, `tetagrot = 1200.` and `tetatemp = 1200.`.
But if you refine your grid, you'll have to strengthen the dissipation = reducing dissipation time scales.
You can check my empirical values in the `gcm.def` to have an idea.
* Zoom parameters:
Set the center of your zoom at `{clon, clat}`.
Then choose magnification constants (`grossismx` and `grossismy`); and its extension as a fraction of the total zone (`dzoomx`, `dzoomy`).
> Note: if your zone of interest is near the poles, you don't need to zoom as much in `x` as in `y` because longitudes narrow in a lon/lat grid.
Then, choose the zoom "stiffness". I let it at 3. It's advised to not go beyond.
The zoom parameters are constrained: you cannot have a strong magnification and a strong extension at the same time.
If you do, you'll encounter the following error message : _"Attention ! La valeur beta calculee dans la routine fyhyp est mauvaise. [...]"_
* Check that `nudging=-nudging` in `main.sh`.
* If there is already a `INIT/` folder, you can remove it (as it contains files for a regular grid): `rm -rf INIT/`
* Then launch `./main.sh`.
You should have created the `INIT/` folder containing starting files `start.nc` and `startphy.nc`, and the grid file `grilles_gcm.nc`;
and the `LIMIT/` folder containing the limit file `limit.nc` (it will force sst and sea ice).
If it didn't work, check your `$SCRATCH/TEST_PROD_v20211013/INIT` folder (`InitOutput` may be useful).
> Note: If your simulation starts after 2018, you'll have to use ERA5 sea ice and SST files instead of amipbc files. To do that, fetch those files here: `/gpfswork/rech/gzi/rgzi019/SSTERA5` and run the script `create_amip.sh`. You'll need an older amipbc file (available here: https://www.lmd.jussieu.fr/~lmdz/pub/3DInputData/Limit/) for doing an interpolation with Ferret.
* Visualize your grid by modifying the script `verif_grid.jnl` (Ferret script).
If you set `nudging=-nudging`, `main.sh` stops after the initialization, and you'll have to launch it again with `init=0` after having created the nudging files.
### II.b : NUDGING CONFIGURATION
In the script `era2gcm.sh`, you can choose:
* the reanalysis (ERA Interim, ERA5 or OPERA)
* the dates (unlike `main.sh`, the script stops at the end of the final month, not at the beginning)
* the nudging variables. If you've got a strong zoom and a coarse resolution, you should nudge in winds, temperature and humidity.
Run the script with `./era2gcm.sh`. It will fetch reanalysis files stored somewhere in Jean-Zay,
and interpolate it on your custom grid defined in `INIT/grilles_gcm.nc`.
If you want to use ERAI after 2014, you'll have to find the files somewhere else.
It may take some time.
Then, go edit the `DEF/guide.def` file and set the nudging constants.
`tau_min_u=0.125` for instance will nudge the zonal wind outside the zoom area with a 3 hours timescale (3h/24h = 0.125).
`tau_max_T=0.5` will nudge inside the zoomed area with a 12 hours timescale (12h/24h = 0.5).
But usually, the model is only nudged outside the zoom, and the nudging constants are set to very high values like `1e6` (`tau=infinity` --> not nudged).
> Note : with `tau_min_u=0.125`, the nudging files aren't called each 3 hours : they are called at each dynamical step, and correct the model with a strength proportional to `1/tau` : `dt(u_model) += (u_model - u_rea)/tau_min_u`.
### II.c : OUTPUT FILES CONFIGURATION
Once you've created the nudging files you have to configure the outputs.
If you're using XIOS, check the line `xios=-xios` in `main.sh`.
Then refer to [this document](https://docs.google.com/document/d/1W1YI7NinzgBTUOUa2orjN40j3Aeg1BkPXCuOAEWUuxc/edit?usp=sharing) to create your custom output files.
## III. EXECUTION
- Set `init=0` in `main.sh`.
Parallelism :
Several small regions communicating together work faster than a big region.
In the file setup.sh :
- the number of mpi processes will separate the grid into rows of longitude (nrows = ny / mpi, should be an integer).
For me, ny=64 and mpi=32 : the model separates the planet into 32 slices of 2 latitudes.
- the number of open_mp processes will split the vertical axis in several layers.
For me, open_mp=8 and nz=95 (not an integer...).
You should set in the header of script_SIMU :
#SBATCH --ntasks-per-node=2 instead of 5, if that's not already the case.
Then launch `./main.sh` again. The simulation should start, and you can go to the `$SCRATCH` folder to follow its advancement.
In your simulation folder in the `$SCRATCH`, you can try:
```sh
grep "[year] /" listing | tail -n 20 # (replace year with 2018 for instance)
squeue -u $USER
```
Once finished, the listing file and the restart files are transferred to the `$STORE`, and you can start again where you left.
The output files stay in the `$SCRATCH` ; you should copy them somewhere, because the `$SCRATCH` may be cleaned regularly.
######
HOW TO RUN ZOOMED-NUDGED LMDZ SIMULATIONS ON JEAN-ZAY WITH TUTORIAL_PROD
######
###
N.B. : the following instructions are specific for running simulations on the Jean-Zay server.
###
LMDZ in the Jean-Zay server is divided between 3 main folders :
$STORE : actually your working directory, where your scripts are stored
$WORK : the LMDZ source code (Fortran files) and such are stored there
$SCRATCH : the simulation outputs will be stored there.
I. INSTALLATION AND COMPILATION
- Go to your $STORE folder : "cd $STORE"
- Copy a TUTORIAL_PROD archive : cp -p /gpfsstore/rech/gzi/rdzt896/test_prod_v20211006.tar
This one is probably outdated, but as we'll update the model later it's not a problem.
- Extract the archive : tar -xvf test_prod_v20211006.tar
- cd TEST_PROD_v20211006/
In the lmdz_env.sh file, check that your active Jean-Zay computation project is the one you want to use.
To know your default group type "idrproj".
You can change "groupe=$groupeLog" by "groupe=your_project".
- Choose your grid resolution in "main.sh" (example : resol=64x64x95)
Horizontal resolutions should be multiples of 2^ngroups (usually ngroups = 3 or 4)
and vertical resolutions should correspond to a L*.def file (37, 47, 59, 79, 95)
- Choose your start and end dates (you should begin with a 1-month simulation for a test)
- Still in "main.sh", check that install="-install", xios="-xios", init=1 and nudging="-nudging"
- Run "./main.sh". That simple user-friendly script launches in fact the more complex "setup.sh" script.
It may take some tens of minutes.
(Note : Each time you'll want to change the grid resolution in the future, you'll have to recompile the model
by editing and running the compile.sh script in $WORK/LMDZ20210927.trunk/modipsl/modeles/LMDZ/).
Once finished :
- Go to the LMDZ source codes folder in your $WORK :
cd $WORK/LMDZ20210927.trunk/modipsl/modeles/LMDZ/libf
Then :
- "module load svn"
- "svn info" (describe actual version)
- "svn update"
- "svn info" (updated version)
LMDZ is now up to date.
II. INITIALISATION
II.a : Configuring the grid
- Set install="" in "main.sh"
- Check that "init=1" in "main.sh"
- Go into the DEF folder. Open "gcm.def".
- "day_step" : number of dynamical steps in a day.
A finer grid needs more timesteps. If you've got N cells along the longitudes, and a coarser reference simulation had M :
day_step(N) = day_step(M) * N/M
Moreover, a zoomed simulation also refines the grid and requires more timesteps, using the relation :
day_step(zoomed) = day_step(regular) * max(grossismx, grossismy)
Mine has 14400 timesteps for 64 longitude cells.
Dynamical timestep : dt_dyn = 86400 / day_step. In my case, dt_dyn = 6s.
- iperiod : I let it at 5.
- idissip : I let it at 5.
- iphysiq : I wanted a physical timestep of 10 minutes = 600s.
As : dt_phys = iphysiq * dt_dyn then iphysiq = dt_phys / dt_dyn = 600 / 6 = 100.
So the model calls the physics every 100 dynamical timesteps.
The physical timestep is also, I believe, the highest possible output frequency.
- Dissipation parameters :
By default, tetagdiv = 600. ; tetagrot = 1200. and tetatemp = 1200.
But if you refine your grid, you'll have to strengthen the dissipation = reducing dissipation time scales.
You can check my empirical values in the "gcm.def" to have an idea.
- Zoom parameters :
Set the center of your zoom at {clon, clat}.
Then choose magnification constants (grossismx and grossismy) ; and its extension as a fraction of the total zone (dzoomx, dzoomy).
Note : if your zone of interest is near the poles, you don't need to zoom as much in x as in y because longitudes narrow in a lon/lat grid.
Then, choose the zoom "stiffness". I let it at 3. It's advised to not go beyond.
The zoom parameters are constrained : you cannot have a strong magnification and a strong extension at the same time.
If you do, you'll encounter the following error message : "Attention ! La valeur beta calculee dans la routine fyhyp est mauvaise. [...]"
- Check that "nudging=-nudging" in "main.sh".
- If there is already a INIT/ folder, you can remove it (as it contains files for a regular grid) : "rm -rf INIT/"
- Then launch "./main.sh".
You should have created the INIT/ folder containing starting files start.nc and startphy.nc, and the grid file grilles_gcm.nc ;
and the LIMIT/ folder containing the limit file limit.nc (it will force sst and sea ice).
If it didn't work, check your $SCRATCH/TEST_PROD_v20211013/INIT folder (InitOutput may be useful).
Note : If your simulation starts after 2018, you'll have to use ERA5 sea ice and SST files instead of amipbc files. To do that, fetch those files here : /gpfswork/rech/gzi/rgzi019/SSTERA5 and run the script create_amip.sh. You'll need an older amipbc file (available here : https://www.lmd.jussieu.fr/~lmdz/pub/3DInputData/Limit/) for doing an interpolation with Ferret.
- Visualize your grid by modifying the script verif_grid.jnl (Ferret script).
If you set "nudging=-nudging, "main.sh" stops after the initialization, and you'll have to launch it again with "init=0"
after having created the nudging files.
II.b : NUDGING CONFIGURATION
In the script "era2gcm.sh", you can choose :
- the reanalysis (ERA Interim, ERA5 or OPERA)
- the dates (unlike "main.sh", the script stops at the end of the final month, not at the beginning)
- the nudging variables. If you've got a strong zoom and a coarse resolution, you should nudge in winds, temperature and humidity.
Run the script with "./era2gcm.sh". It will fetch reanalysis files stored somewhere in Jean-Zay,
and interpolate it on your custom grid defined in INIT/grilles_gcm.nc.
If you want to use ERAI after 2014, you'll have to find the files somewhere else.
It may take some time.
Then, go edit the DEF/guide.def file and set the nudging constants.
tau_min_u=0.125 for instance will nudge the zonal wind outside the zoom area with a 3 hours timescale (3h/24h = 0.125).
tau_max_T=0.5 will nudge inside the zoomed area with a 12 hours timescale (12h/24h = 0.5).
But usually, the model is only nudged outside the zoom, and the nudging constants are set to very high values like 1e6 (tau = infinity --> not nudged).
Note : with tau_min_u=0.125, the nudging files aren't called each 3 hours : they are called at each dynamical step, and correct the model
with a strength proportional to 1/tau : dt(u_model) += (u_model - u_rea)/tau_min_u.
II.c : OUTPUT FILES CONFIGURATION
Once you've created the nudging files you have to configure the outputs.
If you're using XIOS, check the line "xios=-xios" in "main.sh". Then refer to this document to create your custom output files : https://docs.google.com/document/d/1W1YI7NinzgBTUOUa2orjN40j3Aeg1BkPXCuOAEWUuxc/edit?usp=sharing
III. EXECUTION
- Set "init=0" in "main.sh".
Parallelism :
Several small regions communicating together work faster than a big region.
In the file setup.sh :
- the number of mpi processes will separate the grid into rows of longitude (nrows = ny / mpi, should be an integer).
For me, ny=64 and mpi=32 : the model separates the planet into 32 slices of 2 latitudes.
- the number of open_mp processes will split the vertical axis in several layers.
For me, open_mp=8 and nz=95 (not an integer...).
You should set in the header of script_SIMU :
#SBATCH --ntasks-per-node=2 instead of 5, if that's not already the case.
Then launch "./main.sh" again. The simulation should start, and you can go to the $SCRATCH folder to follow its advancement.
In your simulation folder in the $SCRATCH, you can try :
grep "[year] /" listing | tail -n 20 # (replace year with 2018 for instance)
squeue -u $USER
Once finished, the listing file and the restart files are transferred to the $STORE, and you can start again where you left.
The output files stay in the $SCRATCH ; you should copy them somewhere, because the $SCRATCH may be cleaned regularly.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment