Automatisation related to the overlap file's creation
To run the STRATFINDER
algorithm we need overlap files which are in the netcdf files, that you can find here the nomination of these files are these followings Hervo_overlapmodel_${city}_${inst}_${inst-id}_${optical_id}_{$res_s}.nc
where
-
city
: Name of the city -
inst
: Name of the instrument (chm15k for now ) -
inst-id
: ID of the instrument -
optical-id
: ID related to the optical module -
res_s
: spatial resolution written in this form ??m
Here is an exemple of this file
Hervo_overlapmodel_cabauw_chm15k_A_TUB140015_10m.nc
Currently this process has been done manually which will not be possible to do in the near future if we handle all the chm15k
instrument in the eprofile network. The criteria used to know if the overlap file is correct or wrong is used by watching some quicklooks. However @mvanhove has created some method to automatise this process which leads of two possible outcomes
- Creation of the overlap files
- There is not enough days that fullfill the condition to create this overlap file so it creates an error
I will add in the STRATFINDER
worfkflow one task that will verify the existence of the overlap files for each instruments, if it does not exist it means that the optical module has been changed which means that the optical-id also change. So there will be three cases that we can enconter.
- The overlap file already exist so the script will pass to the next step i.e the creation of the L2A script
- The overlap file does not exist and there is not enough data of good quality a script will be called which will parse the log in the .err and .out and will send an email containing these log files
- The overlap file does not exist and there is enough data the overlap file is created and a branch called
add_${optical-id}_${city}
on the STRAFINDER gitlab. The merging of this branch has to be done manually through a merge request, this method avoids some commit conflict if we only push those files on themaster
branch.
What do you think about it @simone.kotthaus and @mvanhove ?
/cc @marc-antoine.drouin