Most of the script are python. The packages must be defined, see <ahref="setup.html#org5298da3">setup.html#org5298da3</a>. I recommend an anaconda environment.
The appropriate environmental variables must be set, especially <code>$PYTHONPATH</code> must include the root directory of the project, see <ahref="setup.html#orgea78dbc">setup.html#orgea78dbc</a>.
Most of the script are python. The packages must be defined, see <ahref="setup.html#org5298da3">Setup::Requirements</a>. I recommend an anaconda environment.
The appropriate environmental variables must be set, especially <code>$PYTHONPATH</code> must include the root directory of the project, see <ahref="setup.html#orgea78dbc">Setup</a>.
</p>
<p>
...
...
@@ -246,7 +255,7 @@ On ciclad, this is all done by launching a script through:
</div>
<p>
On my personnal machine, I use <ahref="https://github.com/direnv/direnv">direnv</a> that automatically reads /.envrc and setup everything when entering the project directory in a terminal. Python script can just be run normally.
On my personnal machine, I use <ahref="https://github.com/direnv/direnv">direnv</a>, which automatically reads /.envrc and setup everything when entering the project directory in a terminal. Python scripts can just be run normally.
<ahref="https://resources.marine.copernicus.eu/product-detail/SST_GLO_SST_L4_REP_OBSERVATIONS_010_024/INFORMATION">ESA SST CCI and C3S reprocessed sea surface temperature analyses</a>
<h3id="orgb64e62e"><spanclass="section-number-3">2.2.</span> MODIS 1km data</h3>
<divclass="outline-text-3"id="text-2-2">
<p>
For 1km resolution data.
Use is limited for results, so this one is for the records. On top of that it is quite a pain in the butt to setup.
Because this dataset is <b>much</b> more heavy, scripts run monthly instead of yearly.
The first script download L2 swaths that intersect a defined region, from cmr.earthdata.nasa.gov. Once downloaded each swath is regrided on a 1km grid.
Finally all swaths are merged daily. It is necessary to check that the correct number of swaths are downloaded (see the corresponding script).
Because the recovered data is only from the Aqua satellite, the cloud coverage is quite high. To avoid problems that <b>could</b> arise because of that (and because it reduces the amount of data to process), we find windows with low cloud coverage in daily snapshot, and only work on those afterwards. Each window is called an ’image’. Scripts that work only on cloud-free images usually contain <code>_images</code>.
This saves for each daily snapshot a txt file containing all the images positions (in pixel and lat/lon). Images can then be recovered using the <ahref="../lib/data/images.py">../lib/data/images.py</a> dataset.
This script uses a fortran code, it has to be compiled before-hand (python will complain with a usefull warning if not). That could be rewritten in Xarray mind you.
We do a little processing before continuing. This script changes the units to celsius, rename the sst variable, removes the ice-covered pixels, drop unecessary variables, and regrid to EPSG32662 (see <ahref="grids.html">Grids</a>).
Processed file are <code>SST_processed_[date].nc</code>, original files are to be deleted manually (if needed).
@@ -7,15 +7,15 @@ This describes the important scripts, the order and the manner they should be ex
Most scripts work on a single year passed by the argument '-year' (see [[file:args.org][Args]]).
* Python scripts execution
Most of the script are python. The packages must be defined, see file:setup.org::Requirements. I recommend an anaconda environment.
The appropriate environmental variables must be set, especially ~$PYTHONPATH~ must include the root directory of the project, see file:setup.org::Setup.
Most of the script are python. The packages must be defined, see [[file:setup.org::Requirements][Setup::Requirements]]. I recommend an anaconda environment.
The appropriate environmental variables must be set, especially ~$PYTHONPATH~ must include the root directory of the project, see [[file:setup.org::Setup][Setup]].
On ciclad, this is all done by launching a script through:
On my personnal machine, I use [[https://github.com/direnv/direnv][direnv]] that automatically reads /.envrc and setup everything when entering the project directory in a terminal. Python script can just be run normally.
On my personnal machine, I use [[https://github.com/direnv/direnv][direnv]], which automatically reads /.envrc and setup everything when entering the project directory in a terminal. Python scripts can just be run normally.
* Downloading Data
~$SUBMESO_COLOR_CODE_DIR~ (or whatever folder name you use) is abbrievated 'DIR' hereafter.
...
...
@@ -24,16 +24,41 @@ Use file:../Download/download_cmems.sh.
See -h argument for help.
examples:
*** Download SST
[[https://resources.marine.copernicus.eu/product-detail/SST_GLO_SST_L4_REP_OBSERVATIONS_010_024/INFORMATION][ESA SST CCI and C3S reprocessed sea surface temperature analyses]]
Use is limited for results, so this one is for the records. On top of that it is quite a pain in the butt to setup.
Because this dataset is *much* more heavy, scripts run monthly instead of yearly.
The first script download L2 swaths that intersect a defined region, from cmr.earthdata.nasa.gov. Once downloaded each swath is regrided on a 1km grid.
Finally all swaths are merged daily. It is necessary to check that the correct number of swaths are downloaded (see the corresponding script).
Because the recovered data is only from the Aqua satellite, the cloud coverage is quite high. To avoid problems that *could* arise because of that (and because it reduces the amount of data to process), we find windows with low cloud coverage in daily snapshot, and only work on those afterwards. Each window is called an 'image'. Scripts that work only on cloud-free images usually contain ~_images~.
This saves for each daily snapshot a txt file containing all the images positions (in pixel and lat/lon). Images can then be recovered using the file:../lib/data/images.py dataset.
This script uses a fortran code, it has to be compiled before-hand (python will complain with a usefull warning if not). That could be rewritten in Xarray mind you.
* Pre-processing data
** OSTIA SST
We do a little processing before continuing. This script changes the units to celsius, rename the sst variable, removes the ice-covered pixels, drop unecessary variables, and regrid to EPSG32662 (see [[file:grids.org][Grids]]).
Processed file are ~SST_processed_[date].nc~, original files are to be deleted manually (if needed).
