Docker-in-Docker (DinD) capabilities of public runners deactivated. More info

Commit f8c477e9 authored by Clément Haëck's avatar Clément Haëck
Browse files

Publish in same directory as org

so that it does not break links to directories/scripts
parent 34cba252
......@@ -12,5 +12,5 @@ Collocate submesoscale fronts to phytoplankton levels using satellite imagery
## Doc
Some documentation is available in orgmode and html format. See ([./docs/index.html](Index)).
Some documentation is available in orgmode and html format. See ([./docs/index.org](Index)).
......@@ -3,7 +3,7 @@
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<!-- 2022-01-05 mer. 20:01 -->
<!-- 2022-01-06 jeu. 13:46 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Args</title>
......@@ -340,7 +340,7 @@ This system is a bit heavy but convenient, as the variable args can be passed to
</div>
<div id="postamble" class="status">
<p class="author">Author: Clément Haëck</p>
<p class="date">Created: 2022-01-05 mer. 20:01</p>
<p class="date">Created: 2022-01-06 jeu. 13:46</p>
</div>
</body>
</html>
......@@ -3,7 +3,7 @@
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<!-- 2022-01-05 mer. 20:01 -->
<!-- 2022-01-06 jeu. 13:46 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Boxes</title>
......@@ -210,7 +210,7 @@ With those, it is easy to define a rectangular area. It can be used to slice a d
</div>
<div id="postamble" class="status">
<p class="author">Author: Clément Haëck</p>
<p class="date">Created: 2022-01-05 mer. 20:01</p>
<p class="date">Created: 2022-01-06 jeu. 13:46</p>
</div>
</body>
</html>
......@@ -3,7 +3,7 @@
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<!-- 2022-01-05 mer. 20:04 -->
<!-- 2022-01-06 jeu. 13:46 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Datasets</title>
......@@ -322,7 +322,7 @@ We define some static &rsquo;zones&rsquo;, this dataset is quite special and hav
</div>
<div id="postamble" class="status">
<p class="author">Author: Clément Haëck</p>
<p class="date">Created: 2022-01-05 mer. 20:04</p>
<p class="date">Created: 2022-01-06 jeu. 13:46</p>
</div>
</body>
</html>
......@@ -3,7 +3,7 @@
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<!-- 2022-01-05 mer. 20:01 -->
<!-- 2022-01-06 jeu. 13:46 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Grids</title>
......@@ -250,7 +250,7 @@ This was choosen recently, so for a time I dealt with 2 grids. This is why most
</div>
<div id="postamble" class="status">
<p class="author">Author: Clément Haëck</p>
<p class="date">Created: 2022-01-05 mer. 20:01</p>
<p class="date">Created: 2022-01-06 jeu. 13:46</p>
</div>
</body>
</html>
......@@ -3,7 +3,7 @@
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<!-- 2022-01-05 mer. 20:01 -->
<!-- 2022-01-06 jeu. 13:46 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Hi</title>
......@@ -226,7 +226,7 @@ How it works (the script).
</div>
<div id="postamble" class="status">
<p class="author">Author: Clément Haëck</p>
<p class="date">Created: 2022-01-05 mer. 20:01</p>
<p class="date">Created: 2022-01-06 jeu. 13:46</p>
</div>
</body>
</html>
......@@ -3,7 +3,7 @@
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<!-- 2022-01-05 mer. 20:01 -->
<!-- 2022-01-06 jeu. 13:46 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Histograms</title>
......@@ -221,7 +221,7 @@ advantages / defaults of using xhistogram
</div>
<div id="postamble" class="status">
<p class="author">Author: Clément Haëck</p>
<p class="date">Created: 2022-01-05 mer. 20:01</p>
<p class="date">Created: 2022-01-06 jeu. 13:46</p>
</div>
</body>
</html>
......@@ -3,7 +3,7 @@
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<!-- 2022-01-05 mer. 20:01 -->
<!-- 2022-01-06 jeu. 13:46 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Index</title>
......@@ -248,7 +248,7 @@ It is far from covering everything but should put you on the right track.
</div>
<div id="postamble" class="status">
<p class="author">Author: Clément Haëck</p>
<p class="date">Created: 2022-01-05 mer. 20:01</p>
<p class="date">Created: 2022-01-06 jeu. 13:46</p>
</div>
</body>
</html>
......@@ -9,8 +9,7 @@
"submeso-color-docs"
:base-directory (file-name-directory (buffer-file-name))
:base-extension "org"
:publishing-directory (concat (file-name-directory (buffer-file-name))
"html")
:publishing-directory (file-name-directory (buffer-file-name))
:recursive t
:publishing-function #'org-html-publish-to-html
:headline-levels 4
......
......@@ -3,7 +3,7 @@
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<!-- 2022-01-05 mer. 20:01 -->
<!-- 2022-01-06 jeu. 13:46 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Setup</title>
......@@ -285,7 +285,7 @@ On ciclad, python script must be executed by using <a href="../launch_python.sh"
</div>
<div id="postamble" class="status">
<p class="author">Author: Clément Haëck</p>
<p class="date">Created: 2022-01-05 mer. 20:01</p>
<p class="date">Created: 2022-01-06 jeu. 13:46</p>
</div>
</body>
</html>
......@@ -3,7 +3,7 @@
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<!-- 2022-01-05 mer. 20:01 -->
<!-- 2022-01-06 jeu. 13:48 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Workflow</title>
......@@ -211,7 +211,16 @@
<li><a href="#org64c0bac">2.1.2. Download CHL</a></li>
</ul>
</li>
<li><a href="#org031cf67">2.2. MODIS data</a></li>
<li><a href="#orgb64e62e">2.2. MODIS 1km data</a>
<ul>
<li><a href="#org0b587d0">2.2.1. Finding images</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#org87a6804">3. Pre-processing data</a>
<ul>
<li><a href="#orgce23ed5">3.1. OSTIA SST</a></li>
</ul>
</li>
</ul>
......@@ -233,8 +242,8 @@ Most scripts work on a single year passed by the argument &rsquo;-year&rsquo; (s
<h2 id="org30869bd"><span class="section-number-2">1.</span> Python scripts execution</h2>
<div class="outline-text-2" id="text-1">
<p>
Most of the script are python. The packages must be defined, see <a href="setup.html#org5298da3">setup.html#org5298da3</a>. I recommend an anaconda environment.
The appropriate environmental variables must be set, especially <code>$PYTHONPATH</code> must include the root directory of the project, see <a href="setup.html#orgea78dbc">setup.html#orgea78dbc</a>.
Most of the script are python. The packages must be defined, see <a href="setup.html#org5298da3">Setup::Requirements</a>. I recommend an anaconda environment.
The appropriate environmental variables must be set, especially <code>$PYTHONPATH</code> must include the root directory of the project, see <a href="setup.html#orgea78dbc">Setup</a>.
</p>
<p>
......@@ -246,7 +255,7 @@ On ciclad, this is all done by launching a script through:
</div>
<p>
On my personnal machine, I use <a href="https://github.com/direnv/direnv">direnv</a> that automatically reads /.envrc and setup everything when entering the project directory in a terminal. Python script can just be run normally.
On my personnal machine, I use <a href="https://github.com/direnv/direnv">direnv</a>, which automatically reads /.envrc and setup everything when entering the project directory in a terminal. Python scripts can just be run normally.
</p>
</div>
</div>
......@@ -270,6 +279,13 @@ examples:
<div id="outline-container-orgbb9cdf2" class="outline-4">
<h4 id="orgbb9cdf2"><span class="section-number-4">2.1.1.</span> Download SST</h4>
<div class="outline-text-4" id="text-2-1-1">
<p>
<a href="https://resources.marine.copernicus.eu/product-detail/SST_GLO_SST_L4_REP_OBSERVATIONS_010_024/INFORMATION">ESA SST CCI and C3S reprocessed sea surface temperature analyses</a>
</p>
<p>
Dataset name: ostia | directory: OSTIA/1days
</p>
<div class="org-src-container">
<pre class="src src-bash"><span style="color: #6a1868;">year</span>=<span style="color: #da8548; font-weight: bold;">2007</span>; <span style="color: #986801;">echo</span> <span style="color: #50a14f;">"bash DIR/Download/download_cmems.sh -d sst_rep_obs_010_024 -p SST -r GS -s </span><span style="color: #b751b6;">$</span><span style="color: #6a1868;">year</span><span style="color: #50a14f;">-01-01 -e </span><span style="color: #b751b6;">$</span><span style="color: #6a1868;">year</span><span style="color: #50a14f;">-12-31 -u chaeck"</span> | qsub -N dl_sst_$<span style="color: #6a1868;">year</span> -q day
</pre>
......@@ -279,6 +295,13 @@ examples:
<div id="outline-container-org64c0bac" class="outline-4">
<h4 id="org64c0bac"><span class="section-number-4">2.1.2.</span> Download CHL</h4>
<div class="outline-text-4" id="text-2-1-2">
<p>
<a href="https://resources.marine.copernicus.eu/product-detail/OCEANCOLOUR_GLO_CHL_L3_REP_OBSERVATIONS_009_085/INFORMATION">Global Ocean Chlorophyll, PP and PFT (Copernicus-GlobColour)</a>
</p>
<p>
Dataset name: globcolour | directory: GlobColour/1days/L3
</p>
<div class="org-src-container">
<pre class="src src-bash"><span style="color: #6a1868;">year</span>=<span style="color: #da8548; font-weight: bold;">2007</span>; <span style="color: #986801;">echo</span> <span style="color: #50a14f;">"bash DIR/Download/download_cmems.sh -d oceancolour_glo_chl_l3_rep_observations_009_085 -p CHL -r GS -s </span><span style="color: #b751b6;">$</span><span style="color: #6a1868;">year</span><span style="color: #50a14f;">-01-01 -e </span><span style="color: #b751b6;">$</span><span style="color: #6a1868;">year</span><span style="color: #50a14f;">-12-31 -u chaeck"</span> | qsub -N dl_chl -q day
</pre>
......@@ -286,24 +309,63 @@ examples:
</div>
</div>
</div>
<div id="outline-container-org031cf67" class="outline-3">
<h3 id="org031cf67"><span class="section-number-3">2.2.</span> MODIS data</h3>
<div id="outline-container-orgb64e62e" class="outline-3">
<h3 id="orgb64e62e"><span class="section-number-3">2.2.</span> MODIS 1km data</h3>
<div class="outline-text-3" id="text-2-2">
<p>
For 1km resolution data.
Use is limited for results, so this one is for the records. On top of that it is quite a pain in the butt to setup.
Because this dataset is <b>much</b> more heavy, scripts run monthly instead of yearly.
The first script download L2 swaths that intersect a defined region, from cmr.earthdata.nasa.gov. Once downloaded each swath is regrided on a 1km grid.
Finally all swaths are merged daily. It is necessary to check that the correct number of swaths are downloaded (see the corresponding script).
</p>
<div class="org-src-container">
<pre class="src src-bash"><span style="color: #986801;">date</span>=<span style="color: #da8548; font-weight: bold;">200701</span>; <span style="color: #986801;">echo</span> <span style="color: #50a14f;">"bash Fronts/Compute/regrid/download_regrid.sh </span><span style="color: #b751b6;">$</span><span style="color: #6a1868;">date</span><span style="color: #50a14f;">"</span> | qsub -N dl_regrid_$<span style="color: #986801;">date</span> -q days3
</pre>
</div>
</div>
<div id="outline-container-org0b587d0" class="outline-4">
<h4 id="org0b587d0"><span class="section-number-4">2.2.1.</span> Finding images</h4>
<div class="outline-text-4" id="text-2-2-1">
<p>
Because the recovered data is only from the Aqua satellite, the cloud coverage is quite high. To avoid problems that <b>could</b> arise because of that (and because it reduces the amount of data to process), we find windows with low cloud coverage in daily snapshot, and only work on those afterwards. Each window is called an &rsquo;image&rsquo;. Scripts that work only on cloud-free images usually contain <code>_images</code>.
To find images and save their position for later:
</p>
<div class="org-src-container">
<pre class="src src-bash"><span style="color: #6a1868;">year</span>=<span style="color: #da8548; font-weight: bold;">2007</span>; <span style="color: #986801;">echo</span> <span style="color: #50a14f;">"bash Fronts/Compute/extract_images.py </span><span style="color: #b751b6;">$</span><span style="color: #6a1868;">date</span><span style="color: #50a14f;">"</span> | qsub -N fd_img_$<span style="color: #986801;">date</span> -q short
</pre>
</div>
<p>
This saves for each daily snapshot a txt file containing all the images positions (in pixel and lat/lon). Images can then be recovered using the <a href="../lib/data/images.py">../lib/data/images.py</a> dataset.
This script uses a fortran code, it has to be compiled before-hand (python will complain with a usefull warning if not). That could be rewritten in Xarray mind you.
</p>
</div>
</div>
</div>
</div>
<div id="outline-container-org87a6804" class="outline-2">
<h2 id="org87a6804"><span class="section-number-2">3.</span> Pre-processing data</h2>
<div class="outline-text-2" id="text-3">
</div>
<div id="outline-container-orgce23ed5" class="outline-3">
<h3 id="orgce23ed5"><span class="section-number-3">3.1.</span> OSTIA SST</h3>
<div class="outline-text-3" id="text-3-1">
<p>
We do a little processing before continuing. This script changes the units to celsius, rename the sst variable, removes the ice-covered pixels, drop unecessary variables, and regrid to EPSG32662 (see <a href="grids.html">Grids</a>).
Processed file are <code>SST_processed_[date].nc</code>, original files are to be deleted manually (if needed).
</p>
<div class="org-src-container">
<pre class="src src-bash"><span style="color: #6a1868;">year</span>=<span style="color: #da8548; font-weight: bold;">2007</span>; <span style="color: #986801;">echo</span> <span style="color: #50a14f;">"bash DIR/Download/process_ostia_sst.py </span><span style="color: #b751b6;">$</span><span style="color: #6a1868;">year</span><span style="color: #50a14f;">"</span> | qsub -N process_sst_$<span style="color: #6a1868;">year</span> -q short
</pre>
</div>
</div>
</div>
</div>
</div>
<div id="postamble" class="status">
<p class="author">Author: Clément Haëck</p>
<p class="date">Created: 2022-01-05 mer. 20:01</p>
<p class="date">Created: 2022-01-06 jeu. 13:48</p>
</div>
</body>
</html>
......@@ -7,15 +7,15 @@ This describes the important scripts, the order and the manner they should be ex
Most scripts work on a single year passed by the argument '-year' (see [[file:args.org][Args]]).
* Python scripts execution
Most of the script are python. The packages must be defined, see file:setup.org::Requirements. I recommend an anaconda environment.
The appropriate environmental variables must be set, especially ~$PYTHONPATH~ must include the root directory of the project, see file:setup.org::Setup.
Most of the script are python. The packages must be defined, see [[file:setup.org::Requirements][Setup::Requirements]]. I recommend an anaconda environment.
The appropriate environmental variables must be set, especially ~$PYTHONPATH~ must include the root directory of the project, see [[file:setup.org::Setup][Setup]].
On ciclad, this is all done by launching a script through:
#+begin_src bash
echo "bash $SUBMESO_COLOR_CODE_DIR/launch_python.sh $SUBMESO_COLOR_CODE_DIR/whatever/script.py arguments" | qsub
#+end_src
On my personnal machine, I use [[https://github.com/direnv/direnv][direnv]] that automatically reads /.envrc and setup everything when entering the project directory in a terminal. Python script can just be run normally.
On my personnal machine, I use [[https://github.com/direnv/direnv][direnv]], which automatically reads /.envrc and setup everything when entering the project directory in a terminal. Python scripts can just be run normally.
* Downloading Data
~$SUBMESO_COLOR_CODE_DIR~ (or whatever folder name you use) is abbrievated 'DIR' hereafter.
......@@ -24,16 +24,41 @@ Use file:../Download/download_cmems.sh.
See -h argument for help.
examples:
*** Download SST
[[https://resources.marine.copernicus.eu/product-detail/SST_GLO_SST_L4_REP_OBSERVATIONS_010_024/INFORMATION][ESA SST CCI and C3S reprocessed sea surface temperature analyses]]
Dataset name: ostia | directory: OSTIA/1days
#+begin_src bash
year=2007; echo "bash DIR/Download/download_cmems.sh -d sst_rep_obs_010_024 -p SST -r GS -s $year-01-01 -e $year-12-31 -u chaeck" | qsub -N dl_sst_$year -q day
#+end_src
*** Download CHL
[[https://resources.marine.copernicus.eu/product-detail/OCEANCOLOUR_GLO_CHL_L3_REP_OBSERVATIONS_009_085/INFORMATION][Global Ocean Chlorophyll, PP and PFT (Copernicus-GlobColour)]]
Dataset name: globcolour | directory: GlobColour/1days/L3
#+begin_src bash
year=2007; echo "bash DIR/Download/download_cmems.sh -d oceancolour_glo_chl_l3_rep_observations_009_085 -p CHL -r GS -s $year-01-01 -e $year-12-31 -u chaeck" | qsub -N dl_chl -q day
#+end_src
** MODIS data
For 1km resolution data.
** MODIS 1km data
Use is limited for results, so this one is for the records. On top of that it is quite a pain in the butt to setup.
Because this dataset is *much* more heavy, scripts run monthly instead of yearly.
The first script download L2 swaths that intersect a defined region, from cmr.earthdata.nasa.gov. Once downloaded each swath is regrided on a 1km grid.
Finally all swaths are merged daily. It is necessary to check that the correct number of swaths are downloaded (see the corresponding script).
#+begin_src bash
date=200701; echo "bash Fronts/Compute/regrid/download_regrid.sh $date" | qsub -N dl_regrid_$date -q days3
#+end_src
*** Finding images
Because the recovered data is only from the Aqua satellite, the cloud coverage is quite high. To avoid problems that *could* arise because of that (and because it reduces the amount of data to process), we find windows with low cloud coverage in daily snapshot, and only work on those afterwards. Each window is called an 'image'. Scripts that work only on cloud-free images usually contain ~_images~.
To find images and save their position for later:
#+begin_src bash
year=2007; echo "bash Fronts/Compute/extract_images.py $date" | qsub -N fd_img_$date -q short
#+end_src
This saves for each daily snapshot a txt file containing all the images positions (in pixel and lat/lon). Images can then be recovered using the file:../lib/data/images.py dataset.
This script uses a fortran code, it has to be compiled before-hand (python will complain with a usefull warning if not). That could be rewritten in Xarray mind you.
* Pre-processing data
** OSTIA SST
We do a little processing before continuing. This script changes the units to celsius, rename the sst variable, removes the ice-covered pixels, drop unecessary variables, and regrid to EPSG32662 (see [[file:grids.org][Grids]]).
Processed file are ~SST_processed_[date].nc~, original files are to be deleted manually (if needed).
#+begin_src bash
year=2007; echo "bash DIR/Download/process_ostia_sst.py $year" | qsub -N process_sst_$year -q short
#+end_src
......@@ -3,7 +3,7 @@
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<!-- 2022-01-05 mer. 20:01 -->
<!-- 2022-01-06 jeu. 13:46 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Zones and land</title>
......@@ -239,7 +239,7 @@ Those rasters are created using <code>Compute/make_land.py</code>. This script u
</div>
<div id="postamble" class="status">
<p class="author">Author: Clément Haëck</p>
<p class="date">Created: 2022-01-05 mer. 20:01</p>
<p class="date">Created: 2022-01-06 jeu. 13:46</p>
</div>
</body>
</html>
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment