Add docs

README.md

@@ -9,34 +9,8 @@ Collocate submesoscale fronts to phytoplankton levels using satellite imagery
 - Compute: Process data, compute diagnostics,... 
 - Plots: Plotting scripts
 
-## Setup
 
-The environment variable `$SUBMESO_COLOR_CODE_DIR` must be defined and point to
-this repository location.
-Bash scripts will source the `setup_env.sh` file in it, which defines other necessary
-variables, notably `$SUBMESO_COLOR_DATA_DIR` for the location of data and plots.
-It also appends this repository to the `$PYTHONPATH` variable.
-The organisation of `setup_env.sh` is left to the user.
-An example is given which works on a personnal computer and on the [Ciclad](https://mesocentre.ipsl.fr/) cluster.
+## Doc
 
-
-## Requirements
-
-    python3.8
-        -numpy
-        -scipy
-        -pandas
-        -xarray
-        -dask
-        -matplotlib
-        -netcdf4
-        -motuclient
-        -cftime
-        -shapely
-        -global-land-mask
-        -filefinder
-    
-- [dateloop](https://github.com/Descanonge/dateloop)
-    
-A conda environment file is provided ([./env.yml](./env.yml))
+Some documentation is available in orgmode format. See ([./docs/index](Index)).
 

docs/args.org

+#+TITLE: Args
+
+All scripts are written with a command-line use in mind.
+To change some parameters, command-line arguments are retrieved. As always, things are quite automated.
+All the magic happens in ~lib.get_args()~.
+
+* Usage
+Typically a script is written as:
+#+begin_src python
+import lib
+
+def main(args):
+    pass
+
+if __name__ == '__main__':
+    args = lib.get_args(['region', 'days', 'fixes'])
+
+    main(args)
+#+end_src
+
+`args` is a dictionnary containing arguments name and their values.
+
+* Default arguments
+The list of arguments names passed above are from a list of default arguments.
+I won't keep this updated so see the source code of ~lib.get_args()~ to have a list of thoses.
+These arguments have a default value that can be set in `lib/conf.ini`
+
+Some arguments go through some processing.
+** date
+date is a YYYY/MM/[DD] format argument. The '/' separator can also be '-', a space, or nothing. ~args['date_str']~ is a tuple of the strings of year, month, and eventually day. ~args['date']~ stores a datetime object.
+** fixes
+Fixes are supplied as '-fix <matcher> <string>'. They can then be found as a dictionnary with the matchers name as keys. See [[*Fixes]]
+
+* Adding other arguments
+Other arguments can be retrieved like so:
+#+begin_src python
+def add_args(parser):
+    parser.add_argument('-argument_name', type=int, default=0)
+
+args = lib.get_args(['...'], add_args)
+#+end_src
+
+* Fixes
+Fixes are a important feature of the filefinder package. When [[file:datasets.org::*Pregex][finding datafiles]], one can 'fix' a matcher (part of the filename that vary, the date for daily files for instance) to a certain value.
+That can be a string (which can be a regex!), or a value. Quick example, we only want the month of may:
+#+begin_src python
+finder = lib.data.dataset.get_finder()
+finder.fix_matchers(m=5)
+#+end_src
+
+Now, we can do this automatically from the command line by supplying the appropriate argument: ~python script.py -fix m 05~ (we can only supply a string, so it must match the date format). This will end up in the args dictionnary, where it can be passed to ~get_finder()~ or ~get_data()~. So:
+#+begin_src python
+args = lib.get_args(['fixes'])
+ds = lib.data.dataset.get_data(args)
+#+end_src
+`ds` will only have data for the month of may. Convenient when testing a script on smaller data.
+
+* Argument processing
+You may have noticed that the function described in [[file:datasets.org::*create_data][create_data]] have the following signature ~(args: Dict = None, **kwargs: Any)~.
+Thoses functions process thoses arguments using ~lib.data.process_args(args_names, args, replace_defaults=None, **kwargs)~. The idea is that we define the argument the function *needs* with args_names. We then supply both args and kwargs. From those are only kept the needed arguments. If arguments are missing a default value is used (either defined with lib/conf.ini or replace_defaults). Arguments found in kwargs will replace those from args.
+
+This system is a bit heavy but convenient, as the variable args can be passed to all functions from different datasets without worrying about which needs what. And if argument are missing, or must be overwritten kwargs can still be used.

docs/boxes.org

+#+TITLE: Boxes
+
+We sometime needs to define a rectangular area (a zone clean of clouds for instance).
+`lib.box` defines several classes to help with this notion.
+With those, it is easy to define a rectangular area. It can be used to slice a dataset easily. It can also be plotted easily.

docs/datasets.org

+#+TITLE: Datasets
+
+* Location
+All data is found in ~$SUBMESOCOLOR_DATA_DIR/[Region]/[dataset name / abbreviation]~.
+The region folder is made to be able to work on different locations at the same time.
+For now I am personally using 'GS' for 'Gulf-Stream'.
+
+* Module file
+To load a dataset in memory, its corresponding python file can be used. It can be found in `lib/data/[module].py`.
+In this file can be found several things.
+** Some constants
+*** Directory arguments
+A set of necessary arguments to get the root directory of data (typically its ~{'region', 'days'}~). See [[file:args.org]].
+*** Pregex
+The files pre-regex. See [[*create_data]].
+*** Grid
+The grid for this data. See [[file:grids.org]].
+** get_root
+A function that returns the root directory of data (including the [[*Time dependency]] folder).
+** create_data
+At this point, we have all the information we need to find data files and load the data in memory. The steps described next are automated using the ~lib.data.create_data~ function. All functions receive the same arguments,
+The first step is to retrieve the data files. Typically there is one file per day, or sometimes per set of parameter. We could use a simple loop or globbing, but this nowhere near as fun as what we do.
+We use the [[https://github.com/Descanonge/filefinder]] package. It allows to find in the root directory all files corresponding to a specific filename structure, specified with the [[*Pregex]]. See the package readme and documentation to see what its other nice features.
+For that step we use the ~get_finder()~ function, which returns a Finder object.
+The next step is to feed those files to Xarray with the function ~get_data()~.
+It returns a dataset using ~xr.open_mfdataset()~.
+
+These functions are created automatically, but some options can be passed to customize some behaviors. See the function docstring.
+
+* Time dependency
+Datasets are typically available daily, as n-days average, or as climatology.
+Each variation has its own subfolder of form '1days', '{n}days', 'climato'.
+For more on the climatology, see [[file:climato.org]].
+
+* Zones
+We define some static 'zones', this dataset is quite special and have its own doc [[file:zones.org][here]].

docs/grids.org

+#+TITLE: Grids
+
+Because of the datasets that I use, I currently uses 3 different grids.
+
+* 1km
+A 1km grid for the MODIS dataset.
+
+* 4km
+Two 4km grids for GlobColour and OSTIA.
+Thoses grids have a slightly different resolution. But we need to overlap (pixel by pixel) the Chl-a data with the HI data (computed from SST), so on 2 different grids.
+To avoid this hassle, we regrid the OSTIA SST on the GlobColour grid with `Compute/process_ostia_sst.py`. We also don't have to deal with 2 grids anymore.
+This was choosen recently, so for a time I dealt with 2 grids. This is why most scripts (and zone/land rasters, see [[file:zones.org]]) actually support choosing one grid or the other, using the grid argument.
+
+** OSTIA grid
+** GlobColour grid

docs/hi.org

+#+TITLE: Hi
+
+* HI definition
+* Compilation
+How to compile.
+How it works (the script).

docs/histograms.org

+#+TITLE: Histograms
+
+* Method of computation
+advantages / defaults of using xhistogram

docs/index.org

+#+TITLE: Index
+
+This is this project documentation.
+It is far from covering everything but should put you on the right track.
+
+* Folders
+** Compute
+For scripts that compute stuff.
+** Download
+For downloading and pre-processing data.
+** Docs
+You're looking at it.
+** lib
+General modules
+*** data
+[[file:datasets.org][Datasets]]
+** Plots
+Plots. Folder is a hot mess. Lot of scripts are not kept up to date. Use at your own risk.
+* Other
+** [[file:setup.org][Setup]]

docs/setup.org

+#+TITLE: Setup
+
+First step is to clone this repository into a folder of your choice.
+
+* Requirements
+** Python
+Python >= 3.8 is necessary.
+Thoses packages:
+- numpy
+- scipy
+- pandas
+- xarray
+- dask
+- matplotlib
+- netcdf4
+- motuclient
+- cftime
+- shapely
+- global-land-mask
+- filefinder
+
+A conda environment file with pinned versions is provided at [[file:../env.yml]].
+To install: ~conda env create -f ./env.yml~ and to update: ~conda env update -f ./env.yml~.
+
+** Dateloop
+A small script to loop over dates. It is used in some scripts.
+Installation details over at: [[https://github.com/Descanong/dateloop]]
+
+* Setup
+Some other work is necessary to finalize the setup.
+The environment variable ~$SUBMESO_COLOR_CODE_DIR~ must be defined and point to
+this repository root directory.
+Bash scripts will source the `setup_env.sh` file in it, which defines other necessary
+variables, notably ~$SUBMESO_COLOR_DATA_DIR~ for the location of data and plots.
+It also appends this repository to the ~$PYTHONPATH~ variable, so that all modules are available.
+The organisation of `setup_env.sh` is left to the user. An example is given which works on a personnal computer and on the [[https://mesocentre.ipsl.fr][Ciclad]] cluster.
+
+On ciclad, python script must be executed by using [[file:../launch_python.sh]].

docs/zones.org

+#+TITLE: Zones and land
+
+Zones and land rasters are accessible from lib.zones.
+
+* Zones
+We sometime have to define static, geometric zones.
+We use a custom Zone class. An instance is created by passing a list of points coordinates. We keep a shapely object, it can be useful for some geometric consideration (see interactions with [[file:boxes.org]]).
+When running `lib/zones.py`, all zones are rasterized on the different [[file:grids.org][grids]] defined. Those rasters are saved to files that can be loaded as any other dataset.
+
+* Land
+It is useful to have a raster that say for every pixel if there is land or not.
+Those rasters can be loaded with ~lib.zones.get_land()~ which works as any other get_data() function.
+Those rasters are created using `Compute/make_land.py`. This script uses either downloaded data containing masks, or the [[https://github.com/toddkarin/global-land-mask][global_land_mask]] package.

launch_python.sh

 #!/usr/bin/env bash
 # Launch a python script on Ciclad
+# All arguments are passed to the python script.
 
 #PBS -v SUBMESO_COLOR_CODE_DIR