Commit 92de9c89 authored by Maude Le Jeune's avatar Maude Le Jeune
Browse files

doc depend directive

parent 41716875
......@@ -221,9 +221,6 @@ directives are present in the segment code the last one is retained.
that are identical. To make use of group, elements of the output set
have to be hashable.
*** Depend directive
*** Orphan segments
By default, orphan segments have no input argument (an empty list),
......@@ -234,6 +231,36 @@ If P is an instance of the pipeline object:
P.push (segname=seg_input)
*** Depend directive
As explained in the introduction section, Pipelet offers the
possibility to spare CPU time by saving intermediate products on disk.
We call intermediate products the input/output data files of the
different segments.
Each segment repository is identified by an unique key which depends
- the segment processing code and parameters (segment and hook
- the input data (identified from the key of the parent segments)
Every change made on a segment (new parameter or new parent) will then
give a different key, and tell the pipelet engine to compute a new
segment instance.
It is possible to add some external dependencies to the key
computation using the depend directive:
#depend file1 file2
At the very beginning of the pipeline execution, all dependencies will
be stored, to prevent any change (code edition) between the key
computation and actual processing.
Note that this mecanism works only for segment and hook
scripts. External dependencies are also read as the beginning of the
pipeline execution, but only used for the key computation.
*** Hierarchical data storage
This system provides versionning of your data and easy access through
......@@ -265,6 +292,9 @@ The storage is organized as follows:
one of its parent's one :
- etc...
*** The segment environment
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment