Skip to content
GitLab
Projects
Groups
Snippets
Help
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Open sidebar
pipelet
Pipelet
Commits
92de9c89
Commit
92de9c89
authored
Nov 23, 2010
by
Maude Le Jeune
Browse files
doc depend directive
parent
41716875
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
33 additions
and
3 deletions
+33
-3
README.org
README.org
+33
-3
No files found.
README.org
View file @
92de9c89
...
...
@@ -221,9 +221,6 @@ directives are present in the segment code the last one is retained.
that are identical. To make use of group, elements of the output set
have to be hashable.
*** Depend directive
*** Orphan segments
By default, orphan segments have no input argument (an empty list),
...
...
@@ -234,6 +231,36 @@ If P is an instance of the pipeline object:
P.push (segname=seg_input)
*** Depend directive
As explained in the introduction section, Pipelet offers the
possibility to spare CPU time by saving intermediate products on disk.
We call intermediate products the input/output data files of the
different segments.
Each segment repository is identified by an unique key which depends
on:
- the segment processing code and parameters (segment and hook
scripts)
- the input data (identified from the key of the parent segments)
Every change made on a segment (new parameter or new parent) will then
give a different key, and tell the pipelet engine to compute a new
segment instance.
It is possible to add some external dependencies to the key
computation using the depend directive:
#depend file1 file2
At the very beginning of the pipeline execution, all dependencies will
be stored, to prevent any change (code edition) between the key
computation and actual processing.
Note that this mecanism works only for segment and hook
scripts. External dependencies are also read as the beginning of the
pipeline execution, but only used for the key computation.
*** Hierarchical data storage
This system provides versionning of your data and easy access through
...
...
@@ -265,6 +292,9 @@ The storage is organized as follows:
one of its parent's one :
/prefix/segname_YFLJ65/segname2_PLMBH9/
- etc...
*** The segment environment
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment