Commit 9808f95e authored by Maude Le Jeune's avatar Maude Le Jeune
Browse files

doc + repo docstring

parent 374f3175
Browsing Pipes
The pipelet webserver and ACL
The pipelet webserver allows the browsing of multiple pipelines.
Each pipeline has to be register using ::
pipeweb track <shortname> sqlfile
To remove a pipeline from the tracked list, use ::
pipeweb untrack <shortname>
As the pipeline browsing implies a disk parsing, some basic security
has to be set also. All users have to be register with a specific access
level (1 for read-only access, and 2 for write access). ::
pipeutils -a <username> -l 2 sqlfile
To remove a user from the user list::
pipeutils -d <username> sqlfile
Start the web server using ::
pipeweb start
Then the web application will be available on the web page http://localhost:8080
To stop the web server ::
pipeweb stop
The web application
In order to ease the comparison of different processing, the web
interface displays various views of the pipeline data :
The index page
The index page displays a tree view of all pipeline instances. Each
segment may be expand or reduce via the +/- buttons.
The parameters used in each segments are resumed and displayed with
the date of execution and the number of related tasks order by
A check-box allows to performed operation on multiple segments :
- deletion : to clean unwanted data
- tag : to tag remarkable data
The filter panel allows to display the segment instances with respect to 2
criterions :
- tag
- date of execution
The code page
Each segment names is a link to its code page. From this page the user
can view all python scripts code which have been applied to the data.
The tree view is reduced to the current segment and its related
The root path corresponding to the data storage is also displayed.
The product page
The number of related tasks, order by status, is a link to the product
pages, where the data can be directly displayed (if images, or text
files) or downloaded.
From this page it is also possible to delete a specific product and
its dependencies.
The log page
The log page can be acceded via the log button of the filter panel.
Logs are ordered by date.
......@@ -25,6 +25,7 @@ import sys, os
# Add any Sphinx extension module names here, as strings. They can be extensions
# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
autoclass_content = 'both'
extensions = ['sphinx.ext.autodoc', 'sphinx.ext.doctest', 'sphinx.ext.todo', 'sphinx.ext.coverage', 'sphinx.ext.pngmath', 'sphinx.ext.viewcode']
# Add any paths that contain templates here, relative to this directory.
.. _example:
The example pipelines
This example illustrates a very simple image processing use.
The problematic is the following : one wants to apply a Gaussian
filter in Fourier domain on several 2D images.
The pipe scheme is::
pipedot = """
where segment ``mkgauss`` computes the Gaussian filter, ``fftimg`` computes the
Fourier transforms of the input images, and ``convol`` performs the
filtering in Fourier domain, and the inverse transform of the filtered
images. ::
P = pipeline.Pipeline(pipedot, code_dir=op.abspath('./'), prefix=op.abspath('./'))
The pipe scheme is output as a .dot file, that can be converted to an
image file with the command line::
dot -Tpng -o pipeline.png
To apply this filter to several images (in our case 4 input images),
the pipe data parallelism is used.
From the main script, a 4-element list is pushed to the ``fftimg``
segment. ::
At execution, 4 instances of the =fftimg= segment will be
created, and each of them outputs one element of this list ::
img = get_input() #( - line 15)
set_output (img) #( - line 38)
On the other side, a single instance of the ``mkgauss`` segment will be
executed, as there is one filter to apply.
The last segment ``convol``, which depends on the two others, will be
executed with a number of instances that is the Cartesian product of
its 4+1 inputs (ie 4 instances)
The instance identifier which is set by the ``fftimg`` output, can be
retrieve with the following instruction: ::
img = get_input('fftimg') #( - line 12)
Running the pipe
Follow the same procedure than for the first example pipeline, to run
this pipe and browse the result.
Running the pipe
This CMB pipeline depends on two external python modules:
+ healpy :
+ spherelib:
This example illustrates a very simple CMB data processing use.
The problematic is the following : one wants to characterize the
inverse noise weighting spectral estimator (as applied to the WMAP 1yr
data). A first demo pipeline is built to check that the algorithm
has correctly been implemented. Then, Monte Carlo simulations are used
to compute error bars estimates.
A design pipeline
The design pipe scheme is: ::
pipe_dot = """
+ ``cmb``: generate a CMB map from LCDM power spectrum.
+ ``noise``: compute the mode coupling matrix from the input hit-count map
+ ``clnoise``: compute the empirical noise power spectrum from a noise
+ ``clcmb``: generate two noise realizations, add them to the CMB map, to compute a
first cross spectrum estimator. Then weighting mask and mode
coupling matrix are applied to get the inverse noise weighting
+ ``clplot``: make a plot to compare pure cross spectrum vs inverse noise
weighting estimators.
As the two first orphan segments depends on a single shared parameter
which is the map resolution nside, this argument is pushed from the
main script.
Another input argument of the cmb segment, is its simulation identifier,
which will be used for latter Monte Carlo. In order to push two
inputs to a single segment instance, we use python tuple data type.::
P.push(cmb=[(nside, 1)])
From the segment, those inputs are retrieved with : ::
(nside,sim_id) = get_input() ##( line 14)
nside = seg_input() ##( line 15)
The last segment produces a plot in which we compare:
+ the input LCDM power spectrum
+ the binned cross spectrum of the noisy CMB maps
+ the binned cross spectrum of which we applied hitcount weight and mode coupling matrix.
+ the noise power spectrum computed by clnoise segment.
In this plot we check that both estimators are corrects, and that the
noise level is the expected one.
From the design pipeline to Monte Carlo
As a second step, Monte Carlo simulations are used to compute error
The ``clnoise`` segment is no longer needed, so that the new pipe scheme
reads : ::
pipe_dot = """
We now use the native data parallelization scheme of the pipe to build
many instances of the ``cmb`` and ``clcmb`` segments. ::
cmbin = []
for sim_id in [1,2,3,4,5,6]:
cmbin.append((nside, sim_id))
......@@ -6,7 +6,13 @@
Welcome to Pipelet's documentation!
Pipelet is a free framework allowing for the creation, execution and
.. automodule:: pipelet
Pipelet is a free framework to handle the creation, execution and
browsing of scientific data processing pipelines. It provides:
+ easy chaining of interdependent elementary tasks,
......@@ -14,25 +20,22 @@ browsing of scientific data processing pipelines. It provides:
+ branch handling,
+ automated distribution of computational tasks.
Both engine and web interface are written in Python. As Pipelet is all
about chaining processing written in Python or using Python as a glue
language, prior knowledge of this language is required.
Both engine and web interface are written in Python.
.. As Pipelet is all about chaining processing written in Python or using Python as a glue language, prior knowledge of this language is required.
.. automodule:: pipelet
Table of contents:
**Table of contents:**
.. toctree::
:maxdepth: 1
:maxdepth: 2
Tutorial <tutorial>
Advanced usage <userguide>
Contributing to pipelet <extra>
Why using pipelines
The pipeline mechanism allows to apply a sequence of processing
steps to some data, in a way that the input of each process is the
output of the previous one. Making visible these different processing
steps, in the right order, is essential in data analysis to keep track
of what you did, and make sure that the whole processing remains
How it works
Pipelet is based on the possibility to save on disk every intermediate
input or output of a pipeline, which is usually not a strong
constraint but offers a lot of benefits. It means that you can stop
the processing whenever you want, and start it again without
recomputing the whole thing: you just take the last products you have
on disk, and continue the processing where it stopped. This logic is
interesting when the computation cost is higher than the cost of disk
space required by intermediate products.
In addition, the Pipelet engine has been designed to
process *data* *sets*. It takes advantage of the parallelisation
opportunity that comes with data which share the same structure (data
arrays), to dispatch the computational tasks on parallel architecture.
The data dependency scheme is also used to save CPU time, and allows
to handle very big data sets processing.
The Pipelet functionalities
Pipelet is a free framework which helps you :
+ to write and manipulate pipelines with any dependency scheme,
+ to keep track of what processing has been applied to your data and perform comparisons,
+ to carry pipelines source code from development to production and adapt to different hardware and software architectures.
What's new in v1.1
.. _hyperlink-name: link-block
+ Speed improvement during execution and navigation to handle pipeline of 100 thousand tasks.
+ Task repository versionning to manage ``group_by`` directive which uses different parent tasks list.
+ New ``glob_seg`` type utility to search data files from parent task only + improvement of I/O and parameters utilities. See :ref:`The segment environnement<my-reference-label>` section
+ Improvement of external dependencies management : the ``depend`` directive induces a copy of external dependencies, the version number (together with RCS revision if exist) of the imported modules are output.
+ Pickle file render available from the Web interface
.. _run:
Running Pipes
The sample main file
A sample main file is made available when creating a new Pipelet
framework. It is copied from the reference file ::
This script illustrates various ways of running pipes. It describes
the different parameters, and also, how to write a
main python script can be used as any binary from the command line
(including options parsing).
Common options
Some options are common to each running modes.
log level
The logging system is handle by the python logging facility module.
This module defines the following log levels :
+ ``DEBUG``
+ ``INFO``
+ ``ERROR``
All logging messages are saved in the different Pipelet log files,
available from the web interface (rotating file logging). It is also
possible to print those messages on the standard output (stream
logging), by setting the desired log level in the launchers options:
For example::
import logging
launch_process(P, N,log_level=logging.DEBUG)
If set to 0, stream logging will be disable.
.. note:: The matplotlib documentation says: "Many users report initial problems trying to use maptlotlib in web application servers, because by default matplotlib ships configured to work with a graphical user interface which may require an X11 connection. Since many barebones application servers do not have X11 enabled, you may get errors if you don’t configure matplotlib for use in these environments. Most importantly, you need to decide what kinds of images you want to generate (PNG, PDF, SVG) and configure the appropriate default backend. For 99% of users, this will be the Agg backend, which uses the C++ antigrain rendering engine to make nice PNGs. The Agg backend is also configured to recognize requests to generate other output formats (PDF, PS, EPS, SVG). "
The easiest way to configure matplotlib to use Agg is to call::
The ``matplotlib`` and ``matplotlib_interactive`` options turn the
matplotlib backend to Agg in order to allow the execution in
non-interactive environment. The two options affects independently the
non interactive execution mode and the interactive mode.
Those two options are set to ``True`` by default in the sample main
script. Setting them to False deactivate this behavior for pipelines
that make no use of matplotlib (and prevents the raise of an exception
if matplotlib is not even available).
The interactive mode
This mode has been designed to ease debugging. If ``P`` is an instance of
the pipeline object, the syntax reads ::
from pipelet.launchers import launch_interactive
w, t = launch_interactive(P)
In this mode, each tasks will be computed in a sequential way.
Do not hesitate to invoque the Python debugger from IPython :``%pdb``
To use the interactive mode, run:: -d
The process mode
In this mode, one can run simultaneous tasks (if the pipe scheme
allows to do so).
The number of subprocess is set by the N parameter ::
from pipelet.launchers import launch_process
launch_process(P, N)
To use the process mode, run::
or:: -p 4
The batch mode
In this mode, one can submit some batch jobs to execute the tasks.
The number of job is set by the N parameter ::
from pipelet.launchers import launch_pbs
launch_pbs(P, N , address=(os.environ['HOST'],50000))
It is possible to specify some job submission options like:
+ job name
+ job header: this string is prepend to the PBS job scripts. You may
want to add some environment specific paths. Log and error files are
automatically handled by the pipelet engine, and made available from
the web interface.
+ cpu time: syntax is: "hh:mm:ss"
The ``server`` option can be disable to add some workers to an existing
To use the batch mode, run:: -b
to start the server, and:: -a 4
to add 4 workers.
Getting started
Pipelet installation
+ Running the *pipelet* engine requires *Python* >= 2.6.
+ The web interface of *pipelet* requires the installation of the *cherrypy3* Python module (on Debian: aptitude install python-cherrypy3).
+ Although default Python installation provides the *sqlite3* module, you may not be able to use it. In that case, you can manually install the *pysqlite2* module.
You may find useful to install some generic scientific tools that nicely interact with *pipelet*:
+ *numpy*
+ *matplotlib*
+ *latex*
Getting Pipelet
.. note:: The first version of the software is currently in the process of stabilisation. The Pipelet engine has now reached the level of desired sophistication. On the other hand, the user interface has been developped in a minimalist way. It includes the main functionalities but with a design which could and will be more user friendly.
Getting last pipelet version
.. code:: bash
git clone git://
Installing Pipelet
.. code:: bash
sudo python install
Running a simple test pipeline
1. Run the test pipeline::
cd test/first_test
2. Add this pipeline to the web interface::
pipeweb track test ./.sqlstatus
3. Set up an account in the access control list and launch the web server::
pipeutils -a username -l 2 .sqlstatus
pipeweb start
4. You should be able to browse the result on the web page
Getting a new pipe framework
To get a new pipeline framework, with example main and segment scripts::
pipeutils -c pipename
This command ends up with the creation of directory named pipename wich contains:
+ a main script (named **) providing functionnalities to execute your pipeline in various modes (debug, parallel, batch mode, ...)
+ an example of segment script (````) which illustrates the pipelet utilities with comments.
The next section describes those two files in more details.
This diff is collapsed.
.. _write:
Writing Pipes
Pipeline architecture
The definition of a data processing pipeline consists in :
+ a succession of python scripts, called segments, coding each step of the actual processing.
+ a main script that defines the dependency scheme between segments, and launch the processing.
The dependencies between segments must form a directed acyclic
graph. This graph is described by a char string using a subset of the
graphviz dot language ( For example the string::
a -> b -> d;
c -> d;
c -> e;
defines a pipeline with 5 segments ``{"a", "b", "c", "d", "e"}``. The
relation ``a->b`` ensures that the processing of the segment "a" will be
done before the processing of its 'child' segment ``b``. Also the output
of ``a`` will be fed as input for ``b``. In the given example, the node
``d`` has two parents ``b`` and ``c``. Both will be executed before ``d``. As
their is no relation between ``b`` and ``c`` which of the two will be
executed first is not defined.
When executing the segment ``seg``, the engine looks for a python script
named ````. If not found, it looks iteratively for script files
named ```` and ````. This way, different segments of the pipeline
can share the same code, if they are given a name with a common root
(this mechanism is useful to write generic segment and is completed by
the hooking system, described in the advanced usage section). The code
is then executed in a specific namespace (see below :ref:`The segment environment<my-reference-label>` section).
The Pipeline object
Practically, the creation of a Pipeline object requires 3 arguments::
from pipelet.pipeline import Pipeline
P = Pipeline(pipedot, code_dir="./", prefix="./")
+ ``pipedot`` is the string description of the pipeline
+ ``code_dir`` is the path where the segment scripts can be found
+ ``prefix`` is the path to the data repository (see below :ref:`Hierarchical data storage<hier-sec>`)
It is possible to output the graphviz representation of the pipeline
(requires the installation of graphviz). First, save the graph string
into a ``.dot`` file with the pipelet function::
Then, convert it to an image file with the dot command::
dot -Tpng -o pipeline.png
Dependencies between segments and data parallelism
The modification of the code of one segment will trigger its
recalculation and the recalculation of all the segments which
depend on it.
The output of a segment is a list of python objects. If a segment as
no particular output this list can be empty and do not need to be
specified. Elements of the list are allowed to be any kind of
pickleable python objects. However, a good practice is to fill the
list with the minimal set of characteristics relevant to describe the
output of the segment and to defer the storage of the data to
appropriate structures and file formats. For example, a segment which
performs computation on large images could virtually pass the results
of its computation to the following segment using the output list. It
is a better practice to store the resulting image in a dedicated file
and to pass in the list only the information allowing a non ambiguous