Skip to content
GitLab
Projects
Groups
Snippets
Help
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Open sidebar
pipelet
Pipelet
Commits
c4d1d03c
Commit
c4d1d03c
authored
Jan 26, 2011
by
Betoule Marc
Browse files
starting a modified environment for branch 1.1
parent
f09dca85
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
58 additions
and
9 deletions
+58
-9
README.org
README.org
+30
-4
pipelet/environment.py
pipelet/environment.py
+28
-5
No files found.
README.org
View file @
c4d1d03c
...
...
@@ -52,6 +52,10 @@ Pipelet is a free framework which helps you :
adapt to different hardware and software architectures.
*** What's new in v1.1
- The glob_seg behavior has been modified for coherence, convenience,
and performance sake. See [[*The%20segment%20environment][The segment environment]].
** Getting started
*** Pipelet installation
**** Dependencies
...
...
@@ -121,7 +125,6 @@ This command ends up with the creation of directory named pipename wich contains
The next section describes those two files in more details.
** Writing Pipes
*** Pipeline architecture
...
...
@@ -348,8 +351,16 @@ The segment code is executed in a specific environment that provides:
2. Functionalities to use the automated hierarchical data storage system.
- =get_data_fn(basename)=: complete the filename with the path to the working directory.
- =glob_seg(seg, regexp)=: return the list of filename in segment seg
working directory matching regexp.
- =glob_seg(seg, regexp)=: Return the list of filename matching the pattern y in the
data directory of parent tasks from the parent segment x.
- =glob_seg_all(seg, regexp)=: Return the list of filename matching
y in the working directory of segment x independantly of
whether the file comes from a task related to the current
task. glob_seg_all is provided to reproduce the behaviour of
old glob_seg for backward compatibility. Its usage should be limited as it:
- potentially breaks the dependancy scheme.
- may hurt performances as all task directories of the segment
x will be searched.
- =get_tmp_fn()=: return a temporary filename.
3. Functionalities to use the automated parameters handling
...
...
@@ -1169,7 +1180,22 @@ And the corresponding cgi script:
#+end_src
* The Pipelet actors
* Contributing to pipelet
** Improvements suggestion
As far as possible, we try to keep pipelet ReadMe driven. If you want
to ask for new features or modifications to pipelet, a good way to do
it is under the form of a patch to the README.org file. If you are
using git:
- Figure out how actually would like pipelet to behave
- Edit the README.org in consequence. Org files are simple text files
that can be conveniently edited using emacs org-mode.
- Commit your proposition with a relevant commit message.
=git commit -a=
- Format the patch for email submission, and send it to us
=git send-email --to HEAD^=
** The Pipelet actors
This section document the code for developers. The code documentation
can be built using the doxygen configuration file
...
...
pipelet/environment.py
View file @
c4d1d03c
...
...
@@ -182,12 +182,33 @@ class Environment(EnvironmentBase):
self
.
logger
.
info
(
"hooking %s"
%
hook_name
)
return
self
.
_hook
(
hook_name
,
glo
)
def
glob_seg
(
self
,
x
,
y
):
""" Globbing limited to the fatherhood
For unlimited globbing see glob_seg_all.
Parameters
----------
x: string, segment name
y: string, globbing pattern
Returns
-------
Return the list of filename matching the pattern y in the
data directory of parent tasks from the parent segment x.
"""
segx
=
self
.
_worker
.
pipe
.
find_seg
(
self
.
_worker
.
task
.
seg
,
x
)
if
segx
is
None
:
self
.
logger
.
warning
(
"No parent segment matching %s found"
%
x
)
def
glob_seg_all
(
self
,
x
,
y
):
""" Return the list of filename matching y in the working
directory of segment x.
Usage of glob_seg_all should be limited:
- potentially breaks the dependancy scheme
- May hurt performances as all task directories of the segment x will be searched
Parameters
----------
x: string, segment name
...
...
@@ -205,7 +226,7 @@ class Environment(EnvironmentBase):
+
glob
(
path
.
join
(
self
.
_worker
.
pipe
.
get_data_dir
(
segx
),
path
.
join
(
'*/'
,
y
)))
def
logged_subprocess
(
self
,
args
,
shell
=
False
,
except_on_failure
=
True
):
def
logged_subprocess
(
self
,
args
,
shell
=
False
,
except_on_failure
=
True
,
name
=
None
):
""" Execute a subprocess and log its output.
Create files process_name.log and process_name.err
...
...
@@ -221,8 +242,10 @@ class Environment(EnvironmentBase):
tuple (outputfile, errorfile)
"""
proc
=
args
[
0
]
of
=
self
.
get_data_fn
(
proc
+
'.log'
)
ef
=
self
.
get_data_fn
(
proc
+
'.err'
)
if
name
is
None
:
name
=
proc
of
=
self
.
get_data_fn
(
name
+
'.log'
)
ef
=
self
.
get_data_fn
(
name
+
'.err'
)
o
=
file
(
of
,
'w'
)
e
=
file
(
ef
,
'w'
)
e
.
write
(
'#'
+
' '
.
join
([
str
(
a
)
for
a
in
args
])
+
'
\n
'
)
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment