Skip to content
GitLab
Projects
Groups
Snippets
Help
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Open sidebar
pipelet
Pipelet
Commits
0eba1ebe
Commit
0eba1ebe
authored
Nov 24, 2010
by
Maude Le Jeune
Browse files
main.py update
parent
478b769e
Changes
3
Hide whitespace changes
Inline
Side-by-side
Showing
3 changed files
with
26 additions
and
10 deletions
+26
-10
README.org
README.org
+16
-2
doc/2010-nov-cmb/seg_scheme.dia
doc/2010-nov-cmb/seg_scheme.dia
+0
-0
pipelet/static/main.py
pipelet/static/main.py
+10
-8
No files found.
README.org
View file @
0eba1ebe
...
...
@@ -66,7 +66,7 @@ sudo python setup.py install
1. Run the test pipeline
cd test
cd test
/first_test
python main.py
2. Add this pipeline to the web interface
...
...
@@ -555,7 +555,7 @@ the different parameters, and also, how to write a
main python script can be used as any binary from the command line
(including options parsing).
*** Common options
Some options are common to each running modes.
**** log level
...
...
@@ -625,6 +625,8 @@ w.run()
In this mode, each tasks will be computed in a sequential way.
Do not hesitate to invoque the Python debugger from IPython : %pdb
To use the interactive mode, run:
main.py -d
*** The process mode
...
...
@@ -635,6 +637,11 @@ The number of subprocess is set by the N parameter :
from pipelet.launchers import launch_process
launch_process(P, N)
To use the process mode, run:
main.py
or
main.py -p 4
*** The batch mode
In this mode, one can submit some batch jobs to execute the tasks.
The number of job is set by the N parameter :
...
...
@@ -653,7 +660,14 @@ It is possible to specify some job submission options like:
The 'server' option can be disable to add some workers to an existing
scheduler.
To use the batch mode, run:
main.py -b
to start the server, and:
main.py -a 4
to add 4 workers.
** Browsing Pipes
*** The pipelet webserver and ACL
...
...
doc/2010-nov-cmb/seg_scheme.dia
View file @
0eba1ebe
No preview for this file type
pipelet/static/main.py
View file @
0eba1ebe
...
...
@@ -80,12 +80,14 @@ def main(print_info=print_info):
import
optparse
parser
=
optparse
.
OptionParser
()
parser
.
add_option
(
'-a'
,
'--add-workers'
,
metavar
=
'N'
,
help
=
'Submit N supplementary jobs without launching a new server.'
,
type
=
'int'
)
parser
.
add_option
(
'-d'
,
'--debug'
,
help
=
'Start jobs in interactive mode'
,
action
=
"store_true"
,
default
=
False
)
parser
.
add_option
(
'-p'
,
'--process'
,
metavar
=
'N'
,
help
=
'Launch jobs as local parallel processes'
,
type
=
'int'
)
help
=
'Launch jobs as local parallel processes'
,
type
=
'int'
,
default
=
1
)
parser
.
add_option
(
'-b'
,
'--batch'
,
help
=
'Start a server. Use -a to add workers after this.'
,
action
=
"store_true"
,
default
=
False
)
parser
.
add_option
(
'-a'
,
'--add-workers'
,
metavar
=
'N'
,
help
=
'Submit N supplementary jobs without launching a new server.'
,
type
=
'int'
)
(
options
,
args
)
=
parser
.
parse_args
()
...
...
@@ -96,14 +98,14 @@ def main(print_info=print_info):
if
options
.
debug
:
w
,
t
=
launch_interactive
(
P
,
log_level
=
log_level
)
w
.
run
()
## Process mode
elif
options
.
process
:
launch_process
(
P
,
options
.
process
,
log_level
=
log_level
)
## PBS mode
# Batch mode
elif
options
.
batch
:
launch_pbs
(
P
,
0
,
address
=
(
os
.
environ
[
'HOST'
],
50000
),
server
=
True
,
job_name
=
job_name
,
cpu_time
=
cpu_time
,
job_header
=
job_header
,
log_level
=
log_level
)
elif
options
.
add_workers
:
launch_pbs
(
P
,
options
.
add_workers
,
address
=
(
os
.
environ
[
'HOST'
],
50000
),
job_name
=
job_name
,
cpu_time
=
cpu_time
,
job_header
=
job_header
)
## Process mode
else
:
launch_p
bs
(
P
,
1
,
address
=
(
os
.
environ
[
'HOST'
],
50000
),
job_name
=
job_name
,
cpu_time
=
cpu_time
,
server
=
True
,
job_header
=
job_header
,
log_level
=
log_level
)
launch_p
rocess
(
P
,
options
.
process
,
log_level
=
log_level
)
if
print_info
:
print
"1- Run 'pipeweb track <shortname> %s'
\n
to add the pipe to the web interface.
\n
"
%
(
P
.
sqlfile
)
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment