smp-beta_1.0.sif deppol no answer
En utilisant une version de deppol installé CC, donc pas celle du conteneur, mais exécuté dans l'env du conteneur, deppol ne rend pas la main
On commence par ouvrir le conteneur smp-beta_1.0.sif en mode interactif, puis on positionne le package phdev au CC
ccwislurm0001:apptainer>apptainer shell --bind /pbs,/sps/ztf --shell /usr/local/bin/_apptainer_shell.sh smp-beta_1.0.sif
Apptainer> pwd
/sps/ztf/share/apptainer
Apptainer> cd /sps/ztf/users/colley/
Apptainer> ls
env env_poloka.sh hist.txt install poloka_waf runs smp_sif
Apptainer> cd smp_sif/
Apptainer> ll
bash: ll: command not found
Apptainer> ls
phdev smp_env_ccin2p3.sh
Apptainer> more smp_env_ccin2p3.sh
export PATH=/sps/ztf/users/colley/smp_sif/phdev/deppol:$PATH
export PYTHONPATH=/sps/ztf/users/colley/smp_sif/phdev/tools:$PYTHONPATH
export SCENE_PATH=/sps/ztf/data/storage/scenemodeling/
Apptainer> . smp_env_ccin2p3.sh
Apptainer> which deppol
/sps/ztf/users/colley/smp_sif/phdev/deppol/deppol
Apptainer>
Apptainer>
Apptainer> deppol --wd=/sps/ztf/data/storage/scenemodeling/pol_fields/test_apptainer --filtercode=zi --func=clean,retrieve_catalogs,preprocess --use-gaia-stars --rm-intermediates -j 40 --ztfname=600_05
Run parameters:
Namespace(ztfname=PosixPath('600_05'), n_jobs=40, wd=PosixPath('/sps/ztf/data/storage/scenemodeling/pol_fields/test_apptainer'), filtercode='zi', func='clean,retrieve_catalogs,preprocess', no_map=False, no_reduce=False, cluster_worker=0, scratch=None, from_scratch=False, exposure_workspace=None, lc_folder=None, astro_degree=5, dump_timings=False, rm_intermediates=True, synchronous_compute=False, astro_min_mag=None, log_std=False, max_seeing=4.0, photom_max_star_chi2=4.0, min_psfstars=None, use_raw=False, astro_max_chi2=4.0, discard_calibrated=False, dump_node_info=False, log_overwrite=False, parallel_reduce=False, refexp=None, compress=False, photom_cat=None, use_gaia_stars=True, ztfin2p3_path=None, ext_catalog_cache=None, recompute_ext_catalogs=False, footprints=None, starflats=False, photom_use_aper=False, ubercal_config_path=None)
Found 1 SN1a
Running pipeline:
clean -> retrieve_catalogs -> preprocess
Running a local cluster with 40 processes.
Dask dashboard at: http://127.0.0.1:8787/status
Building job list for 600_05-zi... 8 quadrants found.
Pipeline function "clean". (building reduce job)(binding)Reduction operation.
Pipeline function "retrieve_catalogs". (building reduce job)(binding)Reduction operation.
Pipeline function "preprocess". (building map jobs)(checkpoint)(binding)8 map operations.
Running. Processing 8 mappings. Processing 2 reductions.
================================================================================
600_05-zi
2024-03-28 14:27:17.776769
Running reduction retrieve_catalogs
Retrieving catalogs for (fieldid, rcid) pairs: [(600, 18), (600, 19), (600, 16), (600, 17)]
Retrieving external catalogs
Getting catalog gaia...
600-18
Catalog size=6210
600-19
Catalog size=11322
600-16
Catalog size=16661
600-17
Catalog size=20913
Saving catalog into /sps/ztf/data/storage/scenemodeling/pol_fields/test_apptainer/600_05/zi/catalogs/gaia_full.parquet
Getting catalog ps1...
600-18
Catalog size=27130
600-19
Catalog size=50114
600-16
Catalog size=72781
600-17
Catalog size=90813
Saving catalog into /sps/ztf/data/storage/scenemodeling/pol_fields/test_apptainer/600_05/zi/catalogs/ps1_full.parquet
Matching Gaia and PS1 catalogs
Saving Gaia/PS1 matched catalogs
================================================================================
après un ctrl+ C
KeyboardInterrupt
Process Dask Worker process (from Nanny):
Traceback (most recent call last):
File "/opt/conda/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/distributed/nanny.py", line 983, in run
await worker.finished()
File "/opt/conda/lib/python3.11/site-packages/distributed/core.py", line 630, in finished
await self._event_finished.wait()
File "/opt/conda/lib/python3.11/asyncio/locks.py", line 213, in wait
await fut
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/lib/python3.11/site-packages/distributed/process.py", line 202, in _run
target(*args, **kwargs)
File "/opt/conda/lib/python3.11/site-packages/distributed/nanny.py", line 1021, in _run
asyncio_run(run(), loop_factory=get_loop_factory())
File "/opt/conda/lib/python3.11/site-packages/distributed/compatibility.py", line 204, in asyncio_run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/asyncio/runners.py", line 123, in run
raise KeyboardInterrupt()
KeyboardInterrupt
Process Dask Worker process (from Nanny):
Traceback (most recent call last):
File "/opt/conda/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/distributed/nanny.py", line 983, in run
await worker.finished()
File "/opt/conda/lib/python3.11/site-packages/distributed/core.py", line 630, in finished
await self._event_finished.wait()
File "/opt/conda/lib/python3.11/asyncio/locks.py", line 213, in wait
await fut
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/lib/python3.11/site-packages/distributed/process.py", line 202, in _run
target(*args, **kwargs)
File "/opt/conda/lib/python3.11/site-packages/distributed/nanny.py", line 1021, in _run
asyncio_run(run(), loop_factory=get_loop_factory())
File "/opt/conda/lib/python3.11/site-packages/distributed/compatibility.py", line 204, in asyncio_run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/asyncio/runners.py", line 123, in run
raise KeyboardInterrupt()
KeyboardInterrupt
on a l'impression que le scheduler de job mouline dans le vide.
Edited by COLLEY Jean-marc