HiPeRTA_Stream issueshttps://gitlab.in2p3.fr/CTA-LAPP/rta/HiPeRTA_Stream/-/issues2023-08-03T10:40:20+02:00https://gitlab.in2p3.fr/CTA-LAPP/rta/HiPeRTA_Stream/-/issues/15Measure start and stop time of SAG-RECO2023-08-03T10:40:20+02:00POLLET VincentMeasure start and stop time of SAG-RECOACADA is required to re-start in less than 2 minutes. This requirement is not yet enforced but will have to be in the final delivered ACADA software. Many ACADA sub-systems are started in parallel, but we are probably one of the last as ...ACADA is required to re-start in less than 2 minutes. This requirement is not yet enforced but will have to be in the final delivered ACADA software. Many ACADA sub-systems are started in parallel, but we are probably one of the last as we're started by SAG-supervisor. Its not clear what our budget in the 2 minutes is going to be. Because we rely on slurm, we should make sure we don't get delays due to the scheduler too.
To get an idea of where we stand, we should measure the start and stop time of SAG-RECO, that is the time between the `hiperta_stream_start` command and the `R0-DL1` job running, for instance recording the the time before `hiperta_stream_start` and have a script check the running jobs to get the time of `R0-DL1` running. For shutdown we could measure the time between when we receive the shutdown command and the RECO process exiting.
We might also want to measure what happens with several `hiperta_stream_start` command at the same time, which will happen when several telescope will be observing.
FYI @sami.caroff @scaroff @pauberthttps://gitlab.in2p3.fr/CTA-LAPP/rta/HiPeRTA_Stream/-/issues/14Update random forests used in DL1 to DL2 tests with a version supported by la...2023-07-04T18:00:13+02:00POLLET VincentUpdate random forests used in DL1 to DL2 tests with a version supported by latest scikit-learn and remove scikit learn dependencyUpdate random forests used in DL1 to DL2 tests with a version supported by latest scikit-learn and remove scikit learn dependency.
The very small random forests used in the unit tests aren't working with scikit-learn>=1.3 : https://gitl...Update random forests used in DL1 to DL2 tests with a version supported by latest scikit-learn and remove scikit learn dependency.
The very small random forests used in the unit tests aren't working with scikit-learn>=1.3 : https://gitlab.in2p3.fr/CTA-LAPP/rta/HiPeRTA_Stream/-/jobs/709024#L936POLLET VincentPOLLET Vincenthttps://gitlab.in2p3.fr/CTA-LAPP/rta/HiPeRTA_Stream/-/issues/13Update conversor2023-06-06T11:12:25+02:00Enrique GarciaUpdate conversorCurrently not compatible with ctapipe ~0.12.1
https://gitlab.in2p3.fr/CTA-LAPP/rta/HiPeRTA_Stream/-/jobs/523170Currently not compatible with ctapipe ~0.12.1
https://gitlab.in2p3.fr/CTA-LAPP/rta/HiPeRTA_Stream/-/jobs/523170Enrique GarciaEnrique Garciahttps://gitlab.in2p3.fr/CTA-LAPP/rta/HiPeRTA_Stream/-/issues/12allow passing reservation queue to `rta_simtel_to_fullr0_per_run`2023-06-06T11:15:24+02:00Enrique Garciaallow passing reservation queue to `rta_simtel_to_fullr0_per_run`Allow to pass a specific IT LP reservation queue by specifying the name
https://gitlab.in2p3.fr/CTA-LAPP/rta/HiPeRTA_Stream/-/blob/master/hiperta_stream/simtel_to_hdf5_conversion/simtel_to_full_r0_hdf5_per_run.py#L82Allow to pass a specific IT LP reservation queue by specifying the name
https://gitlab.in2p3.fr/CTA-LAPP/rta/HiPeRTA_Stream/-/blob/master/hiperta_stream/simtel_to_hdf5_conversion/simtel_to_full_r0_hdf5_per_run.py#L82Enrique GarciaEnrique Garciahttps://gitlab.in2p3.fr/CTA-LAPP/rta/HiPeRTA_Stream/-/issues/11update hiperta_stream_configuration.yml file with last changes2023-06-06T11:16:47+02:00Enrique Garciaupdate hiperta_stream_configuration.yml file with last changesso that rta can be launched by passing only the run number.so that rta can be launched by passing only the run number.Enrique GarciaEnrique Garciahttps://gitlab.in2p3.fr/CTA-LAPP/rta/HiPeRTA_Stream/-/issues/10hiperta_stream_start does not pass correctly config arguments2023-06-06T11:18:04+02:00Enrique Garciahiperta_stream_start does not pass correctly config argumentsthe `hiperta_stream_start` copies on one hand, either the standard config files, either the one passed through arguments to the $DIR/$RUN.
However, at the same time, we specified on the `hiperta_stream_configuration.yml` the name of th...the `hiperta_stream_start` copies on one hand, either the standard config files, either the one passed through arguments to the $DIR/$RUN.
However, at the same time, we specified on the `hiperta_stream_configuration.yml` the name of the lstchain configs... together with the `abs_common_path` changed on the flight.
Clear conflict.
Possible solution: Once `hiperta_stream_start` copies the config file, we rename it so that there is no conflict with the argument passed to slurm, through the `hiperta_stream_configuration` later.Enrique GarciaEnrique Garciahttps://gitlab.in2p3.fr/CTA-LAPP/rta/HiPeRTA_Stream/-/issues/9improve `simtel_to_hdf5_conversion` submodule2023-06-06T11:20:19+02:00Enrique Garciaimprove `simtel_to_hdf5_conversion` submodule`simtel_to_hdf5_conversion` submodile still uses `os.system` as `os.join`.
change them (and tests) for `subprocess.run` and for pathlib.Path objects`simtel_to_hdf5_conversion` submodile still uses `os.system` as `os.join`.
change them (and tests) for `subprocess.run` and for pathlib.Path objectsEnrique GarciaEnrique Garciahttps://gitlab.in2p3.fr/CTA-LAPP/rta/HiPeRTA_Stream/-/issues/8Add load astropy to all calib script2023-05-22T18:22:57+02:00Enrique GarciaAdd load astropy to all calib scripthttps://trello.com/c/3QvHx1vs/62-add-loadastropy-to-the-calibration-scripthttps://trello.com/c/3QvHx1vs/62-add-loadastropy-to-the-calibration-scriptEnrique GarciaEnrique Garciahttps://gitlab.in2p3.fr/CTA-LAPP/rta/HiPeRTA_Stream/-/issues/7Move scripts from HiPeShiftScript to this project2023-05-22T18:22:57+02:00Enrique GarciaMove scripts from HiPeShiftScript to this projectEnrique GarciaEnrique Garciahttps://gitlab.in2p3.fr/CTA-LAPP/rta/HiPeRTA_Stream/-/issues/6Implement `rta_tree_path` to `create_log_summary` script2023-06-06T11:21:04+02:00Enrique GarciaImplement `rta_tree_path` to `create_log_summary` scriptThink where and how the .txt files with the `stream_r0_dl1` jobids will be stored, and where two write this log.Think where and how the .txt files with the `stream_r0_dl1` jobids will be stored, and where two write this log.Enrique GarciaEnrique Garciahttps://gitlab.in2p3.fr/CTA-LAPP/rta/HiPeRTA_Stream/-/issues/5create stream stop and stream kill2023-06-06T11:24:00+02:00Enrique Garciacreate stream stop and stream killscancel filesscancel filesEnrique GarciaEnrique Garciahttps://gitlab.in2p3.fr/CTA-LAPP/rta/HiPeRTA_Stream/-/issues/4unable to create hdf5 config2023-06-06T11:24:27+02:00Vuillaumeunable to create hdf5 config```
root@01b7d1f63d69:/Data# python3 HiPeRTA_Stream/hiperta_stream/create_base_hdf5_config_structure_for_hiperta_stream.py -c HiPeRTA/share/HiPeRTA/default_configuration.h5 -g $DRS4 -p HiPeRTA/share/HiPeRTA/LST-1.1.Run00442.0000_pixel_or...```
root@01b7d1f63d69:/Data# python3 HiPeRTA_Stream/hiperta_stream/create_base_hdf5_config_structure_for_hiperta_stream.py -c HiPeRTA/share/HiPeRTA/default_configuration.h5 -g $DRS4 -p HiPeRTA/share/HiPeRTA/LST-1.1.Run00442.0000_pixel_order_bin.npy -o $DATADIR/base_config_stream.h5
Traceback (most recent call last):
File "HiPeRTA_Stream/hiperta_stream/create_base_hdf5_config_structure_for_hiperta_stream.py", line 225, in <module>
main()
File "HiPeRTA_Stream/hiperta_stream/create_base_hdf5_config_structure_for_hiperta_stream.py", line 217, in main
update_calibration_data(hout, args.gain_pedestal, args.sample_offset)
File "HiPeRTA_Stream/hiperta_stream/create_base_hdf5_config_structure_for_hiperta_stream.py", line 48, in update_calibration_data
hfile_gain_ped = tables.open_file(gain_and_pedestal, 'r')
File "/usr/local/lib/python3.8/dist-packages/tables/file.py", line 315, in open_file
return File(filename, mode, title, root_uep, filters, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/tables/file.py", line 778, in __init__
self._g_new(filename, mode, **params)
File "tables/hdf5extension.pyx", line 492, in tables.hdf5extension.File._g_new
tables.exceptions.HDF5ExtError: HDF5 error back trace
File "H5F.c", line 509, in H5Fopen
unable to open file
File "H5Fint.c", line 1400, in H5F__open
unable to open file
File "H5Fint.c", line 1700, in H5F_open
unable to read superblock
File "H5Fsuper.c", line 411, in H5F__super_read
file signature not found
End of HDF5 error back trace
Unable to open/create file '/Data/DATA/real/calibration/20200218/v05/drs4_pedestal.Run2005.0000.fits'https://gitlab.in2p3.fr/CTA-LAPP/rta/HiPeRTA_Stream/-/issues/3create_base_config not installed2021-05-06T19:25:18+02:00Vuillaumecreate_base_config not installedREADME not up to date ?README not up to date ?https://gitlab.in2p3.fr/CTA-LAPP/rta/HiPeRTA_Stream/-/issues/2CONDA should not be part of the requirements2023-05-22T18:22:57+02:00VuillaumeCONDA should not be part of the requirementsconfig in $CONDA?config in $CONDA?https://gitlab.in2p3.fr/CTA-LAPP/rta/HiPeRTA_Stream/-/issues/1path to images in reorganized dl1 is wrong2023-05-22T18:22:57+02:00Vuillaumepath to images in reorganized dl1 is wrongVuillaumeVuillaume