files being appended to will only be uploaded once
@madrouin I think there is a potential improvement for appended files, unless I miss something. Please advise and I can implement solution 3
scenario:
- 1 h files %H%M%S.dat created on the hour
- latest file appended to every 1 minute
- lftp runs every 5 minutes
example :
- file 020000.dat is created at 1 am with first data
- lftp runs at 1:03 am and transfers 020000.dat with three appended values/rows
- lftp runs at 1:08 am and sees that 020000.dat has already been transferred: the file is not appended to on the ftp destination
- 2:00 am the file 020000.dat is complete on the source but is never fully uploaded to the destination (only the first 3 appended values/rows)
solutions:
1 run rigid timings (e.g. lftp runs every 5 minutes on the hour) - not good when time of data append is not reliable
2 update min_date
https://gitlab.in2p3.fr/ipsl/sirta/tools/files-transfer/-/blob/main/lftp_transfer.py#L208 to have a time offset (grace_time
) for evaluating files that are transferred. e.g. min_date - grace_time. The grace time is for files that may not be completely ready yet (not uploaded from instrument to logger storage) or that may not be totally appended to yet (e.g. in the above scenario)
3 others?
Edited by Will Morrison