Investigate possibility of time chunking
LISA Instrument currently is not suited for long simulations (more than a few months) because of memory pressure. However, we need to run year-long simulations, and therefore need to investigate solutions. One of these solutions is to use time chunking and memory optimizations (releasing memory once it's not needed by downstream processed), and Dask is a well-known framework that can help. A nice side effect is (hopefully) a speed up because of parallel execution (Dask is basically a task scheduler) and the compatibility with most of computing clusters.
We should investigate the use of Dask, i.e.
- Investigate potential show stoppers
- Evaluate what needs to be changed and developed specifically for Dask
- Have a demonstrator that it can work (not to waste resources)
- Make sure that we have the resources (maybe see if we can have students on this), divide up the work and do it
- Test the result, and evaluate the performance
Using Dask probably means that we have to move away from the ForEachObject
API, and use plain Dask arrays (and Numpy arrays under the hood, but the high-level interface is the same). This has been discussed at length (and investigated) by #34 (closed) and !46 (closed). We should use various elements of what's been discussed and implemented there.