Details
-
Spike
-
Must have
-
None
-
Data Processing
-
-
-
3
-
3
-
20
-
Team_HIPPO, Team_NZAPP, Team_SCHAAP, Team_YANDA
-
Sprint 3
-
-
-
-
11.6
-
Stories Completed, Solution Intent Updated, Outcomes Reviewed, Satisfies Acceptance Criteria, Accepted by FO
Description
Spike: review, discuss, and progress detailed processing function library interface.
- Discuss, define and document ( ! ) a number of processing function interfaces. At minimum:
- "High level", like visibility stream currently implemented by receive (containing by design all meta-data to interpret data)
- "Low level", like gridding (focusing on a very particular problem)
- Especially consider modifiability mechanisms:
- How do we make sure that we can adjust interfaces if we find them to hinder progress?
- How do we handle having multiple implementations of the same interface?
- Define API(s) we would use to interact with them ("wrappers")
- We likely want at minimum an easy-to-use way to call these from Python. There should be a Python library you can install to gain access to processing functions
- However, we might also want to consider providing more general-purpose interfaces to allow execution engines to bind against it (i.e. something for tools like EAGLE to work from).
- Note that the Python library might just act as a thin interface layer to external tools. E.g. for visibility streaming it might just wrap the ska-sdp-dal call with some schema checks.
- Data model considerations
- Are we still okay with Arrow as the base data models?
- Do we need a more lower-level representation for CUDA/C?
- Specifically for the Python implementation wrapping/unwrapping into more useful high-level data wrappers (numpy/awkward) is likely appropriate. Try to shoot for Xarray?
- Consider interaction with accelerators - e.g. would we try to treat on-accelerator memory objects the same, or have separate types/kernels? Responsibility for managing this would normally be on execution engines, but maybe we would also have wrappers do this automatically?
- Consideration of design constraints from current processing interfaces compared against SDP architecture.
