As we develop real-time pipeline functionality, the behaviours that SDP will need to implement will become more complex, from ingesting various data streams while publishing calibration and QA data, towards eventually implementing batch processing to delivering data products. The data flows and the way that processing scripts orchestrate them in other components of the system has grown in an ad-hoc manner. Revising how data is written to the configuration database and data queues should set us up better for the challenges of AA2 integration.
As we develop real-time pipeline functionality, the behaviours that SDP will need to implement will become more complex, from ingesting various data streams while publishing calibration and QA data, towards eventually implementing batch processing to delivering data products. The data flows and the way that processing scripts orchestrate them in other components of the system has grown in an ad-hoc manner. Revising how data is written to the configuration database and data queues should set us up better for the challenges of AA2 integration.
Configuration library is refactored to be able to track schema-checked entities in consistent manner, including owner and state information (e.g. via data classes) (Yanda)
A small data queue library is developed that allows pushing schema-checked xarray Dataset (or DataArray) objects to data queues (e.g. via xradio schema checker). Should ideally be fit for usage outside of SDP. (Orca)
Scripting library gains ability to generate "data flow" entities into configuration database (Orca)
Stretch: LMC queue connector Tango device is configured using the data flow entries.
Stretch: QA display is configured using the data flow entries. (Naledi)
Stretch: Visibility receive processing script generates data flow entries and the visibility receiver uses them to configure itself.
Stretch: Pointing calibration processing script generates data flow entries.
Configuration library is refactored to be able to track schema-checked entities in consistent manner, including owner and state information (e.g. via data classes) ( Yanda )
A small data queue library is developed that allows pushing schema-checked xarray Dataset (or DataArray) objects to data queues (e.g. via xradio schema checker). Should ideally be fit for usage outside of SDP. ( Orca )
Scripting library gains ability to generate "data flow" entities into configuration database ( Orca )
Stretch: LMC queue connector Tango device is configured using the data flow entries.
Stretch: QA display is configured using the data flow entries. ( Naledi )
Stretch: Visibility receive processing script generates data flow entries and the visibility receiver uses them to configure itself.
Stretch: Pointing calibration processing script generates data flow entries.