Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-3830

Improve SDP real-time system configurability and maintainability

Details

    • Enabler
    • Must have
    • PI22
    • COM SDP SW
    • None
    • Data Processing
    • Hide

      As we develop real-time pipeline functionality, the behaviours that SDP will need to implement will become more complex, from ingesting various data streams while publishing calibration and QA data, towards eventually implementing batch processing to delivering data products. The data flows and the way that processing scripts orchestrate them in other components of the system has grown in an ad-hoc manner. Revising how data is written to the configuration database and data queues should set us up better for the challenges of AA2 integration.

      Show
      As we develop real-time pipeline functionality, the behaviours that SDP will need to implement will become more complex, from ingesting various data streams while publishing calibration and QA data, towards eventually implementing batch processing to delivering data products. The data flows and the way that processing scripts orchestrate them in other components of the system has grown in an ad-hoc manner. Revising how data is written to the configuration database and data queues should set us up better for the challenges of AA2 integration.
    • Hide
      1. Configuration library is refactored to be able to track schema-checked entities in consistent manner, including owner and state information (e.g. via data classes) (Yanda)
      2. A small data queue library is developed that allows pushing schema-checked xarray Dataset (or DataArray) objects to data queues (e.g. via xradio schema checker). Should ideally be fit for usage outside of SDP. (Orca)
      3. Scripting library gains ability to generate "data flow" entities into configuration database (Orca)
      4. Stretch: LMC queue connector Tango device is configured using the data flow entries.
      5. Stretch: QA display is configured using the data flow entries. (Naledi)
      6. Stretch: Visibility receive processing script generates data flow entries and the visibility receiver uses them to configure itself.
      7. Stretch: Pointing calibration processing script generates data flow entries.
      Show
      Configuration library is refactored to be able to track schema-checked entities in consistent manner, including owner and state information (e.g. via data classes) ( Yanda ) A small data queue library is developed that allows pushing schema-checked xarray Dataset (or DataArray) objects to data queues (e.g. via xradio schema checker). Should ideally be fit for usage outside of SDP. ( Orca ) Scripting library gains ability to generate "data flow" entities into configuration database ( Orca ) Stretch: LMC queue connector Tango device is configured using the data flow entries. Stretch: QA display is configured using the data flow entries. ( Naledi ) Stretch: Visibility receive processing script generates data flow entries and the visibility receiver uses them to configure itself. Stretch: Pointing calibration processing script generates data flow entries.
    • Intra Program
    • 8
    • 8
    • 0
    • Team_NALEDI, Team_ORCA, Team_YANDA
    • Sprint 5
    • PI22 - UNCOVERED

    Attachments

      Issue Links

        Structure

          Activity

            People

              m.ashdown Ashdown, Mark
              m.ashdown Ashdown, Mark
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Feature Progress

                Story Point Burn-up: (75.00%)

                Feature Estimate: 8.0

                IssuesStory Points
                To Do813.0
                In Progress   37.0
                Complete2260.0
                Total3380.0

                Dates

                  Created:
                  Updated:

                  Structure Helper Panel