Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-4611

Implement distributable real-time gain calibration

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • Data Processing
    • Hide

      This is primarily for the benefit of enabling tied-array beamforming modes, but almost more importantly about verifying that we understand how to tackle such tricky distributed real-time processes.

      Show
      This is primarily for the benefit of enabling tied-array beamforming modes, but almost more importantly about verifying that we understand how to tackle such tricky distributed real-time processes.
    • Hide

      Agreed:

      1. Towards distributed code:
        1. Receive visibilities in a distributed fashion 
             Demonstration: LEAPacc to ingest a mock datastream
        2. Stream partial solutions 
              Demonstration: LEAPacc writes it's solutions to a data queue
        3. Stretch: Implement stream processors that aggregate
              Demonstration: A aggregator function that can receive the output from the distrubuted LEAPacc (for interpolate for TAB directions in future PIs)

      Previous:

      1. Pipeline script available that can:
        1. Receive visibilities in a distributed fashion using the real-time visibility streaming interface (i.e. ska-sdp-realtime-receive-processors / ska-sdp-dal), i.e. different processes are configured to receive different frequency chunks (as configured in an execution block).
        2. Stream partial solutions / normal equations to data queues (Kafka topics) using the existing libraries (ska-sdp-dataqueues)
          1. This might involve defining a suitably flexible xarray-based data model for partial solutions / normal equations
          2. Stretch: Demonstrate that this can be distributed and scaled by streaming data to different Kafka topic partitions
        3. Implement stream processors that aggregate and solve partial solutions / normal equations and emit the (possibly also partial) solutions back on Kafka queues
          1. Stretch: Allow re-visiting visibility data for refining solutions at later stages - likely somewhat tricky as it would require keeping visibilities in-memory
        4. Implement final aggregation / interpolation step that generates solutions for CBF onto the queues
      2. Stretch: Use previous solutions to seed / linearise subsequent distributed real-time calibration solver iterations. Might require changing normal equations data model?
      3. Stretch: Demonstrate integration with receive (allow working on live SPEAD stream)
      4. Stretch: Demonstrate integration with queue connector (expose calibration solution over Tango)
      Show
      Agreed: Towards distributed code: Receive visibilities in a distributed fashion      Demonstration: LEAPacc to ingest a mock datastream Stream partial solutions       Demonstration: LEAPacc writes it's solutions to a data queue Stretch: Implement stream processors that aggregate     Demonstration: A aggregator function that can receive the output from the distrubuted LEAPacc (for interpolate for TAB directions in future PIs) Previous: Pipeline script available that can: Receive visibilities in a distributed fashion using the real-time visibility streaming interface (i.e. ska-sdp-realtime-receive-processors / ska-sdp-dal), i.e. different processes are configured to receive different frequency chunks (as configured in an execution block). Stream partial solutions / normal equations to data queues (Kafka topics) using the existing libraries (ska-sdp-dataqueues) This might involve defining a suitably flexible xarray-based data model for partial solutions / normal equations Stretch: Demonstrate that this can be distributed and scaled by streaming data to different Kafka topic partitions Implement stream processors that aggregate and solve partial solutions / normal equations and emit the (possibly also partial) solutions back on Kafka queues Stretch: Allow re-visiting visibility data for refining solutions at later stages - likely somewhat tricky as it would require keeping visibilities in-memory Implement final aggregation / interpolation step that generates solutions for CBF onto the queues Stretch: Use previous solutions to seed / linearise subsequent distributed real-time calibration solver iterations. Might require changing normal equations data model? Stretch: Demonstrate integration with receive (allow working on live SPEAD stream) Stretch: Demonstrate integration with queue connector (expose calibration solution over Tango)
    • 8
    • 8
    • 0
    • Team_YANDA
    • Sprint 5
    • AA2 SDP_AA2_services_realtime_pipelines

    Description

      See Feature frame on DP ART board.

      Context

      Beamformer gain calibration (for tied-array beamformer) is one of the most taxing tasks that realtime processing will need to support. We must assume that towards AA2 and AA* both the amount of data involved and the complexity of the computations will become too high for a single node to tackle. Therefore we must demonstrate early that we can distribute this critical pipeline.

      References

      Attachments

        Structure

          Activity

            People

              p.wortmann Wortmann, Peter
              m.ashdown Ashdown, Mark
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Feature Progress

                Story Point Burn-up: (0%)

                Feature Estimate: 8.0

                IssuesStory Points
                To Do10.0
                In Progress   00.0
                Complete00.0
                Total10.0

                Dates

                  Created:
                  Updated:

                  Structure Helper Panel