Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-1980

Initial Flagging + RCAL pipeline for AA0.5

    XporterXMLWordPrintable

Details

    • Enabler
    • Must have
    • PI12
    • COM SDP SW
    • None
    • Data Processing
    • Hide

      In the SKA design, SDP is able to run a real-time flagging + calibration workflow concurrently with receive of visibility data which in turn provides complex gain solutions to the correlator-beamformer (CBF) which are used to calibrate the tied-array beamformer. 

      For the AA0.5 system, this workflow will be either be a required SDP function or, at the very least be extremely valuable, to have for testing to reduce the system risk of operating this workflow which needs to exercise interfaces with multiple SKA software systems and provide a solution for a single bore-sight tied-array beam which may prove valuable for engineering and science commissioning tests.

      With an SDP system for AA0.5 anticipated to be in ~PI18/19, and with this functionality anticipated to take several more PIs to complete and test before we would be able to hand over a robust and resilient workflow for use with AA0.5 commissioning tests, continuing this development in PI12 is vital.

      Show
      In the SKA design, SDP is able to run a real-time flagging + calibration workflow concurrently with receive of visibility data which in turn provides complex gain solutions to the correlator-beamformer (CBF) which are used to calibrate the tied-array beamformer.  For the AA0.5 system, this workflow will be either be a required SDP function or, at the very least be extremely valuable, to have for testing to reduce the system risk of operating this workflow which needs to exercise interfaces with multiple SKA software systems and provide a solution for a single bore-sight tied-array beam which may prove valuable for engineering and science commissioning tests. With an SDP system for AA0.5 anticipated to be in ~PI18/19, and with this functionality anticipated to take several more PIs to complete and test before we would be able to hand over a robust and resilient workflow for use with AA0.5 commissioning tests, continuing this development in PI12 is vital.
    • Hide
      • A prototype pipeline capable of transferring data through the (RFI) flagger and calibration functions and generating QA metrics for each stage
        • This pipeline can be tested with simulated data but has to be considering a route towards integration and running concurrently with visibility received.
      • An initial prototype of real-time QA displays for the metrics generated by this pipeline (or possibly just one step in PI12 if tacking them both in a single PI is too ambitious)
      • Documented interface, QA metrics explanation and description.
      Show
      A prototype pipeline capable of transferring data through the (RFI) flagger and calibration functions and generating QA metrics for each stage This pipeline can be tested with simulated data but has to be considering a route towards integration and running concurrently with visibility received. An initial prototype of real-time QA displays for the metrics generated by this pipeline (or possibly just one step in PI12 if tacking them both in a single PI is too ambitious) Documented interface, QA metrics explanation and description.
    • 6
    • 6
    • 20
    • 3.333
    • Team_HIPPO, Team_ORCA
    • Sprint 5
    • Hide

      The RFI flagger from the PI12 was refactored to process data that are supplied using the Python interface and returns an array of flags produced by the RFI flagger. The changes and interface for calibration are described on the RFI Confluence page.

      The code, together with the Python interface, is in the
      ska-post-correlation-rfi-flagger repository.

      The flagger has been integrated into the RASCIL real-time calibration (RCAL) pipeline using the Python interface.

      To visualise the RFI flagger performance and help with the development of the RFI flagger, an initial version of QA displays and metrics were developed and integrated. The code for the QA displays is here and description of the display is on the Confluence page related to QA displays.

      Show
      The RFI flagger from the PI12 was refactored to process data that are supplied using the Python interface and returns an array of flags produced by the RFI flagger. The changes and interface for calibration are described on the RFI Confluence page . The code, together with the Python interface, is in the ska-post-correlation-rfi-flagger repository. The flagger has been integrated into the RASCIL real-time calibration (RCAL) pipeline using the Python interface. To visualise the RFI flagger performance and help with the development of the RFI flagger, an initial version of QA displays and metrics were developed and integrated. The code for the QA displays is here and description of the display is on the Confluence page related to QA displays .
    • 14.4
    • Outcomes Reviewed

    • PI22 - UNCOVERED

    Description

      Follows work from PI11 to simulate input data for this pipeline (SP-1670 & SP-1671) and to develop and test RFI flagging (SP-1673) and RCAL (SP-1674) functionality.

      Must progress solution towards a pipeline that can be deployed as a real-time SDP workflow in the AA0.5 system (by ~PI18/19). Not expecting integration of this PI but must be working in that direction and when developing this, be mindful of the roadmap towards that goal. Ideally, we should be aiming for a testable initial solution (using simulated data being sent through the CBF emulator) by ~PI15/16 in order to have a solution we can iterate on ahead of the first operation in the field.

      The first step towards this might be to agree on and implement an interface between the RFI flagger and RCal processing stages developed in PI11.

      The decisions around data format and relevant data transfer/streaming interface are to be made through discussions among the stakeholders. Currently, the data format used for testing both RFI flagging and RCAL independently is a Measurement Set but that will almost certainly no longer be suitable for real-time operation where we will almost certainly want to keep everything in RAM as much as possible and this should be very feasible for AA0.5 where the data rates are fairly low. 

      Interfaces and data format will also need to be developed for QA metrics and debugging displays.

      Attachments

        Issue Links

          relates to

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-1674 Initial prototype of AA0.5 RCAL pipeline functionality

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Done

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-1670 Simulation of data sets for early AA RT RFI pipeline testing

          • Should have - Items labelled as "Should have" are important but not necessary for delivery in the current delivery timebox. While "Should have" items can be as important as "Must have", they are often not as time-critical or there may be another way to satisfy the requirement so that it can be held back until a future delivery timebox.
          • Done

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-1671 Simulation of data sets for early AA RCAL pipeline testing

          • Should have - Items labelled as "Should have" are important but not necessary for delivery in the current delivery timebox. While "Should have" items can be as important as "Must have", they are often not as time-critical or there may be another way to satisfy the requirement so that it can be held back until a future delivery timebox.
          • Done

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-1673 Initial prototype of RT RFI flagging pipeline functionality (considering interfaces that can be tested with the AA0.5 system)

          • Should have - Items labelled as "Should have" are important but not necessary for delivery in the current delivery timebox. While "Should have" items can be as important as "Must have", they are often not as time-critical or there may be another way to satisfy the requirement so that it can be held back until a future delivery timebox.
          • Done

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-1901 Revised simulations and/or data sets for SDP image pipeline testing

          • Should have - Items labelled as "Should have" are important but not necessary for delivery in the current delivery timebox. While "Should have" items can be as important as "Must have", they are often not as time-critical or there may be another way to satisfy the requirement so that it can be held back until a future delivery timebox.
          • Done
          mentioned in

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Structure

            Activity

              People

                b.nikolic Nikolic, Bojan
                f.graser Graser, Ferdl
                Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 6.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete2041.0
                  Total2041.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel