Loading...

Change Owns to Parent Ofs

Set start and due date...

Xporter

XML

Word

Printable

Details

Type: Feature
Priority: Should have
Fix Version/s: PI15
Component/s: COM SDP SW
Labels:
None

ARTs:

Data Processing
Benefit hypothesis:

Hide

To satisfy a number of the AA0.5 commission scenarios, SDP will need to calculate complex gain corrections (Jones Matrices) for a single boresight (in the direction of the centre of the primary beam) tied-array beam and provide these to the CBF beamformer at regular intervals (ie in pseudo-real-time) during an observation.
This will require RFI flagging prior to complex gain calculations.

Show
To satisfy a number of the AA0.5 commission scenarios , SDP will need to calculate complex gain corrections (Jones Matrices) for a single boresight (in the direction of the centre of the primary beam) tied-array beam and provide these to the CBF beamformer at regular intervals (ie in pseudo-real-time) during an observation. This will require RFI flagging prior to complex gain calculations.
Acceptance criteria:
Hide

Integration of 2SM RFI flagger into PFL

Investigation of use of AOFlagger for comparison (using Python interface)

Comparison of flagging performance

Investigation of integration of RFI flagging into RCAL?
Show
Integration of 2SM RFI flagger into PFL Investigation of use of AOFlagger for comparison (using Python interface) Comparison of flagging performance Investigation of integration of RFI flagging into RCAL?
Expect Dependencies:

Intra Program
Feature Points:
3
Initial Size:
3
WSJF:
0
Informed By:

USE-22 Operate and Monitor Telescope
Epic Link:
SDP AA0.5 system
Agile Teams:

Team_HIPPO
Due Sprint:
Sprint 5
Story Point Burn-up:
Overdue:
Outcomes:

Hide

The two-state RFI flagger was merged into the processing function library and is documented (https://gitlab.com/ska-telescope/sdp/ska-sdp-func/-/blob/main/src/ska-sdp-func/twosm_rfi_flagger/sdp_twosm_rfi_flagger.cpp). Comparison of the two-state is made here (https://confluence.skatelescope.org/display/SE/HIP-262+Comparison+between+aoflagger+and+2SM). The two-state RFI flagger, when applied to both time and frequency, seems to be flagging RFI well in both simulation and real data. Comparison with AOflagger in the case of real data is difficult as there is no ground truth, and AOflagger uses different techniques, thus flagging different parts of the data. On the confluence page, different indicators are used in an attempt to quantify this. The speed of the two flaggers was also compared. However, this performance comparison is preliminary.

Show
The two-state RFI flagger was merged into the processing function library and is documented ( https://gitlab.com/ska-telescope/sdp/ska-sdp-func/-/blob/main/src/ska-sdp-func/twosm_rfi_flagger/sdp_twosm_rfi_flagger.cpp ). Comparison of the two-state is made here ( https://confluence.skatelescope.org/display/SE/HIP-262+Comparison+between+aoflagger+and+2SM ). The two-state RFI flagger, when applied to both time and frequency, seems to be flagging RFI well in both simulation and real data. Comparison with AOflagger in the case of real data is difficult as there is no ground truth, and AOflagger uses different techniques, thus flagging different parts of the data. On the confluence page, different indicators are used in an attempt to quantify this. The speed of the two flaggers was also compared. However, this performance comparison is preliminary.
Resolved PI.Sprint:
17.5

Feature Checklist:

Stories Completed, Outcomes Reviewed, Satisfies Acceptance Criteria, Accepted by FO

Demos:
- DP_ART_16.1
Requirement Status:

PI22 - UNCOVERED

Description

This ticket has been split from ~~SP-2556~~ to focus on the the RFI flagger availability for use in RCAL.

Who?

SDP system developer
SDP workflow developer
System Scientist

What? (outcomes)

RFI flaggers exercised via a python interface from the PFL (or external library) and compared for assessment of use in RCAL pipeline
Collaborate on the design and prototyping for feeding data to the provided interface (~~SP-2555~~) from the perspective of required cadence of buffering/flagging/providing solutions.
Stretch: Consideration for QA metrics

Why?

see benefit hypothesis field

References

See outcomes of https://jira.skatelescope.org/browse/SPO-1661
~~SP-2555~~: Interface for exposing/providing gain solutions to the CBF

Previous notes prior to cloning ticket

Not expecting to conclude this pipeline by the end of PI15 as we don't think 8FP is enough, however, we will want to demonstrate progress by the end of the PI so this will need careful planning and discussion as to what is possible. It would be a good idea to reach out to one of the system scientists to discuss this further.

While working on the interface and coupling with visibility receive is important, arguably first addressing calibration using a configurable block of visibilities averaged in channel and time dimensions and fitting/smoothing solutions in these dimensions is more time-critical if the workaround from PI14 for interface with the visibility receive workflow meets the data rate requirements for AA0.5.

We also likely need to look at how flagging is incorporated into this workflow, one might expect this pipeline to be able to use a provided mask and otherwise possibly make use of the python interface to AO Flagger which is a well-tested solution.

Attachments

Issue Links

relates to

SP-2494 Provide DI Jones Solver Python Package

Done

mentioned in: Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...

(2 mentioned in)

Structure

Activity

People

Assignee:: Mort, Ben

Reporter:: Mort, Ben

Votes:: 0 Vote for this issue

Watchers:: 0 Start watching this issue

Feature Progress

Story Point Burn-up: (100.00%)

Feature Estimate: 3.0

	Issues	Story Points
To Do	0	0.0
In Progress	0	0.0
Complete	13	20.5
Total	13	20.5

Dates

Created:: 08/Jun/22 3:59 PM

Updated:: 15/Apr/24 9:43 AM

Resolved:: 08/Feb/23 7:51 PM

RFI improvements for AA0.5 RCAL workflow (continued)