Details
-
Feature
-
Not Assigned
-
None
-
Data Processing
-
-
-
6
-
6
-
20
-
Team_NZAPP, Team_SCHAAP, Team_SIM, Team_YANDA
-
Sprint 5
-
-
-
-
10.4
-
Stories Completed, Integrated, Outcomes Reviewed, NFRS met, Demonstrated, Satisfies Acceptance Criteria, Accepted by FO
-
-
SPO-880
Description
<This ticket is expected to be a collaboration between 2 or more teams. While there is scope for decomposing this into multiple tickets during PI planning the intension is to keep the effort as aligned as possible.>
Following the outcome of SP-1326, this is a timeboxed activity to develop one or more initial continuum imaging pipelines (workflow scripts) that should become the baseline for future SDP workflow development in the period of PI9-PI12 (Feature Set: SP-1308)
In PI9 it is not anticipated that this workflow will integrate into the SDP operational system prototype. It is intended that this workflow is based on current state-of-the-art solutions/codes, following the approach that has already been started as part of the analysis of the SKA1 Instrumental Simulations initiative. When developing this workflow, it is important to consider the future development as an SDP architecturally compliant workflow; In particular, identifying well-defined levels of abstraction with strict interfaces between functional processing components and scalability and data distribution achieved through a two-tiered model where a high-level workflow tier orchestrates data movement and process execution, and a lower-tier supports potentially highly connected processing.
While the exact scope of the pipeline should be agreed following the outcome of SP-1326, the default expectation is that this pipeline will consist of a self-cal loop similar to that described by Figure 8-19 SKA-TEL-SKO-0001075-01_DesignBaselineDocument.
It is important that this pipeline script is made available in SKA Gitlab. This should be linked to a description of the processing steps along with their interfaces which where possible should be mapped to SDP architecture modules and components (continuing work started in SP-1326). Documentation should be either on Confluence or, preferably in the form of readthedocs page(s) accessible via the developer portal (and included or referenced from the page structure produced for SP-1328). Instructions on how to obtain and run the pipeline(s) should also be provided, with sufficient detail to allow another SDP developer to download and run the pipeline (with consideration for availability and well-defined software versioning), including how to obtain input data and check that outputs match expected results (which might form the basis of testing in future PIs).
It is also anticipated that in order to test the pipeline(s) generated by this ticket, additional simulations will need to be run with more complete coverage of the frequency band. This should be discussed as part of SP-1326 (if not before at PI planning) so that scheduling of dependencies can be properly managed and should follow previous SKA1 MID/LOW simulations (ie SP-1183, SP-1057, and SP-1056).
Attachments
Issue Links
- is blocked by
-
SP-1326 Determine initial design for a continuum pipeline and datasets
- Done
- Parent Of
-
SP-1472 Facet-based imaging of SKA1-LOW simulated data.
- Done
- relates to
-
SP-929 Yandasoft Regularised Bandpass Solver - robustness to spectral ripple
- Funnel
-
SP-1489 Determine source model complexity favouring IDGpredict over direct predict
- Funnel
-
SP-672 Identify and provide access to data sets for testing ionospheric calibration
- Done
-
SP-987 Yandasoft Regularised Bandpass Solver - Enhanced smoothing constraints
- Done
-
SP-988 Yandasoft Regularised Bandpass Solver Investigations
- Done
-
SP-1235 Improve direct estimation of screens for DD corrections
- Done
-
SP-1326 Determine initial design for a continuum pipeline and datasets
- Done
-
SP-1328 Developer portal (top level) group for SDP workflow and simulation projects
- Done
-
SP-1411 Enhance the GPU imaging/gain calibration pipeline to support visibility weighting schemes
- Done
-
SP-1449 Heterogenous Beam Simulations: DDFacet
- Done
-
SP-1448 Heterogenous Beam Simulations: WSClean
- Done
- mentioned in
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
- mentioned on