Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-1326

Determine initial design for a continuum pipeline and datasets

    XporterXMLWordPrintable

Details

    • Spike
    • Not Assigned
    • PI9
    • COM SDP SW
    • None
    • Data Processing
    • Hide

      Set the groundwork for developing future SKA1 MID and LOW continuum imaging pipelines.

      Show
      Set the groundwork for developing future SKA1 MID and LOW continuum imaging pipelines.
    • Hide
      1. Organise and/or actively participate in one or more workshops discussing approach towards development roadmap of 'initial SDP workflows', focusing on the development of steps towards (continuum) imaging and calibration pipelines for SKA1 LOW and MID.
      2. Agreed approach (with other SDP teams) for developing initial (continuum) imaging pipelines during the period of PI9 through PI12.
      3. Confluence page(s) documenting inputs to the discussion and outcomes reached.
      Show
      Organise and/or actively participate in one or more workshops discussing approach towards development roadmap of 'initial SDP workflows', focusing on the development of steps towards (continuum) imaging and calibration pipelines for SKA1 LOW and MID. Agreed approach (with other SDP teams) for developing initial (continuum) imaging pipelines during the period of PI9 through PI12. Confluence page(s) documenting inputs to the discussion and outcomes reached.
    • 3
    • 3
    • 26.667
    • Team_NZAPP, Team_SCHAAP, Team_SIM, Team_YANDA
    • Sprint 5
    • Hide

      AC1:
      https://confluence.skatelescope.org/display/SE/DP+ART+PI9+Continuum+Sync+Meetings
      AC2, AC3:
      root page: https://confluence.skatelescope.org/display/SE/Continuum+Imaging+Pipeline+Design+and+Implementation
      Design responsibility is shared across four sub WGs plus program mgmt team;
      S/W repo for artefacts that don't fit existing repos: https://gitlab.com/ska-telescope/sdp/ska-sdp-continuum-imaging-pipelines
      GCS bucket for test data sets and data products: https://console.cloud.google.com/storage/browser/ska1-simulation-data/ska1-low/continuum_sims_SP-1331

      Show
      AC1: https://confluence.skatelescope.org/display/SE/DP+ART+PI9+Continuum+Sync+Meetings AC2, AC3: root page: https://confluence.skatelescope.org/display/SE/Continuum+Imaging+Pipeline+Design+and+Implementation Design responsibility is shared across four sub WGs plus program mgmt team; S/W repo for artefacts that don't fit existing repos: https://gitlab.com/ska-telescope/sdp/ska-sdp-continuum-imaging-pipelines GCS bucket for test data sets and data products: https://console.cloud.google.com/storage/browser/ska1-simulation-data/ska1-low/continuum_sims_SP-1331
    • 10.4
    • Stories Completed, Outcomes Reviewed, NFRS met, Demonstrated, Satisfies Acceptance Criteria, Accepted by FO

    Description

      Size estimate based on 3-4 teams contributing 2-3 people each for 2 days of effort.

      In PI9 we would like to set the groundwork for developing MID and LOW continuum imaging pipelines as a step towards a having an initial set of SDP workflows by the end of PI12 that will be built on and scale in future Feature Sets.

      In order to bootstrap this work, this spike is intended to take the form of a workshop or meeting between SDP teams intending to contribute to the goal of maintaining alignment between MID and LOW solutions. This spike should:

      • Identify the testing strategy
      • Identify pipeline steps (with consideration for state-of-the-art solutions from precursor instruments)
      • Map steps to existing software solutions (including those already identified in work to look at state-of-the ART DD imaging and calibration algorithms) 
      • Consider variations and alternatives
      • Consider the pipeline in the context of the SDP architecture (future integration as an SDP workflow, decomposition into execution engine stages and processing components)
      • Identify steps needed to ensure that the developed pipelines will be a solution that can easily be run and developed further by other SDP teams and where in future PIs we indeed to replacing key steps in the pipeline to achieve scaling at ~10 node level.
      • Identify how pipelines are to be benchmarked for validation of outputs and for compute performance
      • Identify input data (simulation or from precursor/pathfinders) that can be used to test these pipelines

      The outcome of this spike is expected to be one or more confluence pages (possibly in the Solution Intent space) describing the workflows to be implemented.

      Attachments

        Issue Links

          blocks

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-1331 Initial implementation of (continuum) imaging pipeline(s)

          • Not Assigned - Priority is not assigned yet.
          • Done
          followed by

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-1693 Onboard and test complete set continuum imaging pipelines using simulation data (without DD effects)

          • Should have - Items labelled as "Should have" are important but not necessary for delivery in the current delivery timebox. While "Should have" items can be as important as "Must have", they are often not as time-critical or there may be another way to satisfy the requirement so that it can be held back until a future delivery timebox.
          • Done
          relates to

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-929 Yandasoft Regularised Bandpass Solver - robustness to spectral ripple

          • Could have - Items labelled as "Could have" are desirable but not necessary and could improve the user experience or customer satisfaction for a little development cost. These will typically be included if time and resources permit.
          • Funnel

          Spike - A time limited investigation to get some knowledge SP-1329 Exploration of resource availability for SDP simulations and workflows

          • Could have - Items labelled as "Could have" are desirable but not necessary and could improve the user experience or customer satisfaction for a little development cost. These will typically be included if time and resources permit.
          • Funnel

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-987 Yandasoft Regularised Bandpass Solver - Enhanced smoothing constraints

          • Not Assigned - Priority is not assigned yet.
          • Done

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-988 Yandasoft Regularised Bandpass Solver Investigations

          • Not Assigned - Priority is not assigned yet.
          • Done

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-1328 Developer portal (top level) group for SDP workflow and simulation projects

          • Not Assigned - Priority is not assigned yet.
          • Done

          Spike - A time limited investigation to get some knowledge SP-1334 Identify categories and test scenarios for RFI simulation and mitigation

          • Not Assigned - Priority is not assigned yet.
          • Done

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-1335 RFI Simulations for testing future SDP imaging workflows

          • Not Assigned - Priority is not assigned yet.
          • Done

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-1411 Enhance the GPU imaging/gain calibration pipeline to support visibility weighting schemes

          • Not Assigned - Priority is not assigned yet.
          • Done

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-1433 Onboard LEAP and explore Scope for Parallelization

          • Not Assigned - Priority is not assigned yet.
          • Done

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-1447 Demonstrate yandasoft peeling on ASKAP precursor data

          • Not Assigned - Priority is not assigned yet.
          • Done

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-1448 Heterogenous Beam Simulations: WSClean

          • Not Assigned - Priority is not assigned yet.
          • Done

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-1449 Heterogenous Beam Simulations: DDFacet

          • Not Assigned - Priority is not assigned yet.
          • Done

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-1336 Identify and collect data sets with RFI matching criteria

          • Could have - Items labelled as "Could have" are desirable but not necessary and could improve the user experience or customer satisfaction for a little development cost. These will typically be included if time and resources permit.
          • Discarded

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-1450 Heterogenous Beam Simulations: Comparison Report

          • Not Assigned - Priority is not assigned yet.
          • Discarded

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-732 Describe and demonstrate fast Kolmogorov phase screen generation script

          • Not Assigned - Priority is not assigned yet.
          • Done

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-1235 Improve direct estimation of screens for DD corrections

          • Not Assigned - Priority is not assigned yet.
          • Done

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-1331 Initial implementation of (continuum) imaging pipeline(s)

          • Not Assigned - Priority is not assigned yet.
          • Done

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-1448 Heterogenous Beam Simulations: WSClean

          • Not Assigned - Priority is not assigned yet.
          • Done
          mentioned in

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Structure

            Activity

              People

                D.Fenech Fenech, Danielle
                r.brederode Brederode, Ray
                Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 3.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete1133.0
                  Total1133.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel