Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-4229

Establish a simulated-test-data generation pipeline for v0.1 (using Karabo)

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • SRCnet
    • Hide

      Test data is required for v0.1 resembling SKAO data. Synthetically generating this data on demand, will mean sites don't have to store large volumes of data. Increased know-how of simulated data generation will offer a way to exercise compute and storage at a v0.1 site. 

      Karabo has not yet been tested on generating large simulated datasets even though it is capable of doing so. This feature will help find any hiccups in this process, socialise the repo and increase know-how on how to run the pipeline thus allowing for v0.1 suitable data to be generated by a larger group of people in the ART.

      Show
      Test data is required for v0.1 resembling SKAO data. Synthetically generating this data on demand, will mean sites don't have to store large volumes of data. Increased know-how of simulated data generation will offer a way to exercise compute and storage at a v0.1 site.  Karabo has not yet been tested on generating large simulated datasets even though it is capable of doing so. This feature will help find any hiccups in this process, socialise the repo and increase know-how on how to run the pipeline thus allowing for v0.1 suitable data to be generated by a larger group of people in the ART.
    • Hide

      AC1: Test data generation pipeline knowledge documented - so that v0.1 sites can generate data later

      AC2: Test data generated by at least two sites

      AC3: Demo of test data generation

      Stretch: register test data to rucio

       

      Show
      AC1: Test data generation pipeline knowledge documented - so that v0.1 sites can generate data later AC2: Test data generated by at least two sites AC3: Demo of test data generation Stretch: register test data to rucio  
    • 1.5
    • 2
    • 0
    • Team_CHOCOLATE

    • PI23 - UNCOVERED

    • PI23 SRC23-PB SRCNet0.1 Team_Chocolate tests-compilation

    Description

      Simulated test data can be generated using SKA telescope config using the Karabo pipeline. It makes use of OSKAR and RASCIL packages underneath. Example notebook here shows generating visibilities from sources in the GLEAM survey with ASKAP config, and making dirty images of the same. More documentation here: https://i4ds.github.io/Karabo-Pipeline/examples/examples.html.

       

      Need input on what data sizes would be most useful for v0.1. Conversation about resources/where we run it based on this.

      Attachments

        Issue Links

          depends on

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4299 Improvements to Karabo so that simulated v0.1 test data has science metadata attached

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Program Backlog
          relates to

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4228 Define the test campaigns

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Program Backlog

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4132 Identify test data collections for v0.1

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Done

          Structure

            Activity

              People

                r.bolton Bolton, Rosie
                r.joshi Joshi, Rohini
                Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (15.38%)

                  Feature Estimate: 1.5

                  IssuesStory Points
                  To Do611.0
                  In Progress   00.0
                  Complete12.0
                  Total713.0

                  Dates

                    Created:
                    Updated:

                    Structure Helper Panel