Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-4229

Establish a simulated-test-data generation pipeline for v0.1 (using Karabo)

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • SRCnet
    • Hide

      Test data is required for v0.1 resembling SKAO data. Synthetically generating this data on demand, will mean sites don't have to store large volumes of data. Increased know-how of simulated data generation will offer a way to exercise compute and storage at a v0.1 site. 

      Karabo has not yet been tested on generating large simulated datasets even though it is capable of doing so. This feature will help find any hiccups in this process, socialise the repo and increase know-how on how to run the pipeline thus allowing for v0.1 suitable data to be generated by a larger group of people in the ART.

      Show
      Test data is required for v0.1 resembling SKAO data. Synthetically generating this data on demand, will mean sites don't have to store large volumes of data. Increased know-how of simulated data generation will offer a way to exercise compute and storage at a v0.1 site.  Karabo has not yet been tested on generating large simulated datasets even though it is capable of doing so. This feature will help find any hiccups in this process, socialise the repo and increase know-how on how to run the pipeline thus allowing for v0.1 suitable data to be generated by a larger group of people in the ART.
    • Hide

      AC1: Test data generation pipeline knowledge documented - so that v0.1 sites can generate data later

      AC2: Test data generated by at least two sites

      AC3: Demo of test data generation

      Stretch: register test data to rucio

       

      Show
      AC1: Test data generation pipeline knowledge documented - so that v0.1 sites can generate data later AC2: Test data generated by at least two sites AC3: Demo of test data generation Stretch: register test data to rucio  
    • 1.5
    • 2
    • 0
    • Team_CHOCOLATE
    • Hide

      Demo for the ART how to run Karabo

      - https://confluence.skatelescope.org/display/SRCSC/%5BCHOC-20%5D+Prep+and+deliver+Demo+to+the+ART+for+how+to+run+Karabo

      Running Karabo at SPSRC

      Show
      Demo for the ART how to run Karabo - https://confluence.skatelescope.org/display/SRCSC/%5BCHOC-20%5D+Prep+and+deliver+Demo+to+the+ART+for+how+to+run+Karabo Running Karabo at SPSRC https://confluence.skatelescope.org/display/SRCSC/COR-663+%5BKarabo%5D+Installation+and+execution+of+Karabo+in+the+espSRC
    • 23.6
    • Stories Completed, Demonstrated, Satisfies Acceptance Criteria, Accepted by FO

    • PI24 - UNCOVERED

    • PI23 SRC23-PB SRCNet0.1 Team_Chocolate tests-compilation

    Description

      Simulated test data can be generated using SKA telescope config using the Karabo pipeline. It makes use of OSKAR and RASCIL packages underneath. Example notebook here shows generating visibilities from sources in the GLEAM survey with ASKAP config, and making dirty images of the same. More documentation here: https://i4ds.github.io/Karabo-Pipeline/examples/examples.html.

       

      Need input on what data sizes would be most useful for v0.1. Conversation about resources/where we run it based on this.

      Attachments

        Issue Links

          Child Of

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. SP-4870 Science Enabling v0.1 - Roadmap

          • Not Assigned - Priority is not assigned yet.
          • Implementing
          depends on

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4299 Improvements to Karabo so that simulated v0.1 test data has science metadata attached

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Done
          followed by

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4565 v0.1 Test Campaign preparation: Ingest image data into Rucio

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Implementing
          relates to

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4228 Define the test campaigns

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Program Backlog

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4132 Identify test data collections for v0.1

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Done
          mentioned in

          Page Loading...

          Page Loading...

          Structure

            Activity

              People

                S.SanchezExposito Sanchez Exposito, Susana
                r.joshi Joshi, Rohini
                Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (84.62%)

                  Feature Estimate: 1.5

                  IssuesStory Points
                  To Do13.0
                  In Progress   00.0
                  Complete1016.5
                  Total1119.5

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel