Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-4565

v0.1 Test Campaign preparation: Ingest image data into Rucio

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • SRCnet
    • Hide

      SRCNet v0.1 Test Data ingestion as mentioned in  Implementation plan under https://docs.google.com/document/d/11DnUf52hDejtUr76DCaCss6OP9K0IEzxer-tUxOkGyM/edit#heading=h.abg0jc8hdd4r 

      This feature is to generate imaging datasets such that the integration test campaigns have data in the datalake to operate on ie to visualise with CARTA/perform cutouts with SODA etc. Generating simulated datasets takes a non-trivial amount of compute hours, thus the preparation of the test campaigns starting with data generation and ingestion is timely.

      Show
      SRCNet v0.1 Test Data ingestion as mentioned in  Implementation plan under https://docs.google.com/document/d/11DnUf52hDejtUr76DCaCss6OP9K0IEzxer-tUxOkGyM/edit#heading=h.abg0jc8hdd4r   This feature is to generate imaging datasets such that the integration test campaigns have data in the datalake to operate on ie to visualise with CARTA/perform cutouts with SODA etc. Generating simulated datasets takes a non-trivial amount of compute hours, thus the preparation of the test campaigns starting with data generation and ingestion is timely.
    • Hide

      AC1: Generate SKA representative image datasets with Karabo (scripts), ingest to Rucio with ObsCore metadata

      • AC1.1: SKA-representative and meaningful datasets by adding the necessary AA layouts 
      • AC1.2: Generating the imaging data at scale (with visibilities along the way), require addressing this scaling issue.

      AC2: Example notebook that shows discovery via a TAP client or the DM API

      Show
      AC1: Generate SKA representative image datasets with Karabo (scripts), ingest to Rucio with ObsCore metadata AC1.1: SKA-representative and meaningful datasets by adding the necessary AA layouts  AC1.2: Generating the imaging data at scale (with visibilities along the way), require addressing this scaling issue . AC2: Example notebook that shows discovery via a TAP client or the DM API
    • 2.5
    • 2.5
    • 0
    • Team_CHOCOLATE
    • Sprint 4

    • PI24 - UNCOVERED

    • PI24-PB SRCNet0.1 Team_Chocolate
    • SPO-3479

    Description

      Continuation of https://jira.skatelescope.org/browse/SP-4229 and https://jira.skatelescope.org/browse/SP-4299 and prep for https://jira.skatelescope.org/browse/SP-4228

       

      Create larger simulated imaging datasets from Karabo with metadata

      • Liaise with domain experts to determine necessary AA layouts and corresponding parameters to use for simulation
      • Access ingested data with TAP client
      • Liaise with arch core team to determine dataset sizes to generate
      • Address scaling issue found in PI23 work  https://github.com/i4Ds/Karabo-Pipeline/issues/595 

      Note resources risk at CSCS.

      Attachments

        Issue Links

          Child Of

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. SP-4871 Data Management v0.1 - Roadmap

          • Not Assigned - Priority is not assigned yet.
          • Implementing
          informs

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4228 Define the test campaigns

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Program Backlog
          preceded by

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4299 Improvements to Karabo so that simulated v0.1 test data has science metadata attached

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Done

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4229 Establish a simulated-test-data generation pipeline for v0.1 (using Karabo)

          • Should have - Items labelled as "Should have" are important but not necessary for delivery in the current delivery timebox. While "Should have" items can be as important as "Must have", they are often not as time-critical or there may be another way to satisfy the requirement so that it can be held back until a future delivery timebox.
          • Done
          mentioned in

          Page Loading...

          Structure

            Activity

              People

                Nichol.Cunningham Cunningham, Nichol
                r.joshi Joshi, Rohini
                Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (53.66%)

                  Feature Estimate: 2.5

                  IssuesStory Points
                  To Do34.5
                  In Progress   35.0
                  Complete611.0
                  Total1220.5

                  Dates

                    Created:
                    Updated:

                    Structure Helper Panel