Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-4132

Identify test data collections for v0.1

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • SRCnet
    • Hide

      For SRCNet0.1 we need to identify scientifically and technically-motivated data sets that can help us test the 0.1 system. We already have a wealth of possible data sets identified as part of the mini-SRCNet demonstrator, within the current Rucio instance, and relating to the existing example workflows. Let's make a sized plan for data that we'd like to ingest into the DDM system of the SRCNet0.1 sites

      BH: By identifying of data to be ingested into v0.1 and ingestion, we could allow a good testing/benchmarking of the SRCNet components for analysis, scalability and computing/pipelines integration.

      Show
      For SRCNet0.1 we need to identify scientifically and technically-motivated data sets that can help us test the 0.1 system. We already have a wealth of possible data sets identified as part of the mini-SRCNet demonstrator, within the current Rucio instance, and relating to the existing example workflows. Let's make a sized plan for data that we'd like to ingest into the DDM system of the SRCNet0.1 sites BH: By identifying of data to be ingested into v0.1 and ingestion, we could allow a good testing/benchmarking of the SRCNet components for analysis, scalability and computing/pipelines integration.
    • Hide

      AC: List (on confluence) of data available now and in the following months, including input data location, description, contact points (who), scale (volume), etc

      Show
      AC: List (on confluence) of data available now and in the following months, including input data location, description, contact points (who), scale (volume), etc
    • 0.5
    • 0.5
    • 0
    • Team_MAGENTA
    • Sprint 5
    • Hide

      Initial data sets identified at: https://confluence.skatelescope.org/display/SRCSC/SP-4132+Identify+test+data+collections+for+SRCNet+0.1 

      Show
      Initial data sets identified at: https://confluence.skatelescope.org/display/SRCSC/SP-4132+Identify+test+data+collections+for+SRCNet+0.1  
    • 22.6
    • Stories Completed, Satisfies Acceptance Criteria, Accepted by FO

    • PI23 - UNCOVERED

    • data-ingestion-dissemination-and-replication tests-compilation

    Description

      Discussion on possible data that could be ingested/registered for v0.1. That could include:

      • Precursors data: Ingesting precursor data enables the validation and refinement of data processing pipelines, ensuring their readiness for handling real SKA data and the execution of complete science use cases
      • SKA test data: Synthetic datasets generated by SDP (Science Data Processor) pipelines, mirroring the scale and format expected from real SKA observations. By simulating SKA data, the network can assess the efficacy of data management protocols and software solutions under conditions close to actual operations
      • Simulations data: These data facilitate the evaluation of computational models, calibration techniques, and data analysis workflows within the SRCNet infrastructure
      • Science Data challenges data or similar data: Data already analysed in a locally controlled scenario using scientific workflows will provide information on the overhead provided by the SRCNet distributed data and resources

      For each data type, the following aspects need to be documented:

      • Availability: Source or generation mechanism of the data.
      • Contact Point: Responsible individual or entity for data inquiries and coordination.
      • Size/Type: Magnitude and format specifications of the dataset.
      • Science Use Cases/Software: Specific scientific objectives and requisite software tools for analysis.

      These test datasets will be instrumental in evaluating the robustness, scalability, and performance of the SRCNet infrastructure across various operational scenarios, including data management, scalability, stress tests, and more.

      See https://docs.google.com/document/d/1PZ4Il_RgIs2rtR0XawoAa0Q3FXycI4-4yhmjbbrDzLw/ for further details

       

      Attachments

        Issue Links

          clones

          Spike - A time limited investigation to get some knowledge SP-4134 Identify test data sets for SRCNet0.1

          • Not Assigned - Priority is not assigned yet.
          • Discarded
          relates to

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4229 Establish a simulated-test-data generation pipeline for v0.1 (using Karabo)

          • Should have - Items labelled as "Should have" are important but not necessary for delivery in the current delivery timebox. While "Should have" items can be as important as "Must have", they are often not as time-critical or there may be another way to satisfy the requirement so that it can be held back until a future delivery timebox.
          • Implementing
          mentioned in

          Page Loading...

          Page Loading...

          Page Loading...

          Structure

            Activity

              People

                r.bolton Bolton, Rosie
                Jesus.Salgado Salgado, Jesus
                Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 0.5

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete34.0
                  Total34.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel