Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-4870

Science Enabling v0.1 - Roadmap

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • Science Enabling v0.1 - Roadmap
    • SRCnet
    • 0

    Description

      Key Features and Objectives

      Storage Benchmarking:

      • Run tests on at least one SRC (Science Regional Center) to gather metrics and statistics for different storage types. This builds on prior benchmark development and documentation.

      Data Processing Pipeline Development:

      • Collaborate with CanSRC (Red team) to establish a data processing pipeline utilizing Rucio datalake and CADC CANFAR for distributed data transfer and processing.
      • Key Acceptance Criteria (AC):
      • Upload astronomy data products to Rucio datalake.
      • Upload custom-built images to CANFAR for batch processing.
      • Create a multi-stage data processing workflow.
      • Provide documentation for reproducibility.

      Integration of Tools and Feedback:

      • Work with OSKAR and RASCIL developers to improve API alignment, focusing on user needs for processing astronomical data.
      • Address issues related to noise configuration in simulated data to improve data processing accuracy.

      Software Distribution System:

      • Implement a software distribution system as part of the Science Enabling product, incorporating user feedback and common use cases to develop a preliminary list of containers and standard environments.

      Demonstrator Workloads:

      • Utilize UKSRC demonstrator cases with pathfinder and precursor data, allowing these to serve as test workloads for SRCNet.

      SODA Service Extensions:

      • Extend Orange’s SODA service with new operations for data cubes, ensuring client compatibility with updated functionalities.

      Visualization Integration:

      • Integrate AladinLite into the SRCNet Science Gateway for enhanced data visualization and analysis, enabling users to overlay SKA datasets on public HiPS (Hierarchical Progressive Surveys).

      Kubernetes Job Scheduling:

      • Focus on low-level job scheduling within Kubernetes, implementing prioritized queuing for interactive versus batch jobs, potentially utilizing KUEUE for efficient resource management.

      Workflow Performance Tracking:

      • Develop the PyProfQueue package to track hardware usage during workflows, validating its performance against SRCNet workloads and providing insights into resource needs.

      Documentation and Deployment Procedures:

      • Create documentation for the deployment of essential services in SRCNet v0.1, emphasizing GitOps practices and error handling.

      Simulated Data Generation:

      • Generate test data using the Karabo pipeline, seeking input on data sizes useful for initial implementation.

      End-to-End Data Lifecycle Tests:

      • Conduct tests covering data ingestion, replication, processing, and new data uploads, documenting results for future campaigns.

      Cosmic Magnetism Survey:

      • Focus on all-sky spectro-polarimetric surveys using RM synthesis with related telescopes, leveraging public data until proprietary pipelines are available.

      Software Distribution Technologies:

      • Refine recommendations for software distribution in SRCNet based on previous investigations and user expectations.

      HiPS Service Interaction:

      • Investigate integration of HiPS services through VisIVO/CARTA for hierarchical tiling of sky regions, facilitating a progressive survey view.

      CI/CD Pipeline Implementation:

      • Establish a robust CI/CD pipeline for service deployment in Kubernetes, incorporating GitOps principles for automated updates.

      3D Visualization Service Development:

      • Define capabilities for a 3D visualization service based on IVOA and X3D standards.

      Architecture Document Support:

      ••Provide architectural support in preparation for external reviews, ensuring comprehensive coverage of key areas like data discovery and access.

      Attachments

        Issue Links

          Child Of

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. SP-4748 Science Enabling All Versions - Roadmap

          • Not Assigned - Priority is not assigned yet.
          • Funnel

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. SP-4873 Development VS v0.1 - Roadmap

          • Not Assigned - Priority is not assigned yet.
          • Funnel
          Parent Of

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4553 Start an investigation of end-user software distribution requirements:

          • Not Assigned - Priority is not assigned yet.
          • Funnel

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4228 Define the test campaigns

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Program Backlog

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4406 Add support for new SODA functionalities into visualization tools

          • Could have - Items labelled as "Could have" are desirable but not necessary and could improve the user experience or customer satisfaction for a little development cost. These will typically be included if time and resources permit.
          • Program Backlog

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4640 Demonstrate job submission via the orchestration layer

          • Could have - Items labelled as "Could have" are desirable but not necessary and could improve the user experience or customer satisfaction for a little development cost. These will typically be included if time and resources permit.
          • Program Backlog

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4680 Measuring storage performance through Storage Benchmarks for SRCNet v.0.1

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Implementing

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4566 Improve Karabo usability and applicability for SRCNet Test Data

          • Should have - Items labelled as "Should have" are important but not necessary for delivery in the current delivery timebox. While "Should have" items can be as important as "Must have", they are often not as time-critical or there may be another way to satisfy the requirement so that it can be held back until a future delivery timebox.
          • Implementing

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4328 Explore possibilities for HiPS cube visualisation inside Aladin Lite

          • Could have - Items labelled as "Could have" are desirable but not necessary and could improve the user experience or customer satisfaction for a little development cost. These will typically be included if time and resources permit.
          • Implementing

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4515 Sharing UKSRC demonstrator cases

          • Could have - Items labelled as "Could have" are desirable but not necessary and could improve the user experience or customer satisfaction for a little development cost. These will typically be included if time and resources permit.
          • Implementing

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4516 Science software distribution roadmap for during v0.1

          • Could have - Items labelled as "Could have" are desirable but not necessary and could improve the user experience or customer satisfaction for a little development cost. These will typically be included if time and resources permit.
          • Implementing

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4552 Finish reference implementation of science software distribution system

          • Could have - Items labelled as "Could have" are desirable but not necessary and could improve the user experience or customer satisfaction for a little development cost. These will typically be included if time and resources permit.
          • Implementing

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4605 Development of a scalable, multi-container processing pipeline in CADC CANFAR

          • Could have - Items labelled as "Could have" are desirable but not necessary and could improve the user experience or customer satisfaction for a little development cost. These will typically be included if time and resources permit.
          • Implementing

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-3319 BL: Review Architecture Document and Top-Level Roadmap

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Done

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-3323 Orange: Review Architecture Document and Top-Level Roadmap

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Done

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4077 Initialisation of Minimal Viable Set of SRCNet Policies for v0.1

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Done

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4082 perfSONAR network monitoring between multiple SRC sites

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Done

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4119 ALADIN / HiPS integration into Gateway, taking obscore FoV

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Done

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4249 Documentation of SRCNet v0.1 compulsory services deployment

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Done

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4266 Enable Job queueing for CANFAR Science Platform

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Done

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-3929 3D Visualisation service based on IVOA standards

          • Should have - Items labelled as "Should have" are important but not necessary for delivery in the current delivery timebox. While "Should have" items can be as important as "Must have", they are often not as time-critical or there may be another way to satisfy the requirement so that it can be held back until a future delivery timebox.
          • Done

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-3957 GitOps CI/CD workflow to improve the deployment of new services versions

          • Should have - Items labelled as "Should have" are important but not necessary for delivery in the current delivery timebox. While "Should have" items can be as important as "Must have", they are often not as time-critical or there may be another way to satisfy the requirement so that it can be held back until a future delivery timebox.
          • Done

          Spike - A time limited investigation to get some knowledge SP-4063 Document SDP node assumptions to help shape SRC node definitions

          • Should have - Items labelled as "Should have" are important but not necessary for delivery in the current delivery timebox. While "Should have" items can be as important as "Must have", they are often not as time-critical or there may be another way to satisfy the requirement so that it can be held back until a future delivery timebox.
          • Done

          Spike - A time limited investigation to get some knowledge SP-4084 Enabling Kubernetes Workloads on Traditional HPC Resources: Challenges and Solutions

          • Should have - Items labelled as "Should have" are important but not necessary for delivery in the current delivery timebox. While "Should have" items can be as important as "Must have", they are often not as time-critical or there may be another way to satisfy the requirement so that it can be held back until a future delivery timebox.
          • Done

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4118 Verify the deployment of containerised visualization tools within CANFAR Science Platform

          • Should have - Items labelled as "Should have" are important but not necessary for delivery in the current delivery timebox. While "Should have" items can be as important as "Must have", they are often not as time-critical or there may be another way to satisfy the requirement so that it can be held back until a future delivery timebox.
          • Done

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4229 Establish a simulated-test-data generation pipeline for v0.1 (using Karabo)

          • Should have - Items labelled as "Should have" are important but not necessary for delivery in the current delivery timebox. While "Should have" items can be as important as "Must have", they are often not as time-critical or there may be another way to satisfy the requirement so that it can be held back until a future delivery timebox.
          • Done

          Spike - A time limited investigation to get some knowledge SP-4121 Potentially Visualize HiPS surveys with VisIVO/CARTA

          • Could have - Items labelled as "Could have" are desirable but not necessary and could improve the user experience or customer satisfaction for a little development cost. These will typically be included if time and resources permit.
          • Done

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4193 Propose CI/CD/CM recommendations for software distribution

          • Could have - Items labelled as "Could have" are desirable but not necessary and could improve the user experience or customer satisfaction for a little development cost. These will typically be included if time and resources permit.
          • Done

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4204 Perform RM synthesis on an image cube

          • Could have - Items labelled as "Could have" are desirable but not necessary and could improve the user experience or customer satisfaction for a little development cost. These will typically be included if time and resources permit.
          • Done

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4250 Adapt UKSRC profiling tool for quantitative and analytical feedback

          • Could have - Items labelled as "Could have" are desirable but not necessary and could improve the user experience or customer satisfaction for a little development cost. These will typically be included if time and resources permit.
          • Done
          mentioned in

          Page Loading...

          Structure

            Activity

              People

                Debashis.Mitra Mitra, Debashis
                Debashis.Mitra Mitra, Debashis
                Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                  Created:
                  Updated:

                  Structure Helper Panel