Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-1547

SDP Data delivery

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • Enabler
    • Could have
    • PI14
    • COM SDP SW
    • None
    • Data Processing
    • Hide

      To support the commissioning test with AA0.5, we need a solution that allows an operator or AIV engineer to integrate and access data products captured by SDP.

      While a very simple solution would be acceptable as a fallback for AA0.5, working with the SRC prototyping early presents a valuable opportunity to derisk technology solutions early and provide a better user experience than one might otherwise provide if this is left until 1-2 PIs before AA0.5 data is being captured.

      Show
      To support the commissioning test with AA0.5, we need a solution that allows an operator or AIV engineer to integrate and access data products captured by SDP. While a very simple solution would be acceptable as a fallback for AA0.5, working with the SRC prototyping early presents a valuable opportunity to derisk technology solutions early and provide a better user experience than one might otherwise provide if this is left until 1-2 PIs before AA0.5 data is being captured.
    • Hide

      To be discussed with team during planning:

      • Rucio-based example transfer of data (with validation?) in collaboration with SP-2446
      • Report on how example data is discovered, accessed and managed
      • Report on the investigation of data search capability
      Show
      To be discussed with team during planning: Rucio-based example transfer of data (with validation?) in collaboration with SP-2446 Report on how example data is discovered, accessed and managed Report on the investigation of data search capability
    • 5
    • 5
    • 2
    • 0.4
    • Team_NALEDI
    • Sprint 4
    • Hide
      • Rucio-based example transfer of data (with validation?) in collaboration with SP-2446

      Presentation was done during SDP system demo 14.5

      https://confluence.skatelescope.org/display/SE/2022-05-18+DP+ART+System+Demo+14.5

      slides:

      https://docs.google.com/presentation/d/1QaeLm55DvgGyg3YPvcsRJSZCT-bVj7U7/edit?usp=drivesdk&ouid=105994321144790377943&rtpof=true&sd=true

      • Report on how example data is discovered, accessed and managed

      https://confluence.skatelescope.org/display/SE/Rucio+data+discovery%2C+access+and+management

      • Report on the investigation of data search capability

      https://confluence.skatelescope.org/display/SE/Rucio+data+discovery%2C+access+and+management

      Show
      Rucio-based example transfer of data (with validation?) in collaboration with SP-2446 Presentation was done during SDP system demo 14.5 https://confluence.skatelescope.org/display/SE/2022-05-18+DP+ART+System+Demo+14.5 slides: https://docs.google.com/presentation/d/1QaeLm55DvgGyg3YPvcsRJSZCT-bVj7U7/edit?usp=drivesdk&ouid=105994321144790377943&rtpof=true&sd=true Report on how example data is discovered, accessed and managed https://confluence.skatelescope.org/display/SE/Rucio+data+discovery%2C+access+and+management Report on the investigation of data search capability https://confluence.skatelescope.org/display/SE/Rucio+data+discovery%2C+access+and+management
    • 16.6
    • Stories Completed, Outcomes Reviewed, NFRS met, Demonstrated, Satisfies Acceptance Criteria, Accepted by FO

    • PI22 - UNCOVERED

    • SPO-1591

    Description

      We will test a delivery prototype based on Rucio

      Who?

      AA Operators: AIV Engineers, Commissioning Scientists

      Also: SDP developers, SKA Software Integrators

      What?

      A test SDP to SRC delivery of a modest number of AA0.5-size data sets using Rucio. This would aim to prototype a Rucio-based solution to understand how data could be stored, accessed and transferred along with suitable metadata for AA0.5.

      Why?

      Whilst data delivery requirements are relatively modest for AA0.5, there is still a need to enable management, interrogation and potential transfer off-site of the data products and relevant metadata from commissioning tests. This could be of the scale of hundreds of files. Building on recent work to test Rucio as a potential data-delivery solution this presents an opportunity to de-risk a potential long-term solution and improve early user-access.

      -------

      Notes:

      This should build on work from previous PIs see e.g. SP-1701 and SP-2220 which established Rucio instances and tested authentication and data replication.

       

      Old description before the ticket was pivoted (for reference)

      • The SDP deployment is modified such that it allocates a configurable amount of storage, which can be mounted into all workflows.
      • It should be possible to configure it such that its lifetime is not limited to the SDP deployment (i.e. persistent)
      • A directory structure convention is established and documented that keeps different kinds of data (see PB dependency type?) from different EBs/PBs separate
      • Receive and batch workflows are modified such that they mount the storage and they read and write files as specified
      • Stretch: It is demonstrated how we can synchronise the contents of this storage off-site (either as a job/service or part of the workflows)
      • Stretch: (Automated?) service / job to clean up old files
      • Stretch: Provide ska-sdp commands for listing and downloading (/synchronising/cleaning up?) objects in storage
      • Stretch: Tango interface to monitor storage fill status / initiate clean-up

      Attachments

        Issue Links

          clones

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-1211 Explore use of Rucio for SDP Delivery

          • Could have - Items labelled as "Could have" are desirable but not necessary and could improve the user experience or customer satisfaction for a little development cost. These will typically be included if time and resources permit.
          • Discarded

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-1338 Define and create (very minimal) PoC for system-wide data management

          • Not Assigned - Priority is not assigned yet.
          • Discarded
          relates to

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-2446 Support replication of a prototype SDP->SRC data transfer with Rucio

          • Should have - Items labelled as "Should have" are important but not necessary for delivery in the current delivery timebox. While "Should have" items can be as important as "Must have", they are often not as time-critical or there may be another way to satisfy the requirement so that it can be held back until a future delivery timebox.
          • Done

          Architectural Decision - Software Architectural Decision ADR-55 Definition of metadata for data management at AA0.5

          • decided
          mentioned in

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Structure

            Activity

              People

                D.Fenech Fenech, Danielle
                b.mort Mort, Ben
                Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 5.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete1232.0
                  Total1232.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel