Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-3837

Prototype tool for creating links to POSIX DDM files in User Storage (2)

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • SRCnet
    • Hide

      User software in science platform containers that can only read data from a file system obtains read-only random access to a data file in DDM storage without staging (copying the whole file ahead of time). Permissions enforced.

      Suggest rewording:
      Science users will benefit from being able to link to existing replicas of data products at a particular SRC site rather than requiring to download data products from the data lake into user storage (this is currently the case for Storage Inventory and the Cavern user storage). Whilst this is functional, it means that unnecessary replicas are made of data products, which would be undesirable for large and popular data products. If we can instead implement a way that user tools (e.g., but not limited to CANFAR science platform) handle links to these files whilst retaining correct permissions settings we can avoid duplication of data products into user storage.

      Show
      User software in science platform containers that can only read data from a file system obtains read-only random access to a data file in DDM storage without staging (copying the whole file ahead of time). Permissions enforced. Suggest rewording: Science users will benefit from being able to link to existing replicas of data products at a particular SRC site rather than requiring to download data products from the data lake into user storage (this is currently the case for Storage Inventory and the Cavern user storage). Whilst this is functional, it means that unnecessary replicas are made of data products, which would be undesirable for large and popular data products. If we can instead implement a way that user tools (e.g., but not limited to CANFAR science platform) handle links to these files whilst retaining correct permissions settings we can avoid duplication of data products into user storage.
    • Hide

      AC1: Publish link creation tool so it can be installed in CANFAR science platform images.
      AC2: Demonstrate a user creating a link and then accessing a DDM file in the science platform (e.g. visualize a cube with CARTA).

      Show
      AC1: Publish link creation tool so it can be installed in CANFAR science platform images. AC2: Demonstrate a user creating a link and then accessing a DDM file in the science platform (e.g. visualize a cube with CARTA).
    • 2
    • 2
    • 0
    • PI23 - UNCOVERED

    • science-platform-services

    Description

      This is the unfinished work moved from SP-3719

      Original description:

      User processing in a science platform requires access to data files in the data management (archive). Traditionally, users had to stage (copy) the data from archival storage into storage attached to processing resources to run software on it.

      It is feasible to create a tool to allow users to create a link in a science container to a data file in archival storage and access it directly out of storage. Permission checking happens at link creation time. No bytes are copied and user software has read-only random access to the file.

      architectural note: there will be a future prototype feature to enable link to data in object store

      Attachments

        Issue Links

          Structure

            Activity

              People

                r.bolton Bolton, Rosie
                s.goliath sharon goliath
                Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (0%)

                  Feature Estimate: 2.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete00.0
                  Total00.0

                  Dates

                    Created:
                    Updated:

                    Structure Helper Panel