Loading...

Change Owns to Parent Ofs

Set start and due date...

Xporter

XML

Word

Printable

Details

Type: Feature
Priority: Not Assigned
Fix Version/s: PI22
Component/s: None
Labels:
- science-platform-services

ARTs:

National SRC
Benefit hypothesis:

Hide

Though this work needs to be done in some form, there is a potential to leverage this for the benefit of the SRCNet development, by using the data lake to manage the SDC2 dataset and providing read access without triggering downloads.

Show
Though this work needs to be done in some form, there is a potential to leverage this for the benefit of the SRCNet development, by using the data lake to manage the SDC2 dataset and providing read access without triggering downloads.
Acceptance criteria:

Hide

AC1: SDC2 dataset is attached to user containers started in JupyterHub instance

AC2 (stretch): SDC2 dataset is managed by Rucio with science metadata set. RSE storage is attached to user sessions in JupyterHub. DataLink service can be used to determine local path to data for read access.

Show
AC1: SDC2 dataset is attached to user containers started in JupyterHub instance AC2 (stretch): SDC2 dataset is managed by Rucio with science metadata set. RSE storage is attached to user sessions in JupyterHub. DataLink service can be used to determine local path to data for read access.
Feature Points:
0.5
Initial Size:
1.5
WSJF:
0
Agile Teams:

Team_MAGENTA
Due Sprint:
Sprint 2
Story Point Burn-up:
Overdue:
Outcomes:
Hide

Planned work has been done; can be tested at https://jupyterhub.srcdev.skao.int

Summary of work done:

A new volume provisioned on STFC Cloud, SDC2 dataset downloaded onto it

Volume NFS-shared to each of the workers of the K8s workload cluster running the JupyterHub prototype

K8s resources (PV/PVCs) created to make this hostpath visible to the Jupyter user containers

This meets AC1, but not AC2 - this wasn't considered a high priority at backlog prioritisation, so was left for the future.
Show
Planned work has been done; can be tested at https://jupyterhub.srcdev.skao.int Summary of work done: A new volume provisioned on STFC Cloud, SDC2 dataset downloaded onto it Volume NFS-shared to each of the workers of the K8s workload cluster running the JupyterHub prototype K8s resources (PV/PVCs) created to make this hostpath visible to the Jupyter user containers This meets AC1, but not AC2 - this wasn't considered a high priority at backlog prioritisation, so was left for the future.

Requirement Status:

PI23 - UNCOVERED
Labels_MIRO:
science-platform-services

Description

Previously we have presented a customised JupyterHub (with Binder and Dask Gateway services) to the SKAO science team as a means of presenting a common user experience to the SDC participants. They have asked for the SDC2 dataset (a single FITS cube of 1TB) to be attached for further testing/evaluation.

The minimal version of this would be to download the SDC2 dataset to a volume and attach this to all nodes of the cluster on which the JupyterHub is running. Then a PV/PVC can be made and attached as read-only to user containers. (1 FP)

A more sophisticated version of this, perhaps yielding more value to the SRCNet development, would be to store the file in Rucio, and enable data access via RSE storage mounts, as previously demonstrated. It may also be possible to use (manually created) symlinks to provide user access while preserving security for the rest of the RSE data. This could equally then be demonstrated in CANFAR SP as well. (2FP)

Attachments

Issue Links

mentioned in: Page Loading...; Page Loading...

Structure

Activity

People

Assignee:: Bolton, Rosie

Reporter:: Collinson, James

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Feature Progress

Story Point Burn-up: (100.00%)

Feature Estimate: 0.5

	Issues	Story Points
To Do	0	0.0
In Progress	0	0.0
Complete	3	3.0
Total	3	3.0

Dates

Created:: 20/Feb/24 6:20 PM

Updated:: 18/Apr/24 2:45 PM

SDC2 dataset is available to interactive environments in JupyterHub/CANFAR SciPlat for SKAO science team evaluation