DM integration with compute with a Dask workflow discovering Rucio data with IVOA metadata (Dask + Rucio + IVOA)

Previously, Jupyterhub integration with Rucio data has been demonstrated with data discovered via the Rucio plugin (based on DIDs) and download it to a locally mounted Rucio storage element. This feature would explore other aspects of running user workflows on Jupyterhub, namely with IVOA discovered data and processing that data via Dask data frames utilising auth tokens without downloading it all locally. Uploading processed data back to the DL has not been demonstrated in the past and this is essential to complete the data lifecycle loop.

Demo showing: In a JupyterHub notebook, List Rucio DIDs based on IVOA metadata, get Rucio PFNs from Datalink service, have Dask workers point to these storage locations with a valid access token , 'process' the data to generate new data with Dask, upload new data with new metadata to the DL.

Workflow temporarily placed at: https://confluence.skatelescope.org/display/SRCSC/SP-3521+DM+integration+with+compute+with+a+Dask+workflow

Demoed in: https://confluence.skatelescope.org/display/SRCSC/2023-08-17+SRC+ART+System+Demo+19.5+Part+2

Pre-recorded demo link: https://drive.google.com/file/d/1ZL0TuxLmLXBbPBZ4gJqRKONEoTfgmiIR/view?usp=drive_link