Details
-
Feature
-
Not Assigned
-
SRCnet
-
-
-
Intra Program
-
2
-
2
-
0
-
-
-
-
example-workflows-and-benchmarks tests-compilation
Description
Continue work shown in PI21 to extend to more example workflows so that we have notebook (astro query) example using data from SRCNet location or external archive source rather than local copy.
In PI21, the image-coadding task (https://gitlab.com/ska-telescope/src/src-workloads/-/tree/master/tasks/image-coadding-swarp) was modified. The source data was added into the Rucio data lake, with metadata, and the Dockerfiles modified so that a) the SRCNet astroquery module was included, and b) a user could pass an access token in which allow astroquery to download the data from Rucio (with the fallback - if no token is passed - being to use the SDSS SAS). Additionally, a Jupyter notebook was added as an example in the repo to provide examples of how to discover and access the data using the Astroquery module.
For this feature, remaining tasks should, where possible be modified similarly. This will involve:
- Adding tasks' requisite data into the src-workloads scope of the Rucio data lake, with metadata set.
- Modifying Dockerfile's to allow an optional DM_ACCESS_TOKEN env var to be set
- If set, modify download scripts to retrieve data from Rucio
- Provide an example notebook which demonstrates how to discover/download data, and then run the task.
Attachments
Issue Links
- mentioned in
-
Page Loading...