Loading...

Change Owns to Parent Ofs

Set start and due date...

Xporter

XML

Word

Printable

Details

Type: Feature
Priority: Could have
Fix Version/s: PI18
Component/s: None
Labels:

ARTs:

SRCnet
Benefit hypothesis:

Hide

Dask Gateway deployment has been demoed, enabling users to create Dask clusters on demand. Dask is widely used in a range of data science fields, but in order to test this functionally, we would like to have some basic astronomy workflows that we can demonstrate running on such an ephemeral cluster. The resulting demo will be a useful resource to show astronomers unfamiliar with Dask, and the workflow (likely a Jupyter Notebook) will be a good starting point for functional testing deployments, as well as providing a link to the use case work ongoing in WG6.

Show
Dask Gateway deployment has been demoed, enabling users to create Dask clusters on demand. Dask is widely used in a range of data science fields, but in order to test this functionally, we would like to have some basic astronomy workflows that we can demonstrate running on such an ephemeral cluster. The resulting demo will be a useful resource to show astronomers unfamiliar with Dask, and the workflow (likely a Jupyter Notebook) will be a good starting point for functional testing deployments, as well as providing a link to the use case work ongoing in WG6.
Acceptance criteria:

Hide

AC1: A demo and a workflow (ideally a Jupyter Notebook or Python script runnable on a JupyterHub platform) of an astronomical analysis which benefits from Dask acceleration running on an ephemeral Dask cluster.

Show
AC1: A demo and a workflow (ideally a Jupyter Notebook or Python script runnable on a JupyterHub platform) of an astronomical analysis which benefits from Dask acceleration running on an ephemeral Dask cluster.
Feature Points:
1.5
Initial Size:
1.5
WSJF:
0
Agile Teams:

Team_CORAL
Due Sprint:
Sprint 5
Story Point Burn-up:
Overdue:
Outcomes:

Hide

https://confluence.skatelescope.org/display/SRCSC/COR-260%3A+Astronomy+workflow+Dask+-+Deploy+dependencies+and+pipeline+environment

and

https://confluence.skatelescope.org/display/SRCSC/COR-287%3A+Astronomy+workflow+Dask+-+Run+and+validate+pipeline+execution

Show
https://confluence.skatelescope.org/display/SRCSC/COR-260%3A+Astronomy+workflow+Dask+-+Deploy+dependencies+and+pipeline+environment and https://confluence.skatelescope.org/display/SRCSC/COR-287%3A+Astronomy+workflow+Dask+-+Run+and+validate+pipeline+execution
Resolved PI.Sprint:
18.6

Feature Checklist:

Stories Completed, Outcomes Reviewed, Demonstrated, Satisfies Acceptance Criteria, Accepted by FO

Requirement Status:

PI24 - UNCOVERED
Labels_MIRO:
PI18-PB SRC-SciPlat SRC-UseCase

Description

This feature follows previous work demonstrating functionality provided by Dask Gateway. We wish to now identify and demonstrate an astronomy workflow that utilises Dask to distribute processing across multiple nodes.

This could be done with a standalone DaskHub (JupyterHub + Dask Gateway) deployment if to be picked up by a team new to Dask Gateway.

Starting points: Arpan Das/others may have access to pre-existing examples, otherwise there may be an element of adapting an existing workflow to use Python/Dask data structures. Failing that the RASCIL library may be a good starting point (see related feature: https://jira.skatelescope.org/browse/SP-2889)

Attachments

Issue Links

mentioned in: Page Loading...; Page Loading...

Structure

Activity

People

Assignee:: Collinson, James

Reporter:: Collinson, James

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Feature Progress

Story Point Burn-up: (100.00%)

Feature Estimate: 1.5

	Issues	Story Points
To Do	0	0.0
In Progress	0	0.0
Complete	11	22.0
Total	11	22.0

Dates

Created:: 29/Nov/22 2:54 PM

Updated:: 15/Apr/24 9:40 AM

Resolved:: 01/Jun/23 9:42 AM

Identify and demonstrate astronomy workflow that uses Dask