Loading...

Xporter

XML

Word

Printable

Details

Type: Enabler
Priority: Not Assigned
Fix Version/s: PI9
Component/s: COM SDP SW
Labels:
None

ARTs:

Data Processing
Benefit hypothesis:

Hide

Visibility consumers (such as Measurement Set writer and signal displays) can get metadata from Plasma store.

Show
Visibility consumers (such as Measurement Set writer and signal displays) can get metadata from Plasma store.
Acceptance criteria:
Hide

Arrow table schemas defined and documented for (reasonable subset of?) tables defined in https://confluence.skatelescope.org/display/SWSI/ADR-21+Visibility+memory+data+model - possibly as separate library/project so they can be shared in the future

Adjust visibility receiver so that it creates metadata in the above-defined table format in Plasma.

Implement visibility receiver (at minimum measurement set writer). Needs a change to the call schema. Ideally also start tracking call schemas similarly to data models

Stretch: Implement association of visibility data with visibility meta data via Plasma meta data.

Stretch: Implement support for passing table slices through Plasma by changing way we pass references.
Show
Arrow table schemas defined and documented for (reasonable subset of?) tables defined in https://confluence.skatelescope.org/display/SWSI/ADR-21+Visibility+memory+data+model - possibly as separate library/project so they can be shared in the future Adjust visibility receiver so that it creates metadata in the above-defined table format in Plasma. Implement visibility receiver (at minimum measurement set writer). Needs a change to the call schema. Ideally also start tracking call schemas similarly to data models Stretch: Implement association of visibility data with visibility meta data via Plasma meta data. Stretch: Implement support for passing table slices through Plasma by changing way we pass references.
Feature Points:
3
Initial Size:
3
WSJF:
13.333
Epic Link:
SDP PSI MVP
Agile Teams:

Team_YANDA
Due Sprint:
Sprint 5
Story Point Burn-up:
Overdue:
Outcomes:

Hide

The ACs - except for stretch - are met as follows:
AC1: new repo https://gitlab.com/ska-telescope/ska-sdp-dal-schemas/-/tree/master/sdp_dal_schemas/schemas and RTD https://developer.skatelescope.org/projects/sdp-dal-schemas/
AC2: implemented as part of YAN-561
AC3: implemented as part of YAN-563

Show
The ACs - except for stretch - are met as follows: AC1: new repo https://gitlab.com/ska-telescope/ska-sdp-dal-schemas/-/tree/master/sdp_dal_schemas/schemas and RTD https://developer.skatelescope.org/projects/sdp-dal-schemas/ AC2: implemented as part of YAN-561 AC3: implemented as part of YAN-563
Resolved PI.Sprint:
9.1

Feature Checklist:

Stories Completed, Integrated, Solution Intent Updated, BDD Testing Passes (no errors), Outcomes Reviewed, NFRS met, Demonstrated, Satisfies Acceptance Criteria, Accepted by FO

Demos:
- DP_ART_9.1
Requirement Status:

PI22 - UNCOVERED
Goals_MIRO:
SPO-878

Description

So far data is exchanged only via tensors and (pickled) metadata. The more appropriate way to approach this is to use Arrow tables for representing such data. This should both give us options for rich access interfaces (such as Awkward array) as well as enable more portability, especially to non-Python environments.

Note that https://gitlab.com/ska-telescope/sdp-dal-prototype at this point supports tables, but not table slices. Might mean that for visibility streaming a (time) table needs to get created for every chunk.

Attachments

Issue Links

mentioned in: Page Loading...; Page Loading...

Structure

Activity

People

Assignee:: Wortmann, Peter

Reporter:: Brederode, Ray

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Feature Progress

Story Point Burn-up: (100.00%)

Feature Estimate: 3.0

	Issues	Story Points
To Do	0	0.0
In Progress	0	0.0
Complete	3	5.0
Total	3	5.0

Dates

Created:: 03/Nov/20 3:45 PM

Updated:: 15/Apr/24 9:51 AM

Resolved:: 21/Dec/20 5:17 PM

Implement the improved Arrow visibility data model (i.e. metadata via Plasma)